A mental health plan for search

I've been thinking a lot about how to move forwards with search functionality of late. When I think about the sorts of bugs and regressions we were dealing with 2 years ago versus today there are a lot of areas of the code that were constantly breaking or requiring changes 2 years ago that we now rarely touch. We have added barrage of unit tests & retrofitted sanity into the code & eventually reached the end of the line on the whackamole in a lot of areas. Not so much searches & reports....

Part of the problem is that we were late to the party getting searches and reports under unit testing - or at least some of the more obscure search functionality. But much more of the problem is the poor underlying code structure for searches and the in-built reports. In addition search builder and advanced search expose much more functionality than actually works well - creating a back door for new tricky functionality. Compounding this, the original search code had a lot of security holes in it which have sent us into various cycles of whackamole.

When we talk about search people want to

• be able to retrieve data
• take actions on that data
• create groups based on the data

In many cases they can use a report or some custom form to retrieve a group of contacts but then they can't take the actions they want on those contacts. In particular it's currently impossible to create a smart group or utilise our bank of actions without working off the search framework. The problem with getting away from our quickform search structure is that we have a lot of assets tied up with it and so creating a non-quickform search quickly runs into the fact that various actions cannot be accessed and the lift involved in getting past that is too much - so we wind up with 'yet another custom search' or another change to core to bad code.

I don't think we can break the situation where we are spending a tonne of sunk time on search functionality in the short-term. I think adding thises & thats to search & custom searches and reports will continue at least for another year. The data processor extension by @jaapjansma does a pretty good of retrofitting sanity onto the quickform search structure, but I think we should set a goal that in a year's time quickform search structure and BAO_Query object are out of active development and there is a viable alternative.

We already have a pretty good alternative for retrieving the data - apiv4. When you look at apiv4 explorer if 'feels like' all you need to turn that into a replacement for search builder is a 'create smart group function' Many of the things that work badly in search builder already work well in the apiv4 explorer but it is missing the ability to leverage our 'action-bank'. Going down the apiv4-as-smart-groups path also sets us up to offer the ability to have custom apiv4 apis that can also create smart groups and leverage actions.

That was a pretty long intro but the point of this ticket was more to spell out a path that I think allows us to start chipping away at this and to divert some of our 'product maintenance effort' towards breaking the cycle. These are the steps I think need taking (not necessarily in this order). Any of the steps, but in particular 3 & 4 would be suitable as funded CT work & that would speed them up & this whole process considerably, but I think there are things we can be doing now towards the end result.

1. create the ability to programaticially create a smart group based on an apiv4 explorer 'get' action. I think this is actually not that tricky. We would need to define the format of the array saved to 'form_values' - something like ['module' => 'apiv4', 'entity' => 'x', 'where' => [], 'select' => [], 'orderBy => []]. Perhaps ->createSavedSearch() would be an action that can be taken from apiv4? We would need to hobble UI functionality related to the smart group that might not work - ie suppress actions that don't work & the 'edit smart group search criteria' link in the group settings page - we can be a bit clunky on this in this step since it would not be exposed unless a developer exposed it. I feel like this step is pretty manageable & could be done in the next couple of releases. Since I think we should start on this I have created core#1107 (closed) for specific discussion on this issue

2. consider extending the above to cover other api actions - we talked about the idea of it being possible to create a smart group off a report. This is mostly a conversation about what sort of api an extension needs to create for it to be available for a saved search & what metadata is required. It would also involved creating a protocol to configure a url for 'edit smart group search criteria'. This could be done in conjunction with making smart groups possible from extended reports via an apiv4 action as experimental functionality. I don't see this as ever being a feature in core reports as I hope that by the time we've matured all of this we'll be figuring out how to leave them behind, but it seems like a way of opening up the transition process to leaping by extension.

3. tackle the search actions - figure out how to make them less tied to the QF architecture they sit on - could they be apis? Or just a much more sensible php interface? This is probably the deeper dive of this work and it might be a big lift but I also feel it's the sort of thing that we routinely chip away at in the product maintenance sphere & we could do that here. I suspect @mattwire has more idea than I do of that code at the moment...

4. Copy & adapt (or extract & componentise) the api v4 explorer to be search oriented. Mostly tie the search part of the interface with an action selector. Alternatively look at something like https://querybuilder.js.org/ - I feel like this piece is something CT should do as a dedicated chunk of work & which could be done at any point in the process by soliciting funding.

@totten @mattwire @colemanw @seamuslee @jitendra @monish.deb @pradeep @yashodha @bgm @lcdweb @johntwyman @mfb @pfigel @DaveD @justinfreeman