Group filtering in search kit
I'm deliberately framing this issue as group filtering because group filtering has 99% of the gains and 2% of the complexity of adding group support to search kit / apiv4.
In the beginning there were groups are smart groups and you could search on them.
There was no
group_contact_cache table. It was possible to see 'hard' groups on the groups tab but it
was not possible to see smart groups. People asked to be able to see smart groups. Lobo said no.
Around 4.2 there was a sponsored 'improvement' to revert Lobo's 'no'. The group contact cache was added so that it was possible to populate all the smart groups and find out what group a contact was in.
Also around 4.2 and pretty much as soon as the new feature dropped a setting was added to allow sites to turn off the ability to see smart groups on the group tab because it was killing their sites.
Since 4.2 people have struggled with smart group cache performance, and lots of patches have been merged to try to address it.
Also since 4.2 people have been frustrated that there are very few places where you can access the smart groups a contact belongs to - we have had requests to show this on exports and my contact dashboard. However we switched back to the 'no' approach on this because it kills medium+ sizes sites.
More recently it has become effectively possible to turn off the smart group caching again by setting the time out to 0 and turning off opportunistic cache clearing. This is the approach I use and cautiously recommend.
Smart groups, groups and performance
So we have 3 things
- which groups has someone actively been added to
- which are the contacts that THIS saved search / smart group finds
- of all the many many searches saved on this site which groups would find a person
Because we UI-conflate groups & smart groups we also conflate 3 into 1 & 2. I think there is a use-case for 3 but for most sites it is not worth the performance hit. We can deliver 1 & 2 in a performant way by following our existing code practices - most notably the one that made reports work ie:
If there is a group filter in play then
- resolve the group filter to a list of relevant smart groups
- create a temporary table of all the contacts in those groups (potentially but not necessarily using the
- Use the table as the base ie
SELECT * FROM my_temp_table INNER JOIN civicrm_contact .....
Should we bypass the group contact cache?
Probably codewise it's easier to populate it first because we have code that does. However, I have pretty strong doubts that it is very often effectively pre-populated before a query on a particular group runs.
There is a cron to prepopulate smart groups - on some sites this might make sense to use. My recommendation remains 'don't unless you have tested on your site'
Could we filter off the saved search directly
Yeah this is an interesting idea - add in a saved search rather than a group. Could be powerful. Let's keep pondering
How would it look
Contact::get()->setWhere('group_id', 'IN', [8,7,89]);
Note for not in we wouldn't be able to do the Base table inner join trick & TBH we can get the tests & contract right before we do any sort of optimisation