Redis performance issue on delete contact

We have a process where we merge contacts and then delete the deleted contacts after a period of time.

However, we have more or less never run the delete deleted contacts script because it is too slow. I dug into it today and found that

• on staging it takes 15 seconds to delete 500 contacts
• on production it takes 6-10 minutes to delete 500 contacts

I've spent most of the day digging into why & determined that the queries run are identical & timings are similary. However, on production each time this line of code runs Civi::service('prevnext')->deleteItem($id); it takes a bit over 1 second. This is not the case on staging because there are no users populating the prevnext cache with searches. Hence I have diagnosed that the problem is having users & the solution is to lock their accounts.

More specifically the issue is that the code is going through all the Redis keys to remove the contact - which seems to be inefficient.

I did wonder if a quick-fix would be to only call deleteItem if the contact is not already deleted (which they are in our use case) - I would need to change the find to fetch here I think...

Alternatively there is probably some option around queuing the cache clear to happen at the end. However, I think that 1 second + delay is actually not great for users either - e.g when deduping a bunch of contacts than having each form submit take that bit longer would add up.

We are on the cusp of getting coworker going so pushing something to a queue to clear out caches might be an option.