Proactively restart mysqld on test nodes
Issue: MySQL periodically crashes, and we have not been able to find a concrete reason in the logs. It appears to happen most on test-1
(which also handles the most test runs).
Proposed Intervention: Periodically, proactively restart mysqld.
You could easily add Jenkins job which just restarts the daemon; however, the challenge is that there may be some mix of concurrent jobs which are actively using the mysqld. You need to wait for (or create) an opportunity to restart the daemon.
flock seems like it might do the job, as in:
- Pick a naming convention for a lock file (e.g.
~/bknix-dfl/var/mysql-admin-lock
) - At the start of every test job (
CiviCRM-Core-PR
,CiviCRM-Core-Matrix
, etc), wrap all the work in a call toflock
which acquires a shared/read lock. - In some cleanup job (eg
CiviCRM-PR-Cleanup
), wrap themysqld restart
work in an exclusive/write lock.
Alternatively, https://plugins.jenkins.io/build-blocker-plugin might do the job.