Upgrade test infrastructure to support 'edge' versions of dependencies like MySQL8

@totten wrote:

we currently check releases using a matrix with min and max columns

if we expect civicrm to be compatible already (and/or that updates will be quick and straight-forward), then we can use KISS approach and just bump up max (mysql57 => mysql80)

otoh, if the process is going to take longer, then we should add a new column (min, max, edge)

(note: adding an edge column could also be useful wrt to phasing-in support for php73)

... it would mean that reports of errors for infrastructure that is on the edge of getting support are not treated as blockers, and are identifiable as edge errors

Once an edge technology dependency like MySQL 8 or PHP 7.3 are deemed ready to be declared supported, the max column will change, and the edge column will change to NULL or perhaps the next higher version if people are prepared to start work on that.

This issue is focused on adding edge support to the matrix through a new column, and everything that entails in the build infrastructure and error reporting.

In order to do that, MySQL8 happens to be the first edge version to be incorporated. Issues specific to getting MySQL8.0 testing working on infrastructure will be managed at #1144 (closed). Issues regarding MySQL8 compatibility are being managed via the meta issue #392 (closed).

PHP73 is expected to be the next edge dependency.

added Concept approved Doing NYSS epic:mysql8 needs:docs need update needs:planning needs:testing ~196 triaged labels

mentioned in issue #1144 (closed)

changed the description

mentioned in issue #392 (closed)

changed the description

(Comment was saved prematurely - deleted pending update...)

OK, so let me start with a short, terse set of tasks. I believe this will be @monish.deb's first foray into CI generation 3, so I'll also add a section about background/onboarding.

Task list (Short/Terse)

1. Port the mysql80 patches from bknix@master-loco to bknix@master

2. Deploy the edge profile on test-{1,2,3}.

3. Update the CiviCRM-Core-Matrix to add the edge column, and set a constraint so that edge is only executed on master. (There's no point spending 15 hours of compute-time every day on testing edge with past releases.)

Background: Test nodes, nix, and bknix

There are currently 3 active test nodes named test-{1,2,3}. (There's an extra node or two in gcloud for spillover and manual testing, but usually that's turned off.)

Each node has a copy of each profile (min, max, dfl). The profiles are created by the nix package manager -- a profile is just a folder (like /nix/var/nix/profiles/{$MYPROFILE}) in which we've installed various packages -- apacheHttpd, redis, php70, mysql57, etc. (Under the hood, nix has many tricks+optimizations for these profiles... like deduping packages that are used multiple times.)

The profile definitions are stored in the bknix repo. The repo also stores a few other things (docs, custom packages, init scripts).

We're having this conversation at an awkward moment -- because the bknix repo currently has two divergent branches: master (the live branch) and master-loco (a bleeding-edge branch). The two branches use similar profiles, but they have different init/run scripts.

Task 1 Details: Porting mysql80

The standard package library for nix is called nixpkgs. It currently includes mysql57 but not mysql80. So I've sent a PR upstream (nixpkgs#65221) to add mysql80. In the meantime, we can update bknix@master to incorporate nixpkgs#65221.

There's an example of using it in bknix@master-loco - https://github.com/totten/bknix/compare/2ddc554..b66ab2c:

• The my.cnf template needs to be adapted. (Set query_cache_size if using MySQL 5.x; omit query_cache_size if using MySQL 8.x)
• Register a pin for the mysql80 PR.
• Update profiles/edge/default.nix

The pin and profiles should be trivial ports. The my.cnf needs to be done differently in "bknix launcher".

To get started with writing a patch for bknix@master, we can either:

• Ask @monish.deb to download bknix@master and use one of the profiles. There are docs in the repo - but the initial download may be large. At the end, he'll also have a local copy of edge that can be used for local development (fixing bugs on mysql 8.0).
• Ask @monish.deb to work on one of a gcloud server. (This reproduces the test server environment, but it's safer than working on test-{1,2,3}.)

Task 2 Details: Deploy the edge profile

We should do this together -- maybe setup a hangout and follow a "See One / Do One" structure.

assigned to @monish.deb and unassigned @totten

@totten I believe this issue is stalled on feedback from you to @monish.deb in a mattermost channel

Would it be helpful for @bgm to assist @monish.deb with some of the infrastructure challenges he is facing?

assigned to @totten and unassigned @monish.deb

@totten i have just been doing some work on this and a couple of things

1. i have found that i can't see where the query_cache_size is set in current master in the bknix-launcher and the my.cnf template doesn't appear to get copied anywhere so i'm not sure about that
2. i have created a PR in the bknix repo here https://github.com/totten/bknix/pull/2 to port the packages and pins changes as needed. I modified it slightly because of the upstream PR has been merged in
3. There are a couple of AMP PRs that are also required

• Change how MySQL is initialised
• Change how User is created and granted access to database

in my local testing i run into some issues in the Nix format due to Apache issues. I'm about to try out a standard install of ubuntu with basic buildkit set up with the modified amp to see how that goes

Also in regards to Drupal 7 set up (i.e. drupal-clean, drupal-demo) etc

The issue is that there is an outstanding Drupal 7 / Drush issue which causes the Drupal install to fail see this issue https://www.drupal.org/project/drupal/issues/2978575 the patch at comment 24 there has worked for me locally on a ubuntu 18.04 instance however how we would apply it sensibly in the civicrm buildkit is another story

@JoeMurray @totten @monish.deb

So where i believe we are at is the following:

We now have MySQL8 within the bnix edge profile as https://github.com/totten/bknix/pull/2 has been merged. Also we have had 2 key Core PRs merged to allow for the buildkit install of CiviCRM on MySQL 8 first sorting out a grouping term issue Secondly allowing for tests to find tables sensibly.

That leaves us with the following PRs that are needed for this task to be completed:

• Change how MySQL is initialised This is needed so that amp can start up the MySQL8.0 instance when installed via nix
• Change how User is created and granted access to database - This is needed as the previous method for creating a user + granting access to the generated DB has been removed in MySQL 8.
• Patch Drupal/Backdrop when downloading to it can be installed. I was able to successfully install a wp-demo build without any excess patches but Drupal and Backdrop need the patches to install successfully

added 4h of time spent at 2020-01-03

Update on this is that Tim and Monish have reviewed the above patches and have now merged them in to their respective repos. The next step is to determine a deployment plan onto test-{1,2,3} as appropriate and configure the Jenkins jobs @totten @bgm i think its now onto both of you but i also understand there is some hardware upgrades going on as well that might have some affect on this project

@totten I think we should close this issue. And we should make PHP7.4 an edge dependency, assuming that we're comfortable having PHP7.3 as max now. There is an epic issue with several subissues that I created before the break for PHP7.4 compatibility.

I've deployed edge on test-3.

We'll skip test-1 and test-2 since they don't have quite as much headroom/capacity.

To allow remote access to the edge-based builds, we'll need something like infra/ops@f7029acd

Seamus is working on adding a separate job for edge runs (so that we can specifically peg them to test-3). We did some cleanup/updates on civi-test-run and CiviCRM-Core-Matrix to avoid/reduce code-duplication between the jobs.

Since we use min/max as a signal for all releases (incl patch-updates), I'd prefer to switch max to php73 just as soon as it's passing on php73. Is it passing?

@totten AFAIK it is passing on php7.3 but i think there is the bigger question of bumping up say dfl to be php7.1 and doing some other tweaks and maybe bumping min php version for civicrm in general

added 30m of time spent at 2020-01-05

Looks like we may have a couple of test failures on PHP 7.3 https://test.civicrm.org/job/CiviCRM-Core-Edge/CIVIVER=master,label=test-3/4/testReport/(root)/CRM_Contact_BAO_QueryTest/testGetSummaryQueryWithFinancialACLDisabled/ https://test.civicrm.org/job/CiviCRM-Core-Edge/CIVIVER=master,label=test-3/4/testReport/(root)/CRM_Core_Smarty_plugins_CrmMoneyTest/testMoney_with_data_set__1/ https://test.civicrm.org/job/CiviCRM-Core-Edge/CIVIVER=master,label=test-3/4/testReport/(root)/CRM_Utils_MoneyTest/testFormatLocaleNumericRoundedByCurrency/ https://test.civicrm.org/job/CiviCRM-Core-Edge/CIVIVER=master,label=test-3/4/testReport/(root)/CRM_Utils_MoneyTest/testFormatLocaleNumericRoundedByCurrencyEuroThousand/ https://test.civicrm.org/job/CiviCRM-Core-Edge/CIVIVER=master,label=test-3/4/testReport/(root)/CRM_Utils_MoneyTest/testSpaceCurrency/ which looks like it relates to money_format function which is then pulled out in PHP7.4