Let's make some D8 upgrade documentation

changed due date to April 11, 2020

For development I've since switched to the buildkit/non-roundearth install, or a variation of it, as described at https://github.com/civicrm/civicrm-drupal-8/pull/37, and about halfway down this comment are my own notes (you can ignore the first part of the comment) core#1643 (comment 33452).

In theory, upgrading is then possible from the very top-level folder by doing composer require civicrm/civicrm-core:5.x.y, or if you want to be more complete, composer require civicrm/civicrm-core:5.x.y civicrm/civicrm-packages:5.x.y civicrm/civicrm-drupal-8:5.x.y. I say in theory, because I'm doing it with dev-master and so there's nothing to upgrade to - maybe a specific git hash but I haven't tried that, at least not on purpose. It usually takes a few tries to find the right steps where at some point you start randomly trying stuff so I might have tried that in the past, and obviously it didn't work, but that was probably with roundearth.

I don't know if buildkit is going to provide a civibuild way of upgrading. I don't think the current instructions would work as-is.

To switch a roundearth install to a non-roundearth install, I suppose you could use the same technique I posted at that stackexchange link, where you make a new install in its place and then move stuff over from your backup. Haven't tried that - might require taking some stuff from the freshly created civicrm.settings.php and merging into your old, like the $civicrm_paths lines.

Thanks that's @DaveD I actually did do a roundearth install (on CiviHosting) and it works ok. By 'very top level' folder do you mean my web root (where Drupal's index.php lives) or one higher than that, where composer.json lives?

One higher, at the very top where composer.json is, otherwise it will create a second vendor folder and then you get competing versions of libraries, which was the original problem with composer and civi and one reason why civi can't just all live under modules like in drupal 7.

Thanks I'm following these instructions @DaveD and getting this rather vague response "Your requirements could not be resolved to an installable set of packages." Full error attached.

Can you please help?

I'd wager the cause is which version of civicrm-setup is being used (my error says locked at v0.2) but since this is not a parameter in composer.json itself not sure what to do.

upgrade-d8-attempt.txt

So between 5.23 and 5.25 there's going to be some incompatibilities with roundearth because of all the changes in core to support d8 via buildkit/civicrm-setup. For 5.23 it might work to just add civicrm-setup to the require line, i.e.

composer require civicrm/civicrm-core:5.23.4 civicrm/civicrm-packages:5.23.4 civicrm/civicrm-drupal-8:5.23.4 civicrm/civicrm-setup:~0.4.0

An alternative, as pointed out on the stackexchange post is to edit composer.json to have civicrm/civicrm-core:5.23.4 and civicrm/civicrm-packages:5.23.4 and civicrm/civicrm-drupal-8:5.23.4 and then use composer update, which should then update civicrm-setup too. My own experience with composer update is iffy, but it works for others.

Note for 5.23 with roundearth you'll also need to add settings to civicrm.settings.php: https://gitlab.com/roundearth/drupal-civicrm-project/-/merge_requests/2/diffs, i.e.

   $civicrm_paths['civicrm.root']['url'] = '/libraries/civicrm/';
   $civicrm_paths['civicrm.packages']['url'] = '/libraries/civicrm/packages/';

Thanks for the quick response @DaveD ! I've added the lines to civicrm.settings.php and I'm making progress past the civicrm-setup version issue. But now getting warning about zetacomponents/mail. Error attached. Thoughts? upgrade-d8-attempt-3.txt

composer is fun, yeah?

Similar thing for this one - add zetacomponents/mail:1.9.* to the require line.

Um, yes? ;-) @DaveD

Thank you SO much for your help. The upgrade is complete (attached).

Can I review composer with you? So basically the gist of this is... when composer spits out complaints about versions of package X, you add a line to the composer command resolving the confusion. I've found this documentation about version constraints, but I'm still a little fuzzy. https://getcomposer.org/doc/articles/versions.md

civicrm/civicrm-setup:~0.4.0 Does this mean "at least 0.4.0" right...So by using the ~ you forced it to use 0.4.0. But ~ means up to what version in this case?

zetacomponents/mail:1.9.* Does this mean "any version of 1.9 will do", telling composer essentially not to care what version of zetacomponents/mail to use, and proceed as long as it's 1.9.

Is my interpretation correct?

upgrade-d8-completed.txt

Great!

I'm far from a composer expert - everything I know is the result of multiple "Try, Fail, Restore from backup, Try, Fail, ..." attempts. And I'm definitely fuzzy on the ~ and the ^. The * is clearer, and yes it means any 1.9.

Thoughts: One reason I think it's so difficult to document this properly is that the point of composer is to be able to handle all the numerous scenarios. So the exact steps may depend on how the install got to where it is now and what else you have installed on the site (via composer). For example installing drush or a fancy drupal module into the same site may add its own constraints. There's also roundearth from scratch, roundearth added on to an existing drupal, and then now there's also drupal-recommended-project in addition to the previous drupal 8 composer project.

Possibly the composer update method above might work best for most people, since then you can let it figure out some things, just then at the same time you're trusting that it won't update some other library to a version you don't want if you haven't been specific about preventing that, and I don't trust composer.

Yikes, since composer update didn't work for me initially, that's why I had to ask for help. So my impression of composer is that even if you start with the MyDropWizard method for your initial CiviCRM+D8 install, future upgrades using composer and the commands needed are largely dependent on your specific configuration, including Drupal modules. This is, I agree, very difficult to document. I think we could have to explain the process starting with composer update, and if that fails, do what I did (and you helped me with), with some general guidelines on how and why to get around version issues. Is that a good starting place for documentation, you think?

That sounds like a good start. @MikeyMJCO prefers the composer update method.

I think we need a guide to move people from the roundearth method (a good idea to start with, in hindsight has some flaws!) to the "buildkit" way that DaveD has mentioned (we're using this setup - our upgrade is essentially updating the version number in composer.json and running composer update)

Thanks @MikeyMJCO can you please help create a section or spin-off of this page https://lab.civicrm.org/dev/drupal/-/wikis/drupal8-composer dedicated to Civi updates and I will help edit it?

I'd like it to live on docs.civicrm.org if we can - but the wiki is a good place to plan it collaboratively!

https://lab.civicrm.org/dev/drupal/-/wikis/Drupal-8-and-CiviCRM:-Upgrades

@MikeyMJCO agreed. docs.civicrm.org has taken a hit in terms of google visibility lately (sorry, I digress) but I agree that this is the ultimate repository of documentation, including how to update Civi alongside various CMS versions. I imagine that is months away, though. In the meantime, the wiki is a fine place to start.

We're working on fixing docs google-fu at the moment - next week in fact!;-)

@MikeyMJCO I don't seem to have permissions to edit that page. Can you provide please? @Stoob

I can't unfortunately @bgm can you add @Stoob ?

added git:civicrm-drupal git:civicrm-setup sig:usability triaged + 1 deleted label

So I just converted a roundearth (with master) install to a civicrm-setup (master) install. It hardly required doing anything. It's a stock install, with no extra modules or whatever, but seems promising.

Backup civicrm.settings.php and drupal settings.php.
Delete everything in the root folder, including drupal.
cd <one folder up from root>
composer create-project drupal/recommended-project:8.8.4 <root folder>
cd <into root folder>
SET COMPOSER_MEMORY_LIMIT=-1 (or export COMPOSER_MEMORY_LIMIT=-1 for unix)
composer require civicrm/civicrm-asset-plugin:~1.0.1 civicrm/civicrm-core:dev-master civicrm/civicrm-packages:dev-master civicrm/civicrm-drupal-8:dev-master
restore civicrm.settings.php and drupal settings.php
If you have the two extra lines from #115 (comment 34105) above you don't need them anymore.
log in and clear caches

That's more or less what we did - we just didn't nuke the root folder - we nuked vendor.

Cool. In the above I'm at the same time converting it to drupal-recommended-project. If the drupal side has lots of stuff that may or may not work as easily. To be honest I'm not sure of the differences except one says "recommended".

We've ended up pretty much using the "drupal-recommended" method anyway!

mentioned in issue #106 (closed)

Another item is that when doing upgrades, the civicrm-asset-plugin may or may not get triggered to resync the files into /web/libraries/civicrm. If it doesn't then running composer civicrm:publish (from the top folder, above vendor and web) will do it. Optionally you can add the -D flag, which is just the same as deleting the /web/libraries/civicrm folder first. Without that, it won't also delete subfolders that have been removed from the original under /vendor.

If using roundearth, it might work to use composer civicrm:sync-web-assets to do the same thing, but I haven't tested that.

It seems roundearth has now switched to use civicrm-asset-plugin, so there's now even three options when updating a roundearth install:

update as before using their existing plugin
convert the install to the newer roundearth project
convert the install to the "official" civicrm-setup version

@DaveD thanks for your comments. My current saga with D8 has come to an end for the time being, for reasons unrelated to composer updates. We will remain at D7 for now. I don't feel qualified to contribute to update documentation because I find myself far too confused by the variety of challenges presented by composer to offer meaningful input. Sorry! But good luck if you choose to press on...thanks. Just wanted to let you know.

I'm going to be working on updated D8 installation guidance for the official documentation.

Thanks @Stoob for kicking this off and thanks @MikeyMJCO for volunteering.

This might be the wrong place for this question: What is the current approach to upgrade an existing d7+civi site to d8/9+civi? Is this possible and/or is there documentation in place for this? If the question belongs in a new thread (or on stackexchange) let me know.

There's an out-of-date stackexchange post, and I haven't tried since but the gist is that you install a minimal drupal 8+civi site somewhere, run the regular drupal 7-to-8 migrator, copy over your civi tables and if necessary fix up civicrm.settings.php. The mention of roundearth and the details in that stackexchange post might lead you astray since they're old, so ignore the exact details listed.

Thanks @DaveD, so it sounds like this is relatively doable? If I understand it seems like everything still runs the same but mostly just needs some care around the paths involved.

This issue probably isn't the best place for direct support for upgrading. It relates to documentation.

added needs:docs need update label

mentioned in issue #142

Who is able to fix permissions for this page? https://lab.civicrm.org/dev/drupal/-/wikis/Drupal-8-and-CiviCRM:-Upgrades or am I missing a (not obvious to me) edit button somewhere?

Apparently I have edit access to it and not sure what's different about my account, but re-reading this thread I hear the preference would be an issue or PR against the docs repo? But I can't change permissions.

Of course! We have this -> https://docs.civicrm.org/installation/en/latest/drupal8/ this can probably be closed!

Eyeballs/Feedback/Proof Reading on this :-)

documentation/docs/sysadmin!274 (merged)

closed

@DaveD @Stoob - So I noticed that some of your initial comments in this ticket talked about the plugin issue (eg roundearth/civicrm-composer-plugin and civicrm/civicrm-asset-plugin) as meriting docs.

The recently added upgrade doc for D8 is fairly passive about that issue -- e.g. stick with whatever plugin you already have. I think this is good baseline since (a) switching between plugins could break edge-bits (e.g. any external links pointing to extern/*.php) and (b) afaik the original plugin isn't malfunctioning. It just doesn't support as many project-templates and doesn't support git/VCS/fork-based builds. But it's not being maintained, so eventually entropy will kick in and break it.

Perhaps it would make sense add an addendum to https://docs.civicrm.org/sysadmin/en/latest/upgrade/drupal8/ discussing pro/con/steps for migrating plugins?

Should this issue be kept open - or perhaps another issue filed for that part?

Yes. There should be a section about migrating. Nothing against roundearth, but it's inevitable it will diverge even more to the point it's really hard to migrate.

There is probably some information here - https://civicrm.stackexchange.com/questions/35921/drupal-8-how-do-i-shift-a-project-site-from-using-the-roundearth-installation-m although I think that some of my edits perhaps don't fully pick up some of the scope @jitendra might have some thoughts here as well.

Thanks, @DaveD @luke.stewart. Agree the Stackexchange link is a useful starting point.

So at a minimum, we should link to the SE thread? For bonus, we'd transfer content to https://lab.civicrm.org/documentation/docs/sysadmin and do more formatting+testing?

reopened

I think also useful is some instructions on the best way to test, in this case it's not just a case of ensuring composer update runs successfully but also that the resources are copied to the place that civicrm expects. If the goal is to get everything to the same state I guess either a list of things to check have been copied correctly, or a way to generate this list.

Potentially this is useful as a more general (beyond just roundearth -> ) (my menu is not showing/javascript errors etc) things to check list.

Let's make some D8 upgrade documentation

Child items ...

Activity