upgrade.md

# Upgrade Reference

!!! note "Scope"
    This document discusses upgrade conventions used in CiviCRM core code. For extensions and modules which integrate with CiviCRM, upgrade conventions are different. See [Create a Module Extension](/extensions/civix.md) for more discussion of the module-focused upgrader.

## Introduction

Within CiviCRM Core code the upgrader classes handle making any necessary database changes either through PHP or SQL code, printing out any messages before or after the upgrade happens.

## Directory Structure

| Path | Description |
| -- | -- |
| `CRM/Upgrade/Form.php` | Upgrade helper functions and infrastructure code. (Note: The name is misleading; today, the class has no responsibility for processing forms.) |
| `CRM/Upgrade/Headless.php` | Primary entry point for running upgrades through command line tools e.g. wp-cli, drush |
| `CRM/Upgrade/Page/Upgrade.php` | Primary entry point for running upgrades through the User Interface |
| `CRM/Upgrade/Incremental/*.php` | One file and one class for each major release (e.g. 4.7, 4.6), For each minor release (e.g. 4.7.12, 4.6.31) there is one function |
| `CRM/Upgrade/Incremental/*.tpl` | One Smarty SQL file for each minor release |

## Order of Upgrade Steps 

* Incremental upgrades are executed in order of versions
* Pre-upgrade message is generated before doing any upgrades by examining the version in the civicrm-version file
* For a given major-version the `php verifyPreDBState()` runs before any changes are made
* For a given minor-version there may be a php function and some SQL code in the `.tpl` file. If there is a php function for the version it will determine when the SQL is run
* For a given minor-version the "Post Upgrade" message is generated immediately after calling the php

## Incremental PHP Files

Incremental PHP-SQL steps are preferred over the Smarty-SQL where possible. This is because it gives us a chance to query the database before running upgrades and makes upgrades are safer

Each class in the `CRM_Upgrade_Incremental_php` corresponds to a major release of CiviCRM (e.g. 4.1, 4.2, 4.3). The key functions that can be defined are:

| Function | Description |
| --- | --- |
| `php public function setPreUpgradeMessage() ` | Called at the beginning of the upgrade process before any changes have been made. This function should generate any notices that site-administrators should see *before* running the upgrade. (This is called several times – once for each incremental upgrade.) | 
| `php public function verifyPreDBState() ` | Called immediately before executing a major upgrade (e.g. 4.2.4 => 4.3.0). (This is not called for each incremental upgrade.) |
| `php public function upgrade_X_Y_Z() ` | Perform any changes in a specific queue order for version X.Y.Z. During this function you *MUST* call `php $upgrade->processSQL($rev)`. Without calling this function the X.Y.Z.sql.tpl file will not be executed |
| `php public function setPostUpgradeMessage() ` | Called after each incremental upgrade. Any messages generated by this function will be displayed to site-administrators *after* the upgrade has completed. (This is called several times – once for each incremental upgrade.) |

## Tips: Prefer simple SQL semantics over API/BAO/DAO

When writing upgrade steps in PHP its preferable to use direct sql queries rather than relying on BAOs and DAOs as they may change between versions. Whereas writing SQL is more rigorous. Further it is preferable to write SQL statements especially if it involves adding or removing columns, indexes from database tables. There are helper functions such as `CRM_Core_BAO_SchemaHandler::dropIndexIfExists()`, `CRM_Upgrade_Form::addColumn()`, `CRM_Upgrade_Form::dropColumn()`. These helper functions don't depend on version specific business logic and also help protect against hard database fails by checking if what your trying to add or drop exists already.

## Tip: Write upgrades in small chunks

When updating sometimes it can be found that upgrades steps speed can vary depending on the size of the database. Sometimes some upgrades steps may take longer than the websever timeout (e.g. Apache, PHP, nginx). To prevent timeouts it's recommended to write upgrade steps in smaller steps. This also helps in debugging as the failure can be pin pointed to a smaller amount of code.

For a working example, see `CRM_Upgrade_Incremental_php_FourTwo::upgrade_4_2_beta3()`. Notice:

* The `upgrade_X_Y_Z()` function calls `addTask()` – each task will be added to the queue of pending upgrade tasks and executed in a separate HTTP request.
* `addTask()` accepts the name of a function as an argument. Optionally, it also accepts a list of parameters to pass to the function.
* The task function is static; its first argument is an instance of `CRM_Queue_TaskContext`; and its return value is a boolean (`TRUE` for success).

## Incremental SQL Files 

Writing upgrade steps in incremental PHP is preferred to incremental SQL, since the resulting upgrade code is typically clearer, and you are better able to verify the expected DB state beforehand. For example, it is not possible in all versions of MySQL to only create DB tables or index if they don't already exist, and a failed SQL statement in the upgrade process can leave the DB in a state which requires manual correction.

`CRM/Upgrade/Incremental/sql` stores SQL scripts for upgrading to each point release. Some important notes:

* The scripts are not executed directly by MySQL
* Scripts are preprocessed by Smarty which allows for the use of Smarty functions like `{ts}` and `{localize}`
* Scripts are evaluated by PHP and therefore do *Not* support Delimiter Notion or inline comments (Comments need to be placed on their own line with two dashes at the front)

!!! note "SQL Files are required"
    Creating a SQL file is required for each incremental release – even if you don't put any content in them. Why? Because the upgrade system constructs a list of incremental releases by scanning the "sql" directory. If an incremental release doesn't appear in the SQL directory, then the upgrader may skip other important steps (such as executing incremental PHP updates).

## Tip: Smarty Preprocessing SQL Files

All Smarty tags are evaluated before the sql is run. Commonly used Smarty variables and functions:

| Tag | Description |
| -- | -- |
| `{ts escape='sql'}` | The escape param is needed for escaping special characters that may be in the translated string (even if it is not in the source string) |
| `{$domainID}` | Current domain (the one the upgrade is running from). This should not be used in most cases, since usually every domain needs upgrading, not just one. See best practices section. |
| `{localize}` | Translate the following text as per the relevant locale |
| `{$multilingual}` | Is this a multilingual install? If so you may need to iterate through each one i.e.: | 
| `{$locales}` | `{if $multilingual}{foreach from=$locales item=locale} SQL goes here per locale  {/foreach} {else} Single locale SQL here {/if}` | 

## Testing 

When testing upgrades, it's important to run the upgrades against several different databases (representing different versions of CiviCRM and different upgrade paths). The tool "civicrm-upgrade-test" can automate this process. For instructions:

### CiviBuild Upgrade test

When one makes a schema change, one must also prepare and test an upgrade script. The basic cycle is:

1. Modify the upgrade script (`*.mysql` or `*.php` -- eg `CRM/Upgrade/Incremental/php/FourFive.php`)
2. Load a DB snapshot from an older version (e.g. CiviCRM 4.3.0)
3. Execute the upgrade script
4. Repeat until the upgrade works as expected

You can do these steps manually. Of course, it's a bit tedious to generate and track the DB snapshots while reloading them and rerunning the upgrade logic. If you're particularly impatient/mindless, you can use the command:

```
$ civibuild upgrade-test BUILDNAME SQLFILE
```

For example:

```shell
$ cd build/drupal-demo/sites/all/modules/civicrm
$ vi CRM/Upgrade/Incremental/php/FourFive.php
$ civibuild upgrade-test drupal-demo 4.3.0-setupsh.sql.bz2
# Uhoh, that didn't work! Try again...
$ vi CRM/Upgrade/Incremental/php/FourFive.php
$ civibuild upgrade-test drupal-demo 4.3.0-setupsh.sql.bz2
# Hooray! It worked.
```

The file `4.3.0-setupsh.sql.bz2` is a standard DB snapshot bundled with buildkit -- it contains a database from CiviCRM 4.3.0 with randomly-generated data. The `upgrade-test` command will load `4.3.0-setupsh.sql.bz2`, execute a headless upgrade, and write any errors to the log. (See console output for details.)

Of course, it's fairly common to encounter different upgrade issues depending on the original DB -- an upgrade script might work if the original DB is v4.3 but fail for v4.2. It's a good idea to test your upgrade logic against multiple versions:

```shell
$ civibuild upgrade-test drupal-demo '4.2.*' '4.3.*' '4.4.*'
```

All of the tests above use standard DB snapshots with randomly-generated data. If you want to test something more specific, then create your own DB snapshot and use it, eg:

```shell
# Make your own DB with weird data; save a snapshot
$ echo "update civicrm_contact set weird=data" | mysql exampledbname
$ mysqldump exampledbname | gzip > /tmp/my-snapshot.sql.gz
# Write some upgrade logic & try it
$ vi CRM/Upgrade/Incremental/php/FourFive.php
$ civibuild upgrade-test drupal-demo /tmp/my-snapshot.sql.gz
# Uhoh, that didn't work! Try again...
$ vi CRM/Upgrade/Incremental/php/FourFive.php
$ civibuild upgrade-test drupal-demo /tmp/my-snapshot.sql.gz
# Hooray! It worked.
```
If at any point you need to backout and load a "known-working" database, then use the DB created by the original build:

```shell
$ civibuild restore drupal-demo
```

* If using your own processes see [https://github.com/civicrm/civicrm-upgrade-test](https://github.com/civicrm/civicrm-upgrade-test)