testing.md

Testing

Testing in CiviCRM is done by a combination of human code review and testing as well as automated testing. Testing is done based on pull requests (PRs) in GitHub. Pull requests are code submitted in response to issues reported in JIRA, the issue tracking system for CiviCRM.

Automated tests

TO DO:

• http://wiki.civicrm.org/confluence/display/CRMDOC/Testing

Manual testing

When testing the process you should use is:

• Check if the PR is paired with an issue in JIRA
• Read the JIRA ticket
• Look at the bug in JIRA and ask yourself if you agree with the analysis of the bug
• Check if the automated tests have passed
• Attempt to replicate the bug
• Comment on the Github PR if you managed to replicate the issue, and how.
• Check that the bug has been fixed with the patch

In order to reproduce the bug you can reproduce the issue in the appropriate CiviCRM Sandbox. Enabling debugging can help to get more details.

CiviCRM uses a system called Jenkins to do automated testing. To get to Jenkins results for a specific PR click Show all checks and then Details.

When Jenkins runs the tests, it also setups a test site that will be available for aproximately one week. To find the link to the site click "Console Output" on the left hand menu. Click "Full Log" at the top of the page and then you can search for CMS_URL in the output.

There are multiple times that CMS_URL is in the log output.

If a PR is updated or re-tested, the initial test site will be deleted (since a new test site will have been created).

If the tests have failed for something that we suspect is a random failure, we can ask Jenkins to run the tests again by commenting in the PR "Jenkins, test this please" (this is the GitHub PR Builder plugin). For new (unrecognised by Jenkins) contributors, Jenkins will automatically respond "can an admin verify this patch?", and a Github user with admin permissions may approve running the tests by commenting on the PR "ok to test". If the user is trusted, CiviCRM administrators can add the person to the whitelist by commenting "add to whitelist".

Jenkins only builds a Drupal site that is built against the branch that your PR is modifying. If you need to test a patch against another CMS than you will want to test the patch in your own environment.

If you have an existing environment you can use git or hub to checkout the specific PR.

Hub example where 8473 is the PR ID.

hub checkout https://github.com/civicrm/civicrm-core/pull/8473

Example Git command, where 8473 is the PR ID and 4.6 is the branch.

git fetch origin pull/8473/head:4.6

If you do not have an existing environment you can use Buildkit to create one quickly. Buildkit is a powerful command line tool for CiviCRM that allows for rapidly creating CiviCRM environments. Buildkit can create installs for Drupal7, Drupal6, WordPress and Backdrop.

Using Buildkit, you can create a CiviCRM environment with the PR applied. Example Buildkit command:

civibuild create dmaster \
  --url http://localhost:8001 \
  --patch https://github.com/civicrm/civicrm-core/pull/8494 \
  --admin-pass s3cr3t

This will create a test environment with the Drupal, CiviCRM master branch and the patch in PR 8494. More detailed information is in the Civibuild documentation

Every month, a release announcement message is sent to participants in the review process. The announcement is sent on the civicrm-dev mailing-list and issue is posted on github.com/civicrm. The GitHub post mentions the account names of everyone who has submitted a PR recently (so that they get a notification).

Notifications

• When opening a PR, submitter is pointed to the CiviCRM Core contributing documentation
• At start of month @mentions is used to ping authors, e.g. https://github.com/civicrm/release-management/issues/2
• The notice to authors contains the Release Planning spreadsheet. For example.
• You can also signup for the release week planning emails. For example