diff --git a/docs/testing/continuous-integration.md b/docs/testing/continuous-integration.md index 74fd438968df7c84788ba2b55e01e07164d4e8f1..61a67e1e5478c46dc2b863e3c20884fb821c01a4 100644 --- a/docs/testing/continuous-integration.md +++ b/docs/testing/continuous-integration.md @@ -1,43 +1,57 @@ -## Jenkins +# Jenkins -To prevent defects from entering the system or remaining in the system, -the tests are executed automatically by Jenkins - our continuous integration server. +Jenkins is CiviCRM's [continuous integration](https://en.wikipedia.org/wiki/Continuous_integration) server. -The test system runs twice. Firstly, when a change is proposed through Github -(as a "pull-request" or "PR"), Jenkins tests it immediately. (This takes 5-90min.) +To prevent defects from entering the system or remaining in the system, the tests are executed automatically by Jenkins. Jenkins first runs php and javascript style checks through the usage of `civilint` before running unit tests. -Unfortunately, the full test suite would take several hours longer, so Jenkins -only runs some tests at this stage. +Jenkins runs two different types of jobs a PR test job and a matrix Job. The results of the tests are published at [test.civicrm.org](https://test.civicrm.org) There is some additional documenation in the [testing-readme](https://github.com/civicrm/civicrm-core/blob/master/tests/README.md) -The results of the tests are published at [test.civicrm.org][jenkins-test-results] +## Jenkins Whitelist -If the tests have failed for something that we suspect is a random failure, we -can ask Jenkins to run the tests again by commenting in the PR "Jenkins, test -this please" +For new (unrecognised by Jenkins) contributors, Jenkins will automatically respond "can an admin verify this patch?", and a Github user with admin permissions may approve running the tests by commenting on the PR "ok to test". -Jenkins only builds a Drupal site that is built against the branch that your -PR is modifying. If you need to test a patch against another CMS than you -will want to test the patch in your own environment. +If the user is trusted, CiviCRM administrators can add the person to the whitelist by commenting "add to whitelist". -## Jenkins Whitelist +## PR Test Jobs -For new (unrecognised by Jenkins) contributors, Jenkins will automatically -respond "can an admin verify this patch?", and a Github user with admin -permissions may approve running the tests by commenting on the PR "ok to test". +Jenkins runs a "PR Test" job which is triggered whenever a pull request is created or updated in GitHub. PR test jobs can take anywhere from 5 - 90 min to complete. This works for the following repos: -If the user is trusted, CiviCRM administrators can add the person to the -whitelist by commenting "add to whitelist". +* `civicrm-core` +* `civicrm-packages` +* `civicrm-drupal` +* `civicrm-backdrop` -## Build Schedule +Jenkins runs a mix of [PHPUnit](/testing/phpunit.md#suites) [PHPWebTests](/testing/selenium.md) and [Javascript UnitTests](/testing/karma.md). The details of the types of suites and purpose can be found on the index of this [chapter](/testing/index.md) -Jenkins periodically runs the full suite (once every 4-24hr) on the official -codebase. - -Code-style is not checked on this build, but all upgrade and web tests are run. +To run all the tests in one of the suites locally, you can use [civi-test-run](/tools/civi-test-run.md). + +The tests that are run are variable depending on the repository that is triggered. + +* `civicrm-core`: Upgrade, karma, CRM, api_v3, E2E and Civi tests +* `civicrm-packages`: same as civicrm-core +* `civicrm-drupal`: Upgrade, Karma and E2E tests +* `civicrm-backdrop`: same as civicrm-drupal + +For PR test jobs against `civicrm-core` and `civicrm-packages`, Jenkins only builds a Drupal site to run the tests against. If you are fixing an issue with another CMS you may need to build yourself a local test environment with that CMS + +If the tests have failed for something that we suspect is a random failure, we can ask Jenkins to run the tests again by commenting in the PR "Jenkins, test this please" + +## Matrix Test Jobs + +The other type of Job that jenkins runs is what is described as a "matrix" job. This is a much more extended version of the PR job and is usually run against multiple different webserver configurations. -## Running the test suite locally +The main difference between the `CiviCRM-Core-Matrix` job and a PR test job is that it runs more upgrade tests than the `CiviCRM-Core-PR`. + +The other matrix job is one that runs the webtests. + +The matrix jobs operate on two main combinations, a PHP5.5 + MySQL5.5 server and a PHP7.0 + MySQL5.7 test seerver. + +Due to the size of the test matrix, jobs can take from 2 to 24 hours to complete depending on the job. + +### Build Schedule + +Jenkins periodically runs the full suite (once every 4-24hr) on the official codebase. + +Code-style is not checked on this build, but all upgrade and web tests are run. -You can use [civi-test-run](/tools/civi-test-run.md) locally to run a full standard CiviCRM Test suite. -[jenkins-test-results]: https://test.civicrm.org/ -[testing-readme]: https://github.com/civicrm/civicrm-core/blob/master/tests/README.md