To prevent defects from entering the system or remaining in the system, the tests are executed automatically Jenkins. Jenkins first runs php and javascript style checks through the usage of `civilint` before running unit tests.
To prevent defects from entering the system or remaining in the system, the tests are executed automatically by Jenkins. Jenkins first runs php and javascript style checks through the usage of `civilint` before running unit tests.
Jenkins runs two different types of jobs a PR test job and a matrix Job. The results of the tests are published at [test.civicrm.org][jenkins-test-results] [testing-readme]: https://github.com/civicrm/civicrm-core/blob/master/tests/README.md
Jenkins runs two different types of jobs: a pull-request (PR) job and a matrix job. The results of the tests are published at [test.civicrm.org][jenkins-test-results].
## PR Test Jobs
Jenkins runs a "PR Test" job which is triggered whenever a pull request is created or updated in gihub PR test jobs can take anywhere from 5 - 90min to complete. This works for the following repos:
Jenkins runs a "PR Test" job which is triggered whenever a pull request is created or updated in Github. PR test jobs can take anywhere from 5 - 90min to complete. This works for the following repos:
*`civicrm-core`
*`civicrm-packages`
...
...
@@ -23,30 +23,30 @@ There are seven main groups or suites of tests:
* Upgrade Tests - This is a test where it takes a known databasse state loads it into the test db and then runs the upgrade script to try and upgrade the database to the current code version.
* Webtests - These are selenium webtests that are run occassionally and haven't been well maintained
To run all the tests in one of the suites locally you can use [civi-test-run](/tools/civi-test-run.md).
To run all the tests in one of the suites locally, you can use [civi-test-run](/tools/civi-test-run.md).
The tests that are run are variable depending on the repository that is triggered.
* civicrm-core: Upgrade, karma, CRM, api_v3, E2E and Civi tests are run
* civicrm-packages: same as civicrm-core
* civicrm-drupal: Upgrade, Karma and E2E tests are run
* civicrm-backdrop: same as civicrm-drupal
*`civicrm-core`: Upgrade, karma, CRM, api_v3, E2E and Civi tests
*`civicrm-packages`: same as civicrm-core
*`civicrm-drupal`: Upgrade, Karma and E2E tests
*`civicrm-backdrop`: same as civicrm-drupal
For PR test jobs against civicrm-core and civicrm-packages jenkins only builds a drupal site to run the tests against. If you are fixing an issue with another CMS you may need to build yourself a local test environment with that CMS
For PR test jobs against `civicrm-core` and `civicrm-packages`, Jenkins only builds a Drupal site to run the tests against. If you are fixing an issue with another CMS you may need to build yourself a local test environment with that CMS
If the tests have failed for something that we suspect is a random failure, we can ask Jenkins to run the tests again by commenting in the PR "Jenkins, test this please"
## Matrix Test jobs
## Matrix Test Jobs
The other type of Job that jenkins runs is what is desccribed as a "matrix" job. This is a much more extended version of the PR job and is usually run against multiple different webserver configurations.
The other type of job that Jenkins runs is what is described as a "matrix" job. This is a much more extended version of the PR job and is usually run against multiple webserver configurations.
The main difference between the `civicrm-core-matrix` job and a PR test job is that it runs more upgrade tests than the CiviCRMCorePR.
The main difference between the `CiviCRM-Core-Matrix` job and a PR test job is that it runs more upgrade tests than the `CiviCRM-Core-PR`.
The other matrix job is one that runs the webtests.
The matrix jobs operate on two main combinations a PHP5.5 + MySQL5.5 server and a PHP7.0 + MySQL5.7 test seerver.
The matrix jobs operate on two main combinations, a PHP5.5 + MySQL5.5 server and a PHP7.0 + MySQL5.7 test seerver.
Due to the length and the test regieme matrix jobs can take from 2 to 24hours to complete depending on the job.
Due to the length of the full test suites, matrix jobs can take from 2 to 24hours to complete.
