Skip to content
Snippets Groups Projects
Commit 89cea05c authored by totten's avatar totten
Browse files

continuous-integration - Multiple updates:

* Emphasize use-cases first -- and then drill-down to tools/specifics.
* Add a placeholder section for extension PRs
* Re-style and copy-edit the section for core PRs
parent a65e3cea
No related branches found
No related tags found
No related merge requests found
# Jenkins
# Continuous Integration
Jenkins is CiviCRM's [continuous integration](https://en.wikipedia.org/wiki/Continuous_integration) server.
To prevent defects from entering the system or remaining in the system, tests are executed automatically by the Jenkins [continuous integration](https://en.wikipedia.org/wiki/Continuous_integration) server.
To prevent defects from entering the system or remaining in the system, the tests are executed automatically by Jenkins. Jenkins first runs php and javascript style checks through the usage of `civilint` before running unit tests.
Automated tests are executed in a few different circumstances:
Jenkins runs two different types of jobs a PR test job and a matrix Job. The results of the tests are published at [test.civicrm.org](https://test.civicrm.org) There is some additional documenation in the [testing-readme](https://github.com/civicrm/civicrm-core/blob/master/tests/README.md)
* [Proposal testing for core](#pr-core) provides feedback when reviewing specific proposals in [civicrm-core](https://github.com/civicrm/civicrm-core) and other key repositories. This runs as soon as someone submits (or revises) a proposal in Github.
* [Proposal testing for extensions](#pr-ext) provides feedback when reviewing proposals for extensions like [api4](https://github.com/civicrm/api4) and [flexmailer](https://github.com/civicrm/org.civicrm.flexmailer/). This serves a similar purpose, but the mechanics are different.
* [Full matrix testing](#matrix) are the most comprehensive (and resource-intensive) tests. They run every 8-24 hours and feed into decision-making about releases and code-freezes.
## Jenkins Whitelist
The results of the tests are published at [test.civicrm.org](https://test.civicrm.org) and (where feasible) linked into the Github UI.
For new (unrecognised by Jenkins) contributors, Jenkins will automatically respond "can an admin verify this patch?", and a Github user with admin permissions may approve running the tests by commenting on the PR "ok to test".
## Proposal Testing for Core {:#pr-core}
If the user is trusted, CiviCRM administrators can add the person to the whitelist by commenting "add to whitelist".
Whenever a *pull-request* (PR) is submitted or revised on Github, Jenkins runs a set of tests. PR test jobs can take anywhere from 5 min to 2 hours to complete.
## PR Test Jobs
The CiviCRM system is composed of several `git` repositories and several [test-suites](/testing/index.md). To provide quicker results, tests are selected for relevance -- for example, a patch to `civicrm-core` will trigger almost all tests (because `core` code can be referenced directly or indirectly by any test). However, a patch to the `civicrm-backdrop` integration will only trigger end-to-end tests (because the Backdrop integration code is relevant in E2E tests) -- it will skip the APIv3 tests (because the APIv3 tests use a mock CMS which is independent of Backdrop).
Jenkins runs a "PR Test" job which is triggered whenever a pull request is created or updated in GitHub. PR test jobs can take anywhere from 5 - 90 min to complete. This works for the following repos:
| Repository | <center>civilint</center> | <center>PHPUnit</center> | <center>Karma</center> | <center>Upgrade</center> |
|--------------------|--------------------------------|--------------------------|------------------------|--------------------------|
| `civicrm-core` | <center></center> | APIv3, Civi, CRM, E2E | <center></center> | <center></center> |
| `civicrm-packages` | | APIv3, Civi, CRM, E2E | <center></center> | <center></center> |
| `civicrm-drupal` | <center></center> | E2E, Drupal | <center></center> | <center></center> |
| `civicrm-backdrop` | <center></center> | E2E | | |
| ***See also*** | <center>[civilint](/tools/civilint.md)</center> | [PHPUnit Tests](/testing/phpunit.md) | <center>[Karma Tests](/testing/karma.md)</center> | <center>[Upgrade Tests](/testing/upgrades.md)</center> |
* `civicrm-core`
* `civicrm-packages`
* `civicrm-drupal`
* `civicrm-backdrop`
!!! tip "Quick results for code style"
Jenkins runs a mix of [PHPUnit](/testing/phpunit.md#suites) [PHPWebTests](/testing/selenium.md) and [Javascript UnitTests](/testing/karma.md). The details of the types of suites and purpose can be found on the index of this [chapter](/testing/index.md)
Jenkins first runs the fast code-style checks for PHP and Javascript ([civilint](/tools/civilint.md)). If this finds any problems, it will quickly abort and skip slower tests. When a new contributor is learning the code style, this allows quick iteration.
To run all the tests in one of the suites locally, you can use [civi-test-run](/tools/civi-test-run.md).
!!! tip "Executing locally"
The tests that are run are variable depending on the repository that is triggered.
To run all the tests in one of the suites locally, you can use [civi-test-run](/tools/civi-test-run.md).
* `civicrm-core`: Upgrade, karma, CRM, api_v3, E2E and Civi tests
* `civicrm-packages`: same as civicrm-core
* `civicrm-drupal`: Upgrade, Karma and E2E tests
* `civicrm-backdrop`: same as civicrm-drupal
!!! tip "Alternate CMS testing"
For PR test jobs against `civicrm-core` and `civicrm-packages`, Jenkins only builds a Drupal site to run the tests against. If you are fixing an issue with another CMS you may need to build yourself a local test environment with that CMS
For PR test jobs against `civicrm-core` and `civicrm-packages`, Jenkins uses [the `drupal-clean` build type](/tools/civibuild.md). If you are fixing an issue with another CMS, you may need to build yourself a local test environment with that CMS.
If the tests have failed for something that we suspect is a random failure, we can ask Jenkins to run the tests again by commenting in the PR "Jenkins, test this please"
!!! tip "Whitelist"
## Matrix Test Jobs
To prevent abuse of the testbot, Jenkins maintains a whitelist of Github contributors. It only runs automated tests if the contributor is recognized.
For new contributors, Jenkins will automatically post a comment:
> can an admin verify this patch?
A Jenkins administrator should skim the PR. They will then comment:
> jenkins, ok to test
After a new contributor has submitted a few good PRs, the administrator can add them to the whitelist with another comment:
> jenkins, add to whitelist
Thereafter, subsequent PRs will be tested automatically.
!!! tip "Re-running tests"
If the tests have failed for something that we suspect is a random failure, we can ask Jenkins to run the tests again by commenting in the PR:
> jenkins, test this please
## Proposal Testing for Extensions {:#pr-ext}
FIXME
## Full Matrix Testing {:#matrix}
The other type of Job that jenkins runs is what is described as a "matrix" job. This is a much more extended version of the PR job and is usually run against multiple different webserver configurations.
......@@ -48,9 +76,9 @@ The matrix jobs operate on two main combinations, a PHP5.5 + MySQL5.5 server and
Due to the size of the test matrix, jobs can take from 2 to 24 hours to complete depending on the job.
### Build Schedule
!!! tip "Build Schedule"
Jenkins periodically runs the full suite (once every 4-24hr) on the official codebase.
Jenkins periodically runs the full suite (once every 4-24hr) on the official codebase.
Code-style is not checked on this build, but all upgrade and web tests are run.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment