diff --git a/docs/img/repository-access.png b/docs/img/repository-access.png new file mode 100644 index 0000000000000000000000000000000000000000..436ed5e088174b7eea72e756903d4464c0061076 Binary files /dev/null and b/docs/img/repository-access.png differ diff --git a/docs/testing/continuous-integration.md b/docs/testing/continuous-integration.md index 61a67e1e5478c46dc2b863e3c20884fb821c01a4..75529232a5aec4fffbd71a981565c8b137a87168 100644 --- a/docs/testing/continuous-integration.md +++ b/docs/testing/continuous-integration.md @@ -1,42 +1,116 @@ -# Jenkins +# Continuous Integration -Jenkins is CiviCRM's [continuous integration](https://en.wikipedia.org/wiki/Continuous_integration) server. +To prevent defects from entering the system or remaining in the system, tests are executed automatically by the Jenkins [continuous integration](https://en.wikipedia.org/wiki/Continuous_integration) server. -To prevent defects from entering the system or remaining in the system, the tests are executed automatically by Jenkins. Jenkins first runs php and javascript style checks through the usage of `civilint` before running unit tests. +Automated tests are executed in a few different circumstances: -Jenkins runs two different types of jobs a PR test job and a matrix Job. The results of the tests are published at [test.civicrm.org](https://test.civicrm.org) There is some additional documenation in the [testing-readme](https://github.com/civicrm/civicrm-core/blob/master/tests/README.md) +* [Proposal testing for core](#pr-core) provides feedback when reviewing specific proposals in [civicrm-core](https://github.com/civicrm/civicrm-core) and other key repositories. This runs as soon as someone submits (or revises) a proposal in Github. +* [Proposal testing for extensions](#pr-ext) provides feedback when reviewing proposals for extensions like [api4](https://github.com/civicrm/api4) and [flexmailer](https://github.com/civicrm/org.civicrm.flexmailer/). This serves a similar purpose, but the mechanics are different. +* [Full matrix testing](#matrix) are the most comprehensive (and resource-intensive) tests. They run every 8-24 hours and feed into decision-making about releases and code-freezes. -## Jenkins Whitelist +The results of the tests are published at [test.civicrm.org](https://test.civicrm.org) and (where feasible) linked into the Github UI. -For new (unrecognised by Jenkins) contributors, Jenkins will automatically respond "can an admin verify this patch?", and a Github user with admin permissions may approve running the tests by commenting on the PR "ok to test". +## Proposal Testing for Core {:#pr-core} -If the user is trusted, CiviCRM administrators can add the person to the whitelist by commenting "add to whitelist". +Whenever a *pull-request* (PR) is submitted or revised on Github, Jenkins runs a set of tests. PR test jobs can take anywhere from 5 min to 2 hours to complete. -## PR Test Jobs +The CiviCRM system is composed of several `git` repositories and several [test-suites](/testing/index.md). To provide quicker results, tests are selected for relevance -- for example, a patch to `civicrm-core` will trigger almost all tests (because `core` code can be referenced directly or indirectly by any test). However, a patch to the `civicrm-backdrop` integration will only trigger end-to-end tests (because the Backdrop integration code is relevant in E2E tests) -- it will skip the APIv3 tests (because the APIv3 tests use a mock CMS which is independent of Backdrop). -Jenkins runs a "PR Test" job which is triggered whenever a pull request is created or updated in GitHub. PR test jobs can take anywhere from 5 - 90 min to complete. This works for the following repos: +| Repository | <center>civilint</center> | <center>PHPUnit</center> | <center>Karma</center> | <center>Upgrade</center> | +|--------------------|--------------------------------|--------------------------|------------------------|--------------------------| +| `civicrm-core` | <center>✅ </center> | APIv3, Civi, CRM, E2E | <center>✅ </center> | <center>✅ </center> | +| `civicrm-packages` | | APIv3, Civi, CRM, E2E | <center>✅ </center> | <center>✅ </center> | +| `civicrm-drupal` | <center>✅ </center> | E2E, Drupal | <center>✅ </center> | <center>✅ </center> | +| `civicrm-backdrop` | <center>✅ </center> | E2E | | | +| ***See also*** | <center>[civilint](/tools/civilint.md)</center> | [PHPUnit Tests](/testing/phpunit.md) | <center>[Karma Tests](/testing/karma.md)</center> | <center>[Upgrade Tests](/testing/upgrades.md)</center> | -* `civicrm-core` -* `civicrm-packages` -* `civicrm-drupal` -* `civicrm-backdrop` +!!! tip "Quick results for code style" -Jenkins runs a mix of [PHPUnit](/testing/phpunit.md#suites) [PHPWebTests](/testing/selenium.md) and [Javascript UnitTests](/testing/karma.md). The details of the types of suites and purpose can be found on the index of this [chapter](/testing/index.md) + Jenkins first runs the fast code-style checks for PHP and Javascript ([civilint](/tools/civilint.md)). If this finds any problems, it will quickly abort and skip slower tests. When a new contributor is learning the code style, this allows quick iteration. -To run all the tests in one of the suites locally, you can use [civi-test-run](/tools/civi-test-run.md). +!!! tip "Executing locally" -The tests that are run are variable depending on the repository that is triggered. + To run all the tests in one of the suites locally, you can use [civi-test-run](/tools/civi-test-run.md). -* `civicrm-core`: Upgrade, karma, CRM, api_v3, E2E and Civi tests -* `civicrm-packages`: same as civicrm-core -* `civicrm-drupal`: Upgrade, Karma and E2E tests -* `civicrm-backdrop`: same as civicrm-drupal +!!! tip "Alternate CMS testing" -For PR test jobs against `civicrm-core` and `civicrm-packages`, Jenkins only builds a Drupal site to run the tests against. If you are fixing an issue with another CMS you may need to build yourself a local test environment with that CMS + For PR test jobs against `civicrm-core` and `civicrm-packages`, Jenkins uses [the `drupal-clean` build type](/tools/civibuild.md). If you are fixing an issue with another CMS, you may need to build yourself a local test environment with that CMS. -If the tests have failed for something that we suspect is a random failure, we can ask Jenkins to run the tests again by commenting in the PR "Jenkins, test this please" +!!! tip "Whitelist" -## Matrix Test Jobs + To prevent abuse of the testbot, Jenkins maintains a whitelist of Github contributors. It only runs automated tests if the contributor is recognized. + + For new contributors, Jenkins will automatically post a comment: + + > can an admin verify this patch? + + A Jenkins administrator should skim the PR. They will then comment: + + > jenkins, ok to test + + After a new contributor has submitted a few good PRs, the administrator can add them to the whitelist with another comment: + + > jenkins, add to whitelist + + Thereafter, subsequent PRs will be tested automatically. + +!!! tip "Re-running tests" + + If the tests have failed for something that we suspect is a random failure, we can ask Jenkins to run the tests again by commenting in the PR: + + > jenkins, test this please + +## Proposal Testing for Extensions {:#pr-ext} + +!!! caution "Work in progress" + + This describes a service which is currently in alpha. + +The basic ideas are the same -- whenever a *pull-request* (PR) is submitted or revised on Github, Jenkins will be notified. There's a whitelist that determines which repos/PRs/authors will be tested. The test outcome is posted to Github. + +However, unlike core testing, the extension testing uses a "convention over configuration" philosophy. Tests can be enabled with a few clicks as long as the extension follows these conventions: + +* The extension lives in its own public git repository. +* The root folder includes `info.xml` and `phpunit.xml.dist`. +* If the extension requires any other extensions, these are listed in `info.xml` and published in [the Extension Directory](https://civicrm.org/extensions/). +* The PHPUnit tests are organized into two `@group`s -- `headless` and `e2e`. +* The extension is compatible with the `drupal-clean` build type in [civibuild](/tools/civibuild.md). + +The [civix](/extensions/civix.md) code-generator produces compliant code by default. + +!!! tip "Availability" + + The test bot supports extensions in the official `civicrm` organization and some paid partner projects. If you'd like to use it on other projects, please consult with the [core team](https://civicrm.org/teams/core-team). + +!!! tip "Registration for Github repositories" + + To enable testing on a Github repository, visit [https://github.com/apps/civibot](https://github.com/apps/civibot) and proceed to "Configure". You can install `civibot` for any user/organization -- and then authorize it to access some repositories: + +  + + You *may* enable for "All repositories" (current *and* future), even if you have unrelated repositories. Civibot autodetects extensions and ignores other projects. + +!!! tip "Registration for Gitlab repositories" + + At time of writing, we have not yet setup a registration process for Gitlab repositories. However, it has an actively considered requirement. + +!!! tip "Whitelist" + + To prevent abuse of the testbot, `civibot` checks if you are an owner/collaborator/member of the Github repository: + + * If you have access, it will autotest your pull-requests. + * If you have access, it will respect manual commands like `/test`. + * If you do not have access, then you must wait for someone else to issue the `/test` command. + +!!! tip "Re-running tests" + + When reviewing a PR, you may request a new test-run with the command `/test`. The command may be interleaved with other discussion as long as it fits on its own line. For example: + + > We should re-test this PR because it's been sitting around for a month. + > + > /test + +## Full Matrix Testing {:#matrix} The other type of Job that jenkins runs is what is described as a "matrix" job. This is a much more extended version of the PR job and is usually run against multiple different webserver configurations. @@ -48,9 +122,9 @@ The matrix jobs operate on two main combinations, a PHP5.5 + MySQL5.5 server and Due to the size of the test matrix, jobs can take from 2 to 24 hours to complete depending on the job. -### Build Schedule +!!! tip "Build Schedule" -Jenkins periodically runs the full suite (once every 4-24hr) on the official codebase. + Jenkins periodically runs the full suite (once every 4-24hr) on the official codebase. Code-style is not checked on this build, but all upgrade and web tests are run.