|
|
Testy tests are temperamental -- they flip back and forth between pass/fail regularly without an easily discernible cause. In the short term, we need to track these so that we can make more educated/consistent decisions about waiving failures. In the long term, we need the list so that we can organize an effort to tackle them.
|
|
|
|
|
|
# Listing
|
|
|
|
|
|
* `api_v3_SyntaxConformanceTest.testSqlOperators with data set #28` (added 2019-08-07)
|
|
|
* This seems to be influenced by environment. It occurs frequently in the matrix tests on min (`mysql55`+`php70`) since 5.15, but it does not appear in max (`mysql57`+`php72`). Anecdotally, it hasn't been reported on dfl (`mysql57`+`php70`), so that suggests it depends on MySQL version.
|
|
|
* ~~`CRM_Utils_DateTest.testGetFromTo` (added 2019-04-08)~~ hopefully fixed 2020-05-18
|
... | ... | @@ -27,4 +29,57 @@ Additionally, these tests have reliable false-negatives: |
|
|
|
|
|
These should be resolved now:
|
|
|
|
|
|
* `CRM_Contact_Page_View_UserDashBoardTest.testDashboardContentContributionsWithInvoicingEnabled` (added 2019-01-17) |
|
|
\ No newline at end of file |
|
|
* `CRM_Contact_Page_View_UserDashBoardTest.testDashboardContentContributionsWithInvoicingEnabled` (added 2019-01-17)
|
|
|
|
|
|
# Common causes
|
|
|
|
|
|
(*This is not an exhaustive list*)
|
|
|
|
|
|
* The code and/or unit-test relies on *timing*, which may involve:
|
|
|
* __Pace of execution__: The test runs correctly on a fast computer with ample capacity. But if the CPU/RAM/storage is under load, then execution becomes slow/jittery and outcomes become unreliable.
|
|
|
* __Timezone__: The test runs correctly in one timezone and incorrectly in another. Or, the test runs correctly when two TZ's have an aligned date, but they run incorrectly when the TZs have different dates.
|
|
|
* The code relies on an *external service*.
|
|
|
* __Availability__: The external service is offline, or the network path is obstructed, or the flood control kicks in, or a network slowdown causes timeouts.
|
|
|
* __Contract Change__: The external service has made a change in its contract, removed an old end-point, modified the SSL negotiation, etc.
|
|
|
* The specific worker-node running the test may have an issue, e.g.
|
|
|
* __Resource exhaustion__: File system runs out of space or inodes.
|
|
|
* __Internal availability__: One internal service (eg mysqld) crashes - but others (eg php-fpm) remain online.
|
|
|
* __Configuration discrepancy__: Two nodes purport to have similar configurations - but are actually different. This could be intentional (e.g. new configuration is being rolled out incrementally) or unintentional (e.g. untracked files in `/etc`; e.g. build systems on two nodes have different cache content).
|
|
|
|
|
|
# Tips: Pace of execution
|
|
|
|
|
|
As a (simple) example, consider a unit-test for cache expiration logic:
|
|
|
|
|
|
```php
|
|
|
/*1*/ $cache->set('foo', 'bar', ['ttl' => 1]);
|
|
|
/*2*/ assertEquals('bar', $cache->get('foo'));
|
|
|
/*3*/ sleep(1);
|
|
|
/*4*/ assertEquals(NULL, $cache->get('foo'));
|
|
|
```
|
|
|
|
|
|
This test will almost always pass on a fast machine. However, on a slow machine, it may fail sporadically. Note that both `$cache->set()` and `$cache->get()` implicitly rely on `time()` checks - and an assumption that `time()` will be the same. However, in a slow machine, steps 1 and 2 may actually have different `time()`s.
|
|
|
|
|
|
At time of writing (circa 5.27), some tricks for wrangling these issues:
|
|
|
|
|
|
* Use `CRM_Utils_Time::getTime()` or `getTimeRaw()` functions instead of `date()` or `time()`
|
|
|
* Use `CRM_Utils_Time::setTime(...)` to set a mock time
|
|
|
* Examine/manipulate the value of [TIME_FUNC](https://github.com/civicrm/civicrm-core/pull/17414), e.g.
|
|
|
* In a failed CI test, search the console log for `TIME_FUNC` to determine which value was used.
|
|
|
* In a local CLI console, run the test with several different `TIME_FUNC`s, eg
|
|
|
```bash
|
|
|
for TIME_FUNC in frozen linear:333 linear:1000 prng:1000 ; do
|
|
|
export TIME_FUNC
|
|
|
CIVICRM_UF=UnitTests phpunit6 tests/phpunit/Path/To/FailingTest.php
|
|
|
done
|
|
|
```
|
|
|
* In PhpStorm, run the test. Use the "Run/Debug Configurations" to set the environment variable `TIME_FUNC`. Create a breakpoint in `CRM_Utils_Time::getTimeRaw()`. Run the test and consider the backtraces whenever `getTimeRaw()` runs.
|
|
|
|
|
|
# Tips: Worker node issues
|
|
|
|
|
|
These can often be identified in the console output for the test job.
|
|
|
|
|
|
Resource exhaustion and mixed availability will manifest as random (differing) steps crashing.
|
|
|
|
|
|
For a crude pass at checking configuration discrepancies, note the report about the test environment:
|
|
|
|
|
|
![Screen_Shot_2020-06-10_at_1.53.22_PM](uploads/900669eee0707d135767b6f47f2a9aef/Screen_Shot_2020-06-10_at_1.53.22_PM.png) |
|
|
\ No newline at end of file |