CRM_Utils_GeocodeTest throwing test-negatives everywhere
Details
The relevant code is in https://github.com/civicrm/civicrm-core/blob/5.10/tests/phpunit/CRM/Utils/GeocodeTest.php
Here's an example of the test output:
Discussion
Generally, it looks like the tests take the approach of (a) setup example DB, (b) call Google geocoder API, (c) assert that data is updated with accurate-ish info. Some ideas to mitigate:
-
Turn off testing of
CRM_Utils_GeocodeTest
completely. Communicate more loudly about the functionality moving to an extension. Perhaps bundle the extension. -
Keep the test, but reduce its scope/utility: in the test output, it shows geo coords like
0.00,0.00
. Make these acceptable. Generally, start by nulling-out the data; then call the geocoder; then assert that the data is numerical. But don't assert that the data is specifically accurate. This utility of this depends on how we wind up with0.00
values. -
Keep the test, but reduce its scope/utility: setup a dummy geocoder which uses mock data.
-
Exclude the test from execution - but keep it in the codebase.
-
Setup an API key for testing.
- From security perspective, we can have the CI server pass the key as an environment variable, and the content will be automatically censored in logs. It's not exactly hacker-proof, but it provides some protection.
- From a cost perspective, I guess it depends how much we use the api. The content of
CRM_Utils_GeocodeTest
doesn't look like it'll send a lot of requests. (Maybe ballpark of <10?) - Suppose we flag the test so that it only runs in
CiviCRM-Core-Matrix
. With 10 requests-per-test-run * 4 test-runs-per-day * 30.4 days-per-mo * 2 server-envs * 4 versions, then we'd be around 10k requests per month. - Suppose we run the test in all core PRs. If there are 200 PRs/mo and each gets 3 revisions/retests, then that's 200310 or ~6k requests per month.
- Google currently says that they charge $5 per 1k requests; with 10k+6k requests, that's $80/mo. But they also say you get a $200/mo credit. That puts us under the budget.
- If the capacity/requirements stay the same, then that's great. But... if somebody does a build-out on the tests to significantly improve coverage, then it's easy to imagine 5x increase in #requests, at which point the costs flip (5*$80=$400/mo), so we'd be on the hook for a balance of $200/mo. It's really not hard to imagine developers quietly increasing IO usage by 5x...
-
Setup an API key... and also a way for these particular tests to run less often (but still feed smoothly into QA process).
Anyway, I'm holding off on judging for a moment. Just want to track this so others can feedack.