ops issueshttps://lab.civicrm.org/infra/ops/-/issues2019-04-04T20:37:46Zhttps://lab.civicrm.org/infra/ops/-/issues/546INFRA-206 cxnapp: Allow services to detect civicrm.org memberships2019-04-04T20:37:46ZtottenINFRA-206 cxnapp: Allow services to detect civicrm.org membershipsWe'd like to support agreements where CiviCRM Members (on civicrm.org) can receive extra services. For services based on https://github.com/civicrm/cxnapp/ , this means validating that the callback URL is associated an active membership.We'd like to support agreements where CiviCRM Members (on civicrm.org) can receive extra services. For services based on https://github.com/civicrm/cxnapp/ , this means validating that the callback URL is associated an active membership.https://lab.civicrm.org/infra/ops/-/issues/493INFRA-260 Monthly member payments not triggering receipts2019-01-03T17:10:09Zjoshjosh@civicrm.orgINFRA-260 Monthly member payments not triggering receiptsA handful of partners and members have noticed that the monthly membership payments in civicrm.org do not trigger receipts. Most notable is contact ID 8275 (I've been manually triggering receipts).A handful of partners and members have noticed that the monthly membership payments in civicrm.org do not trigger receipts. Most notable is contact ID 8275 (I've been manually triggering receipts).https://lab.civicrm.org/infra/ops/-/issues/843padthai: faulty disk is causing performance issues2019-01-03T17:08:54Zbgmpadthai: faulty disk is causing performance issuesCurrrent status:
* [x] Replace all 3 disks in padthai.c.o
* [x] Re-install padthai OS from scratch
* [x] Configure ~~test-ubu1204-5.c.o~~ test-1.c.o
* [x] (moved to #863) Configure ~~test-ubu1604-1.c.o~~ test-2.c.o
* [x] Restore botdyla...Currrent status:
* [x] Replace all 3 disks in padthai.c.o
* [x] Re-install padthai OS from scratch
* [x] Configure ~~test-ubu1204-5.c.o~~ test-1.c.o
* [x] (moved to #863) Configure ~~test-ubu1604-1.c.o~~ test-2.c.o
* [x] Restore botdylan.c.o
* [x] Restore www-test.c.o
* [ ] Ensure that backups are correctly configured on padthai/test-1/test-2, and that relevant files are backed up (i.e. anything other than `/etc`).
Initial ticket: What seems to be a faulty disk is causing performance issues.
```
root@padthai:~# zpool status
pool: zpadthai
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-9P
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zpadthai ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sda5 ONLINE 0 0 0
sdb5 ONLINE 0 0 0
sdc5 ONLINE 3 0 0
errors: No known data errors
```
```
# dmesg | grep sd
[54960868.851963] sd 0:0:2:0: [sdc] tag#3 CDB: Read(10) 28 00 1d 3a f4 90 00 00 f8 00
[54960868.859534] sd 0:0:2:0: [sdc] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[54960868.860477] sd 0:0:2:0: [sdc] tag#3 Sense Key : Medium Error [current]
[54960868.861381] sd 0:0:2:0: [sdc] tag#3 Add. Sense: Unrecovered read error
[54960868.862272] sd 0:0:2:0: [sdc] tag#3 CDB: Read(10) 28 00 1d 3a f4 90 00 00 f8 00
[54960868.863147] blk_update_request: critical medium error, dev sdc, sector 490402960
[54962799.895947] sd 0:0:2:0: [sdc] tag#1 CDB: Read(10) 28 00 0b 6a ec 40 00 00 18 00
[54962799.903136] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[54962799.904018] sd 0:0:2:0: [sdc] tag#0 CDB: Read(10) 28 00 0b 6a f3 90 00 00 08 00
[54962799.904884] blk_update_request: I/O error, dev sdc, sector 191558544
[54962799.905730] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[54962799.906568] sd 0:0:2:0: [sdc] tag#2 CDB: Read(10) 28 00 0b 6a f9 50 00 00 10 00
[54962799.907392] blk_update_request: I/O error, dev sdc, sector 191560016
[54962799.908224] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[54962799.909032] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current]
[54962799.909828] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[54962799.910607] sd 0:0:2:0: [sdc] tag#1 CDB: Read(10) 28 00 0b 6a ec 40 00 00 18 00
[54962799.911373] blk_update_request: critical medium error, dev sdc, sector 191556672
```https://lab.civicrm.org/infra/ops/-/issues/855c-i: Add in monitoring of MySQL Queries2018-10-16T01:59:44Zseamusleec-i: Add in monitoring of MySQL Queries@bgm @totten
I think we should probably put in some monitoring of long queries on the 3 MySQL instances if possible. I think we should probably alert if a query is running > 300s. I would find it strange on any of our test jobs that a ...@bgm @totten
I think we should probably put in some monitoring of long queries on the 3 MySQL instances if possible. I think we should probably alert if a query is running > 300s. I would find it strange on any of our test jobs that a query would be running for longer than 300shttps://lab.civicrm.org/infra/ops/-/issues/861Drupal demo sites failed to rebuild because of missing sendmail2018-11-13T06:50:27ZbgmDrupal demo sites failed to rebuild because of missing sendmailSince 2018-11-09 01:00 UTC, Drupal demo sites (dmaster/dcase) failed to rebuild because of missing sendmail:
```
Starting Drupal installation. This takes a few seconds ... [ok]
sh: 1: /usr/sbin/sendmail: not found
WD ma...Since 2018-11-09 01:00 UTC, Drupal demo sites (dmaster/dcase) failed to rebuild because of missing sendmail:
```
Starting Drupal installation. This takes a few seconds ... [ok]
sh: 1: /usr/sbin/sendmail: not found
WD mail: Error sending e-mail (from admin@example.com to [error]
admin@example.com).
Installation complete. User name: admin User password: ZUxW5Ai0soy4 [ok]
Unable to send e-mail. Contact the site administrator if the problem [error]
persists.
+ set +x
```https://lab.civicrm.org/infra/ops/-/issues/883Backup lab.c.o and chat.c.o to backups-1.c.o2022-12-14T01:50:40ZbgmBackup lab.c.o and chat.c.o to backups-1.c.oCurrently only backed up to sushi.
* [ ] lab.c.o
* [ ] chat.c.o
but both seem pretty critical.Currently only backed up to sushi.
* [ ] lab.c.o
* [ ] chat.c.o
but both seem pretty critical.bgmbgmhttps://lab.civicrm.org/infra/ops/-/issues/893Develop regular gitlab/mattermost crons to feed data to CiviCRM2019-08-29T14:43:40Zjoshjosh@civicrm.orgDevelop regular gitlab/mattermost crons to feed data to CiviCRMDevelop regular crons to feed data to CiviCRM from GitLab and MatterMost. Ideally, we are able to delineate in CiviCRM users that are actively contributing vs. those that are not or are no longer. Currently, the contributor log is only t...Develop regular crons to feed data to CiviCRM from GitLab and MatterMost. Ideally, we are able to delineate in CiviCRM users that are actively contributing vs. those that are not or are no longer. Currently, the contributor log is only tracking/valuing contributions within a rolling 12 months, so for the sake of consistency it makes sense to base "active" status on participation within the past rolling 12 months. That said, ideally we're able to differentiate between past contributors (i.e. people that have participated in some manner) vs. those that have registered but not participated.
Notes from chat with @bgm
- active mattermost user
- active gitlab user
- ex: commented in the past X months.bgmbgmhttps://lab.civicrm.org/infra/ops/-/issues/895For in-app distribution, rename "CiviMobile" to "CiviMobile Web App"2019-12-10T15:11:24ZtottenFor in-app distribution, rename "CiviMobile" to "CiviMobile Web App"Goal: The extensions `com.webaccessglobal.module.civimobile` and `com.agiliway.civimobileapi` are both called "CiviMobile", but the pretty names should each have an extra descriptor to disambiguate, i.e.
* "CiviMobile Web App" (`com.we...Goal: The extensions `com.webaccessglobal.module.civimobile` and `com.agiliway.civimobileapi` are both called "CiviMobile", but the pretty names should each have an extra descriptor to disambiguate, i.e.
* "CiviMobile Web App" (`com.webaccessglobal.module.civimobile`)
* "CiviMobile API" (`com.agiliway.civimobileapi`)
Caveat: Ensure that the web URL for the published CiviMobile extension remains the same or provides a redirect.https://lab.civicrm.org/infra/ops/-/issues/904Proactively restart mysqld on test nodes2019-07-10T01:43:11ZtottenProactively restart mysqld on test nodes__Issue__: MySQL periodically crashes, and we have not been able to find a concrete reason in the logs. It appears to happen most on `test-1` (which also handles the most test runs).
__Proposed Intervention__: Periodically, proactively ...__Issue__: MySQL periodically crashes, and we have not been able to find a concrete reason in the logs. It appears to happen most on `test-1` (which also handles the most test runs).
__Proposed Intervention__: Periodically, proactively restart mysqld.
You could easily add Jenkins job which just restarts the daemon; however, the challenge is that there may be some mix of concurrent jobs which are actively using the mysqld. You need to wait for (or create) an opportunity to restart the daemon.
[flock](https://linux.die.net/man/1/flock) seems like it might do the job, as in:
1. Pick a naming convention for a lock file (e.g. `~/bknix-dfl/var/mysql-admin-lock`)
2. At the start of every test job (`CiviCRM-Core-PR`, `CiviCRM-Core-Matrix`, etc), wrap all the work in a call to `flock` which acquires a *shared/read lock*.
3. In some cleanup job (eg `CiviCRM-PR-Cleanup`), wrap the `mysqld restart` work in an *exclusive/write lock*.
Alternatively, https://plugins.jenkins.io/build-blocker-plugin might do the job.https://lab.civicrm.org/infra/ops/-/issues/907alert.civicrm.org: upgrade to PHP 7.3 (community-messages)2021-03-21T13:24:13Zbgmalert.civicrm.org: upgrade to PHP 7.3 (community-messages)* Currently runs PHP 5.6
* Service is on www-prod.c.o.o, which runs also PHP 7.3 for docs (running as php-fpm, so both can co-exist).
https://github.com/civicrm/civicrm-community-messages.git* Currently runs PHP 5.6
* Service is on www-prod.c.o.o, which runs also PHP 7.3 for docs (running as php-fpm, so both can co-exist).
https://github.com/civicrm/civicrm-community-messages.gittottentottenhttps://lab.civicrm.org/infra/ops/-/issues/910Extension-SHA: PR tests should run on current base+head2019-07-13T20:05:46ZtottenExtension-SHA: PR tests should run on current base+headReport/discussion from [Mattermost](https://chat.civicrm.org/civicrm/pl/ouhttuz537fhjf1cj94rtbdaac):
> (**seamuslee**) @totten also I'm wondering if we should be doing a similar thing in the Extension-SHA job to be that it applies the ...Report/discussion from [Mattermost](https://chat.civicrm.org/civicrm/pl/ouhttuz537fhjf1cj94rtbdaac):
> (**seamuslee**) @totten also I'm wondering if we should be doing a similar thing in the Extension-SHA job to be that it applies the commits from the PR ontop of the current branch of the ext rather than as it does now which is a git checkout <commit> ping @coleman
>
> (**totten**) @seamuslee yeah, i think i agree with that behavior, but need an example to be clear
so if `civicrm/api4` has PR#1234 with base-branch `civicrm:56.78` and head-branch `alice:fix-foo`, then the test should be executed on the merged result of `civicrm:56.78 + alice:fix-foo`
>
> if so, that should be the behavior of https://github.com/civicrm/civicrm-infra/blob/master/jenkins-examples/Extension-SHA.bash#L62-L68
>
> which in turn is responsible for the bits of [output](https://test.civicrm.org/job/Extension-SHA/594/console) for " Build test site... `civibuild download`... `git clonebh` ..."
>
> ([git clonebh](https://github.com/civicrm/civicrm-buildkit/blob/master/bin/git-clonebh))
>
> if it's not behaving that way...then first thing is to check if the given test-run has the right inputs for `GIT_BASE` and `GIT_HEAD`
>
> (**seamuslee**) @totten i think @coleman has found that for example PR is created based on commit #5 from master (i.e. that is when it branches) but then we have a PR or something that adds commits #6, #7, #8 to master
> the PR test on the branch from #5 sometimes works sometimes doesn't because it isn't including the most recent commitshttps://lab.civicrm.org/infra/ops/-/issues/922OVH - Boxes/billing for 2020 (migrate VMs from padthai to paella)2020-01-31T18:33:43ZtottenOVH - Boxes/billing for 2020 (migrate VMs from padthai to paella)I've been running some numbers re:`padthai` and `barbecue` hosting. Notes:
1. There are 5 VMs running on these boxes (`test-1`, `test-3`, `www-test`, `www-demo`, `botdylan`)
2. Currently, padthai goes through `ca.ovh.com` (which allows...I've been running some numbers re:`padthai` and `barbecue` hosting. Notes:
1. There are 5 VMs running on these boxes (`test-1`, `test-3`, `www-test`, `www-demo`, `botdylan`)
2. Currently, padthai goes through `ca.ovh.com` (which allows 1/3/6/12mo arrangements), and bbq goes through `us.ovhcloud.com` (which is strictly month-to-month).
3. `padthai` bills at a higher rate than `barbecue` ($122/mo vs $99/mo), but `barbecue` is markedly faster than `padthai`. (25% faster single-thread per Passmark). This is definitely noticeable when running PR tests. (`test-1` takes longer to run tests than `test-3`.)
4. Being able to prepay reduces the business taxes. I don't know an exact figure, but if we prepay $2k and would have otherwise paid abut 30% * $2k intaxes, then I think it saves ~$600-700.
5. Rough notes/estimation: https://docs.google.com/spreadsheets/d/1FGnxC7Gki_mgU4xNcbKggsy9sglF7EsZCQ87eCRu3k4/edit?usp=sharing
I'm thinking that an optimal path might be to deploy 2x instances of the "Advance-2" (Xeon-E 2136, 64gb, 2x500 SSD) on the "ca.ovh.com" account. This would:
* Enable us to prepay more upfront (*reducing taxes*)
* Get better CPUs for both boxes (*faster cores and more of them*) ==> *running PRs faster* and maybe adding some GL worker capacity?
* Pay lower rate per box ($122+99/mo ==> $97+$97/mo)
* Manage both boxes from the same account (*less billing/admin*)
* Most of the VM's (`test-1`, `www-test`, `www-demo`, `botdylan`) would still be at OVH in the same CA data-center... so potentially one could keep the IPs and simply move the disk-images?
That saves about $1k for the year on tax+hosting. The main Q is the cost/difficulty of migrating the VMs to new physical boxes. @bgm Do you think this is something we could do easily?
If the labor is a problem, then a fallback option would be to prepay padthai. (There's no technical work for anyone, and we get some of the tax savings, but gotta stick it out on older CPU.)https://lab.civicrm.org/infra/ops/-/issues/924Automate ESR distribution using Gitlab2020-10-23T13:17:24ZbgmAutomate ESR distribution using GitlabProposed by Josh and brainstormed on CT calls: to use Gitlab as a way to automate the distribution of [ESR](https://civicrm.org/esr) releases.
* [x] Create a Gitlab project for ESR users (anyone with a download key)
* [ ] Automatically ...Proposed by Josh and brainstormed on CT calls: to use Gitlab as a way to automate the distribution of [ESR](https://civicrm.org/esr) releases.
* [x] Create a Gitlab project for ESR users (anyone with a download key)
* [ ] Automatically add people to that project when they go through a CiviCRM form (new member form, or ESR key request for members)
* [membership form](https://civicrm.org/civicrm/contribute/transact?reset=1&id=64)
* [existing member form](https://civicrm.org/civicrm/profile/create?gid=137&reset=1)
* [x] Semi-automate the upload of CiviCRM tar files as release artifacts on Gitlab (ex: dummy repo, create tag, upload artifact) [example](https://lab.civicrm.org/extensions/civiexportexcel/snippets/19)
* [ ] Provide a way so that ESR users can easily download the files, ex, using the Gitlab API and personal access token. [example](https://gitlab.com/gitlab-com/support-forum/issues/4154)
Question: any accounting info that needs to be handled? What's the process for Xero?
cc @totten @joshhttps://lab.civicrm.org/infra/ops/-/issues/931PR testing should validate ts() strings2020-04-03T01:38:14ZbgmPR testing should validate ts() stringsPR review sometimes catches incorrect uses of `ts`, but not always. The gettext extraction scripts are pretty good at catching many invalid use-cases.
It would be nice to have to run string-extraction on pull-requests. Technically, it s...PR review sometimes catches incorrect uses of `ts`, but not always. The gettext extraction scripts are pretty good at catching many invalid use-cases.
It would be nice to have to run string-extraction on pull-requests. Technically, it should be a quick check to add, similar to checking code syntax.
One issue we have, is that there are many 20-25 errors that are systematically thrown by the string-extractor. Some are annoying to fix issues, others are incorrect uses of 'ts' that are difficult to workaround.
It would be really useful to have a way to flag tolerated or known-issues, so that we can at least start applying some checks moving forward.
Worst case, it could be a list of regexs of code to ignore?
cc @seamuslee @totten @davedhttps://lab.civicrm.org/infra/ops/-/issues/934Remove or Update access for civicrm-docs repo on GitHub2020-06-02T17:16:29ZhomotechsualRemove or Update access for civicrm-docs repo on GitHubThe [CiviCRM Docs](https://github.com/civicrm/civicrm-docs) repo on GitHub is currently unmaintained/unmanaged - it should be a mirror of the [docs-publisher](https://lab.civicrm.org/documentation/docs-publisher) repo from GitLab.
I'd l...The [CiviCRM Docs](https://github.com/civicrm/civicrm-docs) repo on GitHub is currently unmaintained/unmanaged - it should be a mirror of the [docs-publisher](https://lab.civicrm.org/documentation/docs-publisher) repo from GitLab.
I'd like to suggest that we do the following:
1. Shut down the pull requests/issues tracker (I think the latter is already done!)
2. Rename the repo to docs-publisher (the repo itself has very little to do with actual docs content)
3. Set this up to mirror from docs-publisher on lab.
I initially had thought to just remove it but @eileen depends on it :-)homotechsualhomotechsualhttps://lab.civicrm.org/infra/ops/-/issues/936latest.civicrm.org causes cv cron problems2020-02-22T20:53:14Zbgmlatest.civicrm.org causes cv cron problemsReported by @AllenShaw on mattermost:
> That url is reporting something that looks like bad behavior too:
https://latest.civicrm.org/stable.php?format=summary
```
{"malformed":{"name":"malformed","severity":"warning","title":"Version C...Reported by @AllenShaw on mattermost:
> That url is reporting something that looks like bad behavior too:
https://latest.civicrm.org/stable.php?format=summary
```
{"malformed":{"name":"malformed","severity":"warning","title":"Version Check Failed","message":"The server failed to report on available versions. Perhaps the request was malformed."}}
```tottentottenhttps://lab.civicrm.org/infra/ops/-/issues/945c.o: ldapcivi service should not require "admin civicrm"2020-05-10T16:22:13Zbgmc.o: ldapcivi service should not require "admin civicrm"The ldap service for LDAP should not require the equivalent of "administer CiviCRM". The custom API calls probably do not have the correcty alterAPIPermission settings.
(I'm trying to reduce the number of roles on the site, and people/b...The ldap service for LDAP should not require the equivalent of "administer CiviCRM". The custom API calls probably do not have the correcty alterAPIPermission settings.
(I'm trying to reduce the number of roles on the site, and people/bots with admin roles)bgmbgmhttps://lab.civicrm.org/infra/ops/-/issues/948latest.civicrm.org: upgrade to PHP 7.32020-06-02T17:20:46Zbgmlatest.civicrm.org: upgrade to PHP 7.3This currently runs on PHP 5.6: https://latest.civicrm.org/stable.php?format=json
on stats.civicrm.org
Related issues (although not on stats.c.o): #907, #908, #909.
cc @tottenThis currently runs on PHP 5.6: https://latest.civicrm.org/stable.php?format=json
on stats.civicrm.org
Related issues (although not on stats.c.o): #907, #908, #909.
cc @tottenhttps://lab.civicrm.org/infra/ops/-/issues/953Schema definitions!2020-10-23T13:31:57ZhomotechsualSchema definitions!This is kind of a `/dev` issue and partly `/infra`.
We've defined a schema definition (using [JSON Schema](https://json-schema.org)) for docs-books and are in the process of defining one for APIv3 (using [OpenAPI](https://www.openapis.o...This is kind of a `/dev` issue and partly `/infra`.
We've defined a schema definition (using [JSON Schema](https://json-schema.org)) for docs-books and are in the process of defining one for APIv3 (using [OpenAPI](https://www.openapis.org/))
To work out and be useful for validating docs files and/or providing documentation and client generation options for the API these docs should be hosted somewhere (ideally using something like `schema.civicrm.org/v1/docs-book.json`.
We can then reference these schemas for validation of their respective files or in the case of OpenAPI to generate client implementations programmatically.
So onto the "ask". I think the ideal hosting place is to use a repo on GitLab (somewhere?) ideally with GitLab pages enabled and the domain `schema.civicrm.org` connected to it. There are additional schemas that would be useful - extension info.xml files could have a schema to allow validation. Essentially any structured data file can be validated with a schema and it's a useful way to document the available options in the file.
Currently planned schemas include:
- Docs Books
- APIv3
- Extension info.xml
Pinging @bgm, @totten & @seamusleehttps://lab.civicrm.org/infra/ops/-/issues/1012POT Scan: Update for sequentialcreditnotes, eventcart, flexmailer, etc2023-11-09T18:52:39ZtottenPOT Scan: Update for sequentialcreditnotes, eventcart, flexmailer, etcOver the past half-year, there have been a few new folders within the `civicrm-core` file-hierarchy, e.g.
```
setup
ext/sequentialcreditnotes
ext/flexmailer
ext/eventcart
ext/ewaysingle
ext/financialacls
ext/greenwich
ext/afform
ext/sea...Over the past half-year, there have been a few new folders within the `civicrm-core` file-hierarchy, e.g.
```
setup
ext/sequentialcreditnotes
ext/flexmailer
ext/eventcart
ext/ewaysingle
ext/financialacls
ext/greenwich
ext/afform
ext/search
```
The POT scanner should check these folders. Eventually, we're going to notice missing strings. There are mitigating factors - e.g. the corpus of strings isn't huge, and a lot of the strings existed before (either in different core folders or in a standalone extension), so there may be a bit of lag-time between the file-reorg and the appearance of symptoms.
There are currently two distinct localization flows -- iirc:
* The core flow - which splits strings into multiple POT files (`contribute.pot`, `event.pot`, etc), sends those to transifex, gets the PO's back out, and recombines into one MO (`civicrm.mo`). To capture the updated folders in this flow, we'd need to update `create-pot-files.sh`.
* The ext flow - which scans multiple repos, creating one POT per repo, sends those to transifex, gets the PO's back out, and publishes multiple MOs. To capture the updated folders in this flow, we'd need to update `civiextensions-update-transifex.php` and/or `create-pot-files-extensions.sh` and/or the scheduled-job.