CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-04-02T05:03:29Zhttps://lab.civicrm.org/dev/core/-/issues/1768CiviMail mail job fails to complete, even though mail is sent2023-04-02T05:03:29ZspalmstromCiviMail mail job fails to complete, even though mail is sentOverview
----------------------------------------
This relates to [CiviMail mail job fails to complete, even though mail is sent](https://civicrm.stackexchange.com/questions/35582/civimail-mail-job-fails-to-complete-even-though-mail-is-s...Overview
----------------------------------------
This relates to [CiviMail mail job fails to complete, even though mail is sent](https://civicrm.stackexchange.com/questions/35582/civimail-mail-job-fails-to-complete-even-though-mail-is-sent) that someone else has reported, but I have also seen. If a mail job fails to complete, it does not record the mails it successfully sent, so when it starts again, it sends the emails again.
Reproduction steps
----------------------------------------
It is difficult to reproduce this because we do not always see the behaviour in our environment (Joomla in a shared hosted environment), where we sometimes see this behaviour and at other times we do not because the jobs complte.
Current behaviour
----------------------------------------
Successful emails sent are only reported at the end of each batch.
Expected behaviour
----------------------------------------
The database is updated after each email is sent, so in the event of the batch failing to complete, CiviMail still knows what it has sent.
Environment information
----------------------------------------
* __Browser: Not applicable
* __CiviCRM: Last seen with 5.24.x
* __PHP:_7.2__
* __CMS:__ _Joomla 3.9.x, but probably not relevant._
* __Database:__ _Not relevant.
* __Web Server:__ Not applicable
Comments
----------------------------------------
As aforesaid, we only see the issue when batches fail to complete. We worked around the issue by reducing the mailer batch size to 5.https://lab.civicrm.org/dev/core/-/issues/3620"View in Browser" does not feed mailing statistics2024-02-19T05:03:28Ztotten"View in Browser" does not feed mailing statisticsI just want to log the existence of the issue generally... suppose you send a newsletter and it has a "View in Browser" link.
Suppose the recipient uses that link. The view the mailing and then drill-down, clicking through some more lin...I just want to log the existence of the issue generally... suppose you send a newsletter and it has a "View in Browser" link.
Suppose the recipient uses that link. The view the mailing and then drill-down, clicking through some more links.
What happens to the stats?
* The web-based view does not count as an "Open". The MUA may or may not report back the "Open" on the original message.
* You will not have any "Click Through" events logged.
This is because the "View in browser" variant of the newsletter is not instrumented with tracking codes (while the MUA variant of the newsletter does have tracking codes).https://lab.civicrm.org/dev/core/-/issues/1734Remove `profile listings and forms` permission2023-03-08T05:03:29ZwmortadaRemove `profile listings and forms` permissionOverview
----------------------------------------
The `profile listings and forms` permission is shorthand for four other permissions relating to profiles - `profile create`, `profile edit`, `profile listings`, `profile view`.
I'm guess...Overview
----------------------------------------
The `profile listings and forms` permission is shorthand for four other permissions relating to profiles - `profile create`, `profile edit`, `profile listings`, `profile view`.
I'm guessing that there was a historic reason for this but I think it is confusing and potentially leaves sites less secure as a result. I propose that it is removed. Instead each of the four permissions are set individually as required.
Current behaviour
----------------------------------------
There are five permissions relating to profiles:
* `profile create`
* `profile edit`
* `profile listings`
* `profile view`
* `profile listings and forms`
The latter is a catch all for the other four.
Proposed behaviour
----------------------------------------
There are four permissions relating to profiles:
* `profile create`
* `profile edit`
* `profile listings`
* `profile view`
As part of the upgrade process the user is prompted to review permissions and ensure that any user roles that currently have the `profile listings and forms` permission are given each of the above permissions.
Comments
----------------------------------------
The reason I have raised this issue is that it was unclear to me what each of these permissions were four. Following some discussion, I have made [some](https://github.com/civicrm/civicrm-user-guide/issues/355) [proposals](https://github.com/civicrm/civicrm-user-guide/pull/441) to improve the documentation but think it would be better if this permission were just removed.
The permissions are defined in [CRM/Core/Permission.php](https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/Permission.php#L289-L326).
I've found two uses of this permission in core:
* [CRM/Contact/BAO/Contact.php](https://github.com/civicrm/civicrm-core/blob/363ce69adee8c277d94d9b31ef3ca8782db9394f/CRM/Contact/BAO/Contact.php#L3672)
* [CRM/Core/BAO/UFGroup.php](https://github.com/civicrm/civicrm-core/blob/4cef91ce8a2df4c34ac30fc05a61900ffb046e6f/CRM/Core/BAO/UFGroup.php#L1290)
And one in CiviVolunteer:
* [api/v3/VolunteerUtil.php](https://github.com/civicrm/org.civicrm.volunteer/blob/master/api/v3/VolunteerUtil.php#L176)
(There may be more, but these are the ones I've spotted.)https://lab.civicrm.org/dev/core/-/issues/1730Allow extensions to skip entity operations by catching exceptions thrown in p...2023-03-06T05:03:25ZjensschuppeAllow extensions to skip entity operations by catching exceptions thrown in pre and post hooksOverview
----------------------------------------
Extensions should be able to skip actions performed on entities with a warning/error message. `hook_civicrm_pre()` and `hook_civicrm_post()` implementations do not allow for skipping alto...Overview
----------------------------------------
Extensions should be able to skip actions performed on entities with a warning/error message. `hook_civicrm_pre()` and `hook_civicrm_post()` implementations do not allow for skipping altogether, but only for altering parameters.
Example use-case
----------------------------------------
The [de.systopia.donrec](https://github.com/systopia/de.systopia.donrec) extension forbids contributions be deleted when they have been legally receipted, since allowing deletion of such contributions would be considered fraudulent behavior by tax authorities.
Current behaviour
----------------------------------------
Currently, exceptions thrown in implementations of those hooks produce fatal errors, which, when run via Ajax, appear to "hang the system", because the XHR returns a 500. There is no way for extensions to skip such an action in a way that would be handeled by the UI.
Proposed behaviour
----------------------------------------
`hook_civicrm_pre()` and `hook_civicrm_post()` implementations should be able to skip execution of the current action (e.g. deleting an entity). The invoking code should therefore catch exceptions thrown by those implementations. This would allow for displaying exception messages as error message popups in the UI.
Comments
----------------------------------------
I'm not sure whether this can be done generically or would have to be implemented separately for each caller. I guess it's the latter. In that case, consider this issue as a discussion starter for whether this is desirable to include in CiviCRM Core.https://lab.civicrm.org/dev/core/-/issues/1727Allow sites to not have Billing Address Location Type2023-03-09T05:03:25ZJoeMurrayAllow sites to not have Billing Address Location TypeOverview
----------------------------------------
Currently CiviCRM has a confusing array of default location types and flags. This issue proposes changes that would allow the Is Billing flag to be the only indication that a location is ...Overview
----------------------------------------
Currently CiviCRM has a confusing array of default location types and flags. This issue proposes changes that would allow the Is Billing flag to be the only indication that a location is the billing location rather than also having a Location Type of Billing. The intent is to allow instances to experiment with simpler approaches to Location Types in order to improve the usability of the product.
Example use-case
Add a hook to allow billing location to be determined by an alternative implementation. Sample LExIM extension approach would return the address which has isBilling==True if there is no LocationType of 'Billing' or if there is no address passed to function with LocationType of 'Billing' from https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/Payment/BaseIPN.php#L491
In current places where code creates a 'Billing' type address one potential approach would assume it should be home location type for individual and household contacts and work for organizational contacts. In all cases the Is Billing would still be set. The objective of this proposal is to refactor and introduce hooks in order to allow a LExIM extension to work. More details about specific places in code that need to be changed and ideas on what exactly to do in a simpler approach to be provided if there is potential for this proposal to go forward.
Comments
----------------------------------------
Just testing water for potential support before developing more detailed proposal. Otherwise we'll likely just override core for this particular client.JoeMurrayJoeMurrayhttps://lab.civicrm.org/dev/core/-/issues/1726Allow financial types to not have Expense account defined2022-08-26T20:51:52ZJoeMurrayAllow financial types to not have Expense account definedThis is a proposal for discussion and refinement.
Overview
----------------------------------------
Simplify accounting configuration to remove requirement for, and default creation of, widely unused stuff. In particular, don't require ...This is a proposal for discussion and refinement.
Overview
----------------------------------------
Simplify accounting configuration to remove requirement for, and default creation of, widely unused stuff. In particular, don't require Expenses account for every financial type, nor create relations to Expense and Premium accounts by default when creating a financial type.
Example use-case
----------------------------------------
1. Click on **Administer > CiviContribute > Financial Types**.
1. Click **Add Financial Type**.
1. Enter **Name** and click **Save**.
1. In Financial Accounts, there are Banking Fees and Premiums accounts, which is **undesirable**.
1. Click **Accounts** on the new Financial Type row.
1. Beside the 'Expense Account is', click **Delete**, then confirm by clicking **Delete** again.
1. Click on **Contributions > New Contribution**.
1. Select the Financial Type created above that does not have an Expense Account set up for it anymore, fill in **Contributor** and **Total Amount**, and click **Save**.
1. Try to edit the contribution but not in a popup, for example, go to the contact's page, right click on the edit button for the contribution and Copy Link Address, then paste address into a new tab. You'll see "Sorry, due to an error, we are unable to fulfill your request at the moment. You may want to contact your administrator or service provider with more details about what action you were performing when this occurred.
One of parameters (value: ) is not of the type Integer". This is caused by missing Expense account, **even though it is not needed**.
Current behaviour
----------------------------------------
See above.
Proposed behaviour
----------------------------------------
On creation of Financial Type, no Expense or Premiums account relationship would be setup. On editing a contribution (with a line item) with a financial type without an Expense account relationship setup, no error would occur.
Comments
----------------------------------------
The expectation when this was implemented circa 2014 was that payment processors would all soon record banking fees. That hasn't been the case for a variety of reasons.Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/1697set is_deceased to not null in schema and upgrade script2020-04-09T23:07:11ZMichael McAndrewset is_deceased to not null in schema and upgrade scriptOver the past few years, compelling evidence has has emerged to suggest that Zombies do exist in CiviCRM databases. Here are a couple of the most high profile cases:
- https://civicrm.org/extensions/zombie-check
- https://civicrm.org/bl...Over the past few years, compelling evidence has has emerged to suggest that Zombies do exist in CiviCRM databases. Here are a couple of the most high profile cases:
- https://civicrm.org/extensions/zombie-check
- https://civicrm.org/blog/simonparkervitiligosocietyorguk/solution-all-recipients-not-showing-up-for-sending-mass-email
Whilst various attempts have been made to eliminate the scourge of zombies once and for ever, the most notable of which being https://lab.civicrm.org/dev/core/-/blob/master/CRM/Upgrade/Incremental/sql/4.7.beta2.mysql.tpl#L15-17 ...
```sql
-- CRM-17147 People with empty deceased-flag ('is null') get removed from recipient list of a mailing
UPDATE civicrm_contact SET is_deceased = 0 WHERE is_deceased IS NULL;
ALTER TABLE civicrm_contact ALTER COLUMN is_deceased SET DEFAULT 0;
```
... it has come to light that Zombies persist across CiviCRM instances.
Although the upgrade script defaults all existing fields to 0, and new sites have a default of 0, this does not prevent the columns from being asssigned a null value.
:coffin: :cross:
PR here that alters the definition for new tables and adds another upgrade step: https://github.com/civicrm/civicrm-core/pull/170255.26.0https://lab.civicrm.org/dev/core/-/issues/1690Multiple scheduled reminders per contact when scheduled job fails2023-03-11T05:03:20ZMartinMultiple scheduled reminders per contact when scheduled job failsOverview
----------------------------------------
We're seeing scheduled reminders occurring more than once. In our case we have reminders set up before expiry of memberships (e.g. 7 days before). In some cases when we look at the activi...Overview
----------------------------------------
We're seeing scheduled reminders occurring more than once. In our case we have reminders set up before expiry of memberships (e.g. 7 days before). In some cases when we look at the activities for a contact, there are multiple entries for a reminder (all with status "completed"). For a given contact these show up once per cron run, for multiple subsequent cron intervals (i.e. 1 activity each hour for multiple hours in a row).
These occurrences coincide with failures of the send_reminder scheduled job. Looking at the job log shows this result:
```
Entity: Job Action: send_reminder
Summary
Finished execution of Scheduled reminders sender with result: Failure, Error message: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away
Details
Parameters parsed (and passed to API method):
a:1:{s:7:"version";i:3;}
Full message:
Finished execution of Scheduled reminders sender with result: Failure, Error message: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away
```
It appears when this failure happens for the scheduled job, the contact still get an activity for their scheduled reminder, and they continue to do so each cron run (in our case this scheduled job is set to be hourly), until the job shows a success (at this point is the last activity they get).
Looking at the code in CRM_Core_BAO_ActionSchedule it appears that the email is sent (sendReminderEmail) before the activity is created (createMailingActivity), so I expect the contact has in fact received multiple emails, NOT that multiple activities were created without emails being sent.
Without digging further, I'm guessing maybe civi considers ALL reminders have failed in the batch if the scheduled job fails, even if some were sent out and the job failed partway through.
Reproduction steps
----------------------------------------
In our case the "MySQL server has gone away" db error appears to be causing this, which of course is a problem unto itself, but would be preferable if Civi could handle this situation better.
Current behavior
----------------------------------------
Contact receives multiple emails and activity entries when only 1 should occur. They are getting n+1 of these, where n is the number of times the scheduled job fails.
Expected behavior
----------------------------------------
It would be preferable for only 1 email to be sent to each contact, and that if the job fails, only the unsent emails are then sent later.
Contacts should only get 1 email reminder, and 1 "completed" activity. Maybe they should also get other activities for the failed cron runs with another status such as "failed" or "cancelled", but that probably doesn't matter much either way.
Environment information
----------------------------------------
* __CiviCRM:__ 5.23.0
* __PHP:__ 7.2.29
* __CMS:__ Drupal 7.69
* __Database:__ MySQL 5.7.29https://lab.civicrm.org/dev/core/-/issues/1681End of life planning for MySQL server 5.0 - 5.62024-02-06T05:03:27ZtottenEnd of life planning for MySQL server 5.0 - 5.6Overview
--------
This issue aims to consolidate the range of supported versions of MySQL.
At time of writing, all versions of MySQL from 5.5 and below are have reached their "end of life" (i.e. Oracle's "Extended Support Period" with ...Overview
--------
This issue aims to consolidate the range of supported versions of MySQL.
At time of writing, all versions of MySQL from 5.5 and below are have reached their "end of life" (i.e. Oracle's "Extended Support Period" with security updates has ended). MySQL 5.6 will be EOL in Feb 2021. Decent summary: https://dba.stackexchange.com/questions/234261/mysql-5-7-community-server-end-of-life-support
Initial Status (March 2020)
---------------------------
Regarding the declared requirements:
* In `CRM/Utils/Check`, there appears to be no specific assertion about the MySQL server version.
* In `CRM/Upgrade`, there appears to be no specific assertion about the MySQL server version.
* In `Civi/Install` and `install/`, there does appear to be an assertion that MySQL is at least 5.1.
* NOTE: There are some assertions about the MySQL *client* version, but that's a separate matter.
* In the automated test systems, we run tests against MySQL 5.5 (`min`), 5.7 (`dfl`/`max`), and 8.0 (`edge`).
* In the [Sysadmin Guide: Requirements](https://docs.civicrm.org/sysadmin/en/latest/requirements/), it says: "Your MySQL version should be 5.7.5 or greater or MariaDB 10.0.2 or greater. CiviCRM does still work on older versions but there is some functionality missing, notably around ensuring various database actions don't compete for the same resources."
Regarding actual usage: `stats.civicrm.org` reports the 10 most popular DBs, but it doesn't report on the less used ones. So I dug into the data for a list of sites that have pinged back within the past year to summarize their current/latest DB version. (*Queries: https://gist.github.com/totten/cc22772673579d878601b728f446518a*)
This shows overall percentages:
```sql
+-------+----------+----------+
| MySQL | site_cnt | site_pct |
+-------+----------+----------+
| 5.0 | 5 | 0.0362 |
| 5.1 | 141 | 1.0214 |
| 5.5 | 1666 | 12.0690 |
| 5.6 | 1856 | 13.4454 |
| 5.7 | 4509 | 32.6644 |
| 8.0 | 7 | 0.0507 |
| 10.0 | 807 | 5.8461 |
| 10.1 | 2333 | 16.9009 |
| 10.2 | 1334 | 9.6639 |
| 10.3 | 908 | 6.5778 |
| 10.4 | 238 | 1.7241 |
+-------+----------+----------+
```
(Note: To simplify the results, I've made no distinction between MySQL / MariaDB / Percona. Within the 5.x series, the numbers should be broadly comparable. Outside 5.x, they diverge more, but you can easily discern MySQL 8.x and MariaDB 10.x.)
Of course, sites with old versions of MySQL often have old versions of Civi. Folks on Civi 3.x probably don't care too much how Civi 5.x handles the MySQL deprecation/EOL process, but folks on Civi 5.x may. Let's break down the usage further.
```sql
+------+-------+----------+----------+
| Civi | MySQL | site_cnt | site_pct |
+------+-------+----------+----------+
| 2.x | 5.5 | 1 | 0.0072 |
| 3.x | 5.0 | 2 | 0.0145 |
| 3.x | 5.1 | 2 | 0.0145 |
| 3.x | 5.5 | 22 | 0.1594 |
| 4.x | 5.0 | 3 | 0.0217 |
| 4.x | 5.1 | 111 | 0.8041 |
| 4.x | 5.5 | 651 | 4.7160 |
| 5.x | 5.1 | 28 | 0.2028 |
| 5.x | 5.5 | 992 | 7.1863 |
+------+-------+----------+----------+
```
Additional notes:
* I don't want to overload with another chart, but FYI those Civi 5.x users are actually quite spread around (5.0 through 5.22+.)
* The last Ubuntu with MySQL 5.5 as standard was released in 2014. See [Distrowatch summary](https://distrowatch.com/table.php?distribution=ubuntu) for more details.
Proposal
--------
Over the next few months, incrementally phase out support for versions of MySQL 5.5 and earlier. We can move more aggressively with MySQL 5.0-5.4 because they've been out of circulation longer and have far fewer users.
* __Circa Civi 5.24 (Apr 2020) or 5.25 (May 2020)__: *Deprecate MySQL 5.1 - 5.5.*
* The status-check, installer, and upgrader should all warn if a site uses MySQL 5.5 or lower. Recommend upgrades to MySQL 5.7.
* Note: ~7% of current sites run Civi 5.x with an affected MySQL release.
* __Circa Civi 5.26 (Jun 2020)__: *Remove MySQL 5.1 - 5.4.*
* The installer and upgrader should no longer allow deployments on MySQL 5.1 - 5.4.
* Note: ~0.2% of current sites run Civi 5.x with an affected MySQL release.
* Note: The current ESR is 5.21. We might aim for the next ESR (circa 5.27) to be in this bucket - being the last to support MySQL 5.5.
* __Circa Civi 5.28 (Aug 2020) or 5.29 (Sep 2020)__: *Remove MySQL 5.5.*
* The installer and upgrader should no longer allow deployments on MySQL 5.5.
* Note: ~7% of current sites run Civi 5.x with an affected MySQL release. This would work out to ~5 months notice.
* Update automated testing so that the lowest tested version is MySQL 5.6.
That basically gets us caught-up so that the minimum matches the oldest supported version of MySQL.
Looking forward, upstream is marking MySQL 5.6 EOL in Feb 2021. Additional measures to prep for that could be:
* __Circa Civi 5.26 (Jun 2020)__: *Deprecate MySQL 5.6.*
* The status-check, installer, and upgrader should all warn if a site uses MySQL 5.6 or lower. Recommend upgrades to MySQL 5.7.
* Note: At time of writing, ~13% of current sites run Civi 5.x with MySQL 5.6.
* __Circa Civi 5.34 (Feb 2021)__: *Remove MySQL 5.6.*
* The installer and upgrader should no longer allow deployments on MySQL 5.6.
* Note: At time of writing, ~13% of current sites run Civi 5.x with MySQL 5.6. This would work out to ~7 months notice.
* Update automated testing so that the lowest tested version is MySQL 5.7.
Comments
--------
I'm not quite certain about the growing divergence between MySQL and MariaDB. My hope is that it doesn't matter much while the minimum is 5.6 and lower, but it could be more of an issue when the minimums get to 5.7 and beyond. At that point, we'll probably have to revise the test matrix to run 3 or 4 different DBMS variants.https://lab.civicrm.org/dev/core/-/issues/1680Convert email contact "to" boxes to Entity Ref fields2023-03-01T05:03:24ZmeppsConvert email contact "to" boxes to Entity Ref fieldsCurrently, the email "to" boxes in the email contact form use an older form of search. It would be a better user experience to convert this to using the Entity Ref search box from here: https://github.com/civicrm/civicrm-core/blob/db9e09...Currently, the email "to" boxes in the email contact form use an older form of search. It would be a better user experience to convert this to using the Entity Ref search box from here: https://github.com/civicrm/civicrm-core/blob/db9e0947e008ba06308da5664e5c36a3bc2736df/js/Common.js#L739.https://lab.civicrm.org/dev/core/-/issues/1678Export time out avoidance2023-03-01T05:03:23ZeileenExport time out avoidanceSites have a php time out to prevent long running processes & out of control loops. However, there are some processes that are known to need more time that others & we can extend the timeout within php to keep them alive. This is pretty ...Sites have a php time out to prevent long running processes & out of control loops. However, there are some processes that are known to need more time that others & we can extend the timeout within php to keep them alive. This is pretty safe because it's within a loop so we extend a little each row rather a huge time extension
Basically we can check each iteration here & if less than 10 seconds remains we can add 10 seconds as a sort of heartbeat https://www.php.net/manual/en/function.set-time-limit.phphttps://lab.civicrm.org/dev/core/-/issues/1671Deleting parent address should delete shared address2021-03-22T15:20:47ZsamuelsovDeleting parent address should delete shared addressCurrently, in CiviCRM, when we do a shared address :
* all the address fields are copied from the master address to the shared address.
* when the master address is deleted, the link is simply deleted (field master_id set to NULL) but th...Currently, in CiviCRM, when we do a shared address :
* all the address fields are copied from the master address to the shared address.
* when the master address is deleted, the link is simply deleted (field master_id set to NULL) but the child address is kept as it is
From a user/business logic perspective, I believe it doesn't makes sense to keep the child address when the master is deleted.
I propose to delete all the shared address when a master address is deleted.
At the very least, we should be able to find the addresses that previously had a primary address that has been deleted.
Any concern or use case where the child addresses should be kept?https://lab.civicrm.org/dev/core/-/issues/1665MySQL uses filesort index when building the query which can cause performance...2020-08-13T00:03:03Zjustinfreeman (Agileware)MySQL uses filesort index when building the query which can cause performance issuesMySQL uses filesort index when building the query which can cause performance issues. This can be solved by simply removing the GROUP BY and ORDER BY options from query in **function alphabetQuery**, **CRM/Contact/BAO/Query.php** see htt...MySQL uses filesort index when building the query which can cause performance issues. This can be solved by simply removing the GROUP BY and ORDER BY options from query in **function alphabetQuery**, **CRM/Contact/BAO/Query.php** see https://github.com/civicrm/civicrm-core/blob/master/CRM/Contact/BAO/Query.php#L5012
Original query:
~~~
$query = "SELECT DISTINCT LEFT(contact_a.sort_name, 1) as sort_name
{$sqlParts['from']}
{$sqlParts['where']}
{$sqlParts['having']}
GROUP BY sort_name
ORDER BY sort_name asc";
~~~
Change query to:
~~~
$query = "SELECT DISTINCT LEFT(contact_a.sort_name, 1) as sort_name
{$sqlParts['from']}
{$sqlParts['where']}
{$sqlParts['having']}";
~~~
This change has no visible impact on the search results pager or listing.
The query was generated by CiviCRM when doing an Advanced Search and selecting 6 CiviCRM Groups (not Smart Groups) - displaying results as contacts.
Before this change, note the "Using filesort" on 24754 rows. Query takes longer than 90 seconds to complete - can trigger a PHP timeout.
~~~
MariaDB [ajpulmse_crm]> explain SELECT DISTINCT LEFT(contact_a.sort_name, 1) as sort_name FROM civicrm_contact contact_a LEFT JOIN civicrm_group_contact `civicrm_group_contact-5e742600a44d
7` ON (contact_a.id = `civicrm_group_contact-5e742600a44d7`.contact_id AND `civicrm_group_contact-5e742600a44d7`.status IN ('Added')) LEFT JOIN civicrm_group_contact `civicrm_group_contact-
5e742600a47a7` ON (contact_a.id = `civicrm_group_contact-5e742600a47a7`.contact_id AND `civicrm_group_contact-5e742600a47a7`.status IN ('Added')) LEFT JOIN civicrm_group_contact `civicrm_gr
oup_contact-5e742600a4a78` ON (contact_a.id = `civicrm_group_contact-5e742600a4a78`.contact_id AND `civicrm_group_contact-5e742600a4a78`.status IN ('Added')) LEFT JOIN civicrm_group_contact
`civicrm_group_contact-5e742600a4d39` ON (contact_a.id = `civicrm_group_contact-5e742600a4d39`.contact_id AND `civicrm_group_contact-5e742600a4d39`.status IN ('Added')) LEFT JOIN civicrm_g
roup_contact `civicrm_group_contact-5e742600a4fde` ON (contact_a.id = `civicrm_group_contact-5e742600a4fde`.contact_id AND `civicrm_group_contact-5e742600a4fde`.status IN ('Added')) LEFT JO
IN civicrm_group_contact `civicrm_group_contact-5e742600a529d` ON (contact_a.id = `civicrm_group_contact-5e742600a529d`.contact_id AND `civicrm_group_contact-5e742600a529d`.status IN ('Adde
d')) WHERE ( ( ( ( `civicrm_group_contact-5e742600a44d7`.group_id IN ("498") ) ) ) OR ( ( ( `civicrm_group_contact-5e742600a47a7`.group_id IN ("499") ) ) ) OR ( ( ( `civicrm_group_contact-5
e742600a4a78`.group_id IN ("505") ) ) ) OR ( ( ( `civicrm_group_contact-5e742600a4d39`.group_id IN ("504") ) ) ) OR ( ( ( `civicrm_group_contact-5e742600a4fde`.group_id IN ("503") ) ) ) OR
( ( ( `civicrm_group_contact-5e742600a529d`.group_id IN ("502") ) ) ) ) AND (contact_a.is_deleted = 0) GROUP BY sort_name;
+------+-------------+-------------------------------------+------+----------------------------+----------------------------+---------+---------------------------+-------+------------------
-----------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra
|
+------+-------------+-------------------------------------+------+----------------------------+----------------------------+---------+---------------------------+-------+-----------------------------------------------------------+
| 1 | SIMPLE | contact_a | ref | index_is_deleted_sort_name | index_is_deleted_sort_name | 1 | const | 24754 | Using where; Using index; Using temporary; Using filesort |
| 1 | SIMPLE | civicrm_group_contact-5e742600a44d7 | ref | UI_contact_group | UI_contact_group | 4 | ajpulmse_crm.contact_a.id | 1 | Using where; Distinct |
| 1 | SIMPLE | civicrm_group_contact-5e742600a47a7 | ref | UI_contact_group | UI_contact_group | 4 | ajpulmse_crm.contact_a.id | 1 | Using where; Distinct |
| 1 | SIMPLE | civicrm_group_contact-5e742600a4a78 | ref | UI_contact_group | UI_contact_group | 4 | ajpulmse_crm.contact_a.id | 1 | Using where; Distinct |
| 1 | SIMPLE | civicrm_group_contact-5e742600a4d39 | ref | UI_contact_group | UI_contact_group | 4 | ajpulmse_crm.contact_a.id | 1 | Using where; Distinct |
| 1 | SIMPLE | civicrm_group_contact-5e742600a4fde | ref | UI_contact_group | UI_contact_group | 4 | ajpulmse_crm.contact_a.id | 1 | Using where; Distinct |
| 1 | SIMPLE | civicrm_group_contact-5e742600a529d | ref | UI_contact_group | UI_contact_group | 4 | ajpulmse_crm.contact_a.id | 1 | Using where; Distinct |
+------+-------------+-------------------------------------+------+----------------------------+----------------------------+---------+---------------------------+-------+-----------------------------------------------------------+
7 rows in set, 1 warning (0.00 sec)
~~~
After this change, note absence of the "Using filesort". Query completes in 5 seconds or less.
~~~
MariaDB [ajpulmse_crm]> explain
-> SELECT DISTINCT LEFT(contact_a.sort_name, 1) as sort_name FROM civicrm_contact contact_a LEFT JOIN civicrm_group_contact `civicrm_group_contact-5e742600a44d7` ON (contact_a.id = `civ
icrm_group_contact-5e742600a44d7`.contact_id AND `civicrm_group_contact-5e742600a44d7`.status IN ('Added')) LEFT JOIN civicrm_group_contact `civicrm_group_contact-5e742600a47a7` ON (contact
_a.id = `civicrm_group_contact-5e742600a47a7`.contact_id AND `civicrm_group_contact-5e742600a47a7`.status IN ('Added')) LEFT JOIN civicrm_group_contact `civicrm_group_contact-5e742600a4a78`
ON (contact_a.id = `civicrm_group_contact-5e742600a4a78`.contact_id AND `civicrm_group_contact-5e742600a4a78`.status IN ('Added')) LEFT JOIN civicrm_group_contact `civicrm_group_contact-5e
742600a4d39` ON (contact_a.id = `civicrm_group_contact-5e742600a4d39`.contact_id AND `civicrm_group_contact-5e742600a4d39`.status IN ('Added')) LEFT JOIN civicrm_group_contact `civicrm_grou
p_contact-5e742600a4fde` ON (contact_a.id = `civicrm_group_contact-5e742600a4fde`.contact_id AND `civicrm_group_contact-5e742600a4fde`.status IN ('Added')) LEFT JOIN civicrm_group_contact `
civicrm_group_contact-5e742600a529d` ON (contact_a.id = `civicrm_group_contact-5e742600a529d`.contact_id AND `civicrm_group_contact-5e742600a529d`.status IN ('Added')) WHERE ( ( ( ( `civicr
m_group_contact-5e742600a44d7`.group_id IN ("498") ) ) ) OR ( ( ( `civicrm_group_contact-5e742600a47a7`.group_id IN ("499") ) ) ) OR ( ( ( `civicrm_group_contact-5e742600a4a78`.group_id IN
("505") ) ) ) OR ( ( ( `civicrm_group_contact-5e742600a4d39`.group_id IN ("504") ) ) ) OR ( ( ( `civicrm_group_contact-5e742600a4fde`.group_id IN ("503") ) ) ) OR ( ( ( `civicrm_group_conta
ct-5e742600a529d`.group_id IN ("502") ) ) ) ) AND (contact_a.is_deleted = 0);
+------+-------------+-------------------------------------+------+----------------------------+----------------------------+---------+---------------------------+-------+------------------
------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra
|
+------+-------------+-------------------------------------+------+----------------------------+----------------------------+---------+---------------------------+-------+------------------
------------+
| 1 | SIMPLE | contact_a | ref | index_is_deleted_sort_name | index_is_deleted_sort_name | 1 | const | 24754 | Using index; Usin
g temporary |
| 1 | SIMPLE | civicrm_group_contact-5e742600a44d7 | ref | UI_contact_group | UI_contact_group | 4 | ajpulmse_crm.contact_a.id | 1 | Using where; Dist
inct |
| 1 | SIMPLE | civicrm_group_contact-5e742600a47a7 | ref | UI_contact_group | UI_contact_group | 4 | ajpulmse_crm.contact_a.id | 1 | Using where; Dist
inct |
| 1 | SIMPLE | civicrm_group_contact-5e742600a4a78 | ref | UI_contact_group | UI_contact_group | 4 | ajpulmse_crm.contact_a.id | 1 | Using where; Dist
inct |
| 1 | SIMPLE | civicrm_group_contact-5e742600a4d39 | ref | UI_contact_group | UI_contact_group | 4 | ajpulmse_crm.contact_a.id | 1 | Using where; Dist
inct |
| 1 | SIMPLE | civicrm_group_contact-5e742600a4fde | ref | UI_contact_group | UI_contact_group | 4 | ajpulmse_crm.contact_a.id | 1 | Using where; Dist
inct |
| 1 | SIMPLE | civicrm_group_contact-5e742600a529d | ref | UI_contact_group | UI_contact_group | 4 | ajpulmse_crm.contact_a.id | 1 | Using where; Dist
inct |
+------+-------------+-------------------------------------+------+----------------------------+----------------------------+---------+---------------------------+-------+------------------
------------+
7 rows in set (0.00 sec)
~~~
Agileware ref: CIVICRM-14575.30.0https://lab.civicrm.org/dev/core/-/issues/1653Civi::paths() - Allow bare variable lookup via getUrl/getPath2023-02-24T05:03:58ZtottenCivi::paths() - Allow bare variable lookup via getUrl/getPathOverview
----------------------------------------
The `Civi::paths()->getUrl(...)` and `Civi::paths()->getPath(...)` have a quirky interpretation of bare variable expressions (eg `getPath('[civicrm.packages]')`).
Example use-case
------...Overview
----------------------------------------
The `Civi::paths()->getUrl(...)` and `Civi::paths()->getPath(...)` have a quirky interpretation of bare variable expressions (eg `getPath('[civicrm.packages]')`).
Example use-case
----------------------------------------
```
# (A) With a file/folder/subpath
cv ev "echo Civi::paths()->getUrl('[civicrm.packages]/foo/bar');"
# (B) No slash, no dot
cv ev "echo Civi::paths()->getUrl('[civicrm.packages]');"
# (C) With a slash
cv ev "echo Civi::paths()->getUrl('[civicrm.packages]/');"
# (D) With a slash and dot
cv ev "echo Civi::paths()->getUrl('[civicrm.packages]/.');"
```
Current behavior
----------------------------------------
In scenarios (A), (C), and (D), the value of `[civicrm.packages]` is substituted. But in situation (B), it is treated as a literal file-name and given the default prefix.
```
$ cv ev "echo Civi::paths()->getUrl('[civicrm.packages]');"
http://site/[civicrm.packages]
```
Proposed behavior
----------------------------------------
All examples -- including (B) -- should do substitution.
The `Civi\Core\PathsTest` should be expanded to check scenario (B).
Comments
----------------------------------------
[Historically](https://lab.civicrm.org/dev/wordpress/issues/47#note_33046), when `getPath()`/`getUrl()` were first drafted to support expressions like `[civicrm.files]/persist/contribute`, the primary concerns were (1) backward compatibility for older absolute+relative expressions (without any `[foo]` expressions) and (2) new use-cases like (A). The `[foo]` notation was conceived as an *optional prefix* (with an *implied default*) - and not as a straight-up *variable*. And if you *just* wanted a raw variable, I'd've imagined it'd be faster to call `getVariable()` (bypass the string-munging with `getPath()` etc).
OTOH, if you learned of this API by skimming docblocks or examples (without that historical context), then it'd be natural to assume that (B) would work, and it really is a more approachable interface that way.
Strictly speaking, it is a change in the contract for an obscure edge-case: if you had a file named `/var/www/sites/default/files/civicrm/[foo]`, and if you requested `getPath('[foo]')`, then it currently resolves to the file. With this change, it would interpret `[foo]` as a variable - the variable is probably undefined, which leads to an exception. You'd have to rewrite the call as `getPath('[civicrm.files]/[foo]')`. That's an exceedingly marginal edge-case, and I don't really think it's worth preserving.https://lab.civicrm.org/dev/core/-/issues/1648Determine how to provide parity for apiv4 vs v3 Contact.delete2023-03-26T05:03:15ZeileenDetermine how to provide parity for apiv4 vs v3 Contact.deleteThere are 4 types of Contact.delete in CiviCRM
1) soft-delete - updates is_deleted to true, deletes uf match entries
2) hard delete, - deletes contact record, aborts if related entities exist that block delete, cleans up various entries...There are 4 types of Contact.delete in CiviCRM
1) soft-delete - updates is_deleted to true, deletes uf match entries
2) hard delete, - deletes contact record, aborts if related entities exist that block delete, cleans up various entries that an FK delete will not do
3) contact.update to is_deleted = 1 does 1 without the cleanup
4) contact.delete - does 2 without the cleanup or protection
Currently apiv3 does 1 or 2 depending on the site configuration. It can be 'pushed' to do 2 but will still only do that dependent on user permissions if check_permissions is on.
apiv4 does 3 and 4
I know @colemanw is reluctant to replicate v3 behaviour but I don't think current v4 is an acceptable alternative. Nor do I think the calling function should need to check site delete confighttps://lab.civicrm.org/dev/core/-/issues/1644Merge related activities not deleted when permanently deleting a contact2023-02-26T05:03:38ZeileenMerge related activities not deleted when permanently deleting a contactWhen a contact is merged into a another contact the following happens
- the data is moved over
- the merged contact is 'deleted'
- an activity is created of type 'Contact Merged' - source is logged in contact, target is the retained...When a contact is merged into a another contact the following happens
- the data is moved over
- the merged contact is 'deleted'
- an activity is created of type 'Contact Merged' - source is logged in contact, target is the retained contact
- If the merged contact is being deleted to trash an activity is created of type 'Contact Deleted by Merge' - source is logged in contact, target is deleted contact
When we later fully delete the contact deleted-by-merge only activities linked to no other contact are deleted - in other words the 'Contact Deleted by Merge' activity remains, linked to the source contact.
I believe this activity should be deleted on true-death , as it would not be present had they been fully deleted in the first instance
@pfigel @DaveDhttps://lab.civicrm.org/dev/core/-/issues/1640Update pending contribution status action also send email without warning2020-03-21T19:38:53ZjaapjansmaUpdate pending contribution status action also send email without warningWhen you search for contributions with status pending. You have an action to batch update all contribution statuses to completed. See screenshots
![Screenshot_from_2020-03-10_22-38-02](/uploads/e533c6e42f3ed4d1545f3855d3dd81b8/Screensho...When you search for contributions with status pending. You have an action to batch update all contribution statuses to completed. See screenshots
![Screenshot_from_2020-03-10_22-38-02](/uploads/e533c6e42f3ed4d1545f3855d3dd81b8/Screenshot_from_2020-03-10_22-38-02.png)
![Screenshot_from_2020-03-10_22-39-20](/uploads/d57cd774f224eb3ec62c73dc446628e7/Screenshot_from_2020-03-10_22-39-20.png)
When you do this the system also sends an email receipt to the donor.
**There is no warning about this e-mail.**
Possible solutions:
1. Checkbox for sending e-mails or not (this will give the user control)
2. A warning text indicating that this action also sends an e-mai.
I will see if I can work on option 1.jaapjansmajaapjansmahttps://lab.civicrm.org/dev/core/-/issues/1638Introduce "civi.dao.preUpdate" and "civi.dao.preInsert" events2020-05-18T17:53:27ZhaystackIntroduce "civi.dao.preUpdate" and "civi.dao.preInsert" eventsOverview
----------------------------------------
Following on from the action taken some time ago in [Add civi.dao.preDelete event](https://issues.civicrm.org/jira/browse/CRM-20458) (and the proposal raised in #161 to an extent) it woul...Overview
----------------------------------------
Following on from the action taken some time ago in [Add civi.dao.preDelete event](https://issues.civicrm.org/jira/browse/CRM-20458) (and the proposal raised in #161 to an extent) it would be helpful to apply the same `hook_civicrm_pre` + `hook_civicrm_post` pattern to the Symfony events in `CRM_Core_DAO::save()`. I propose CiviCRM introduces `civi.dao.preUpdate` and `civi.dao.preInsert` events to complement the existing `civi.dao.preDelete` event.
Symfony events have advantages over "old-style" hooks of the form `hook_civicrm_preSave`. The old-style hook format is likely to lead some devs to write callbacks in the format `extensionname_civicrm_preSave_table_name` which are difficult to unhook or prevent from running endlessly. Symfony events _can_ be unhooked and therefore make it relatively easy to avoid endless loops such as @colemanw raises concerns about in [his comment](https://lab.civicrm.org/dev/core/issues/161#note_9549). Also, they are largely undocumented and therefore avoid the effort of having to add to the Docs :-)
Example use-case
----------------------------------------
Let's look at what can be done with operations on "Option Value" data:
In both the [CiviCRM Event Organiser](https://github.com/christianwach/civicrm-event-organiser) and the [CiviCRM ACF Integration](https://github.com/christianwach/civicrm-acf-integration) plugins, I'd like to be able to inspect an Option Value _before_ it is saved to the database so that I know what the full state of the Option Value was before the update is applied. Once the edit has happened, there's (obviously) no way of retrieving the required data.
(FWIW, this is to perform actions on synced entities in WordPress rather than modifying the Option Value before it is saved, but the same applies should Option Value data need to be modified.)
Current behaviour
----------------------------------------
As far as I can tell, there isn't a hook available for me to inspect an Option Value data prior to it being created or updated programatically. I _partially_ solve this for changes made via the CiviCRM UI using the equivalent pre-post combination of `hook_civicrm_preProcess` and `hook_civicrm_postProcess`. However changes made via AJAX actions on the main "Event Type Options" listing screen can only be detected _after_ the edit has been made.
Proposed behaviour
----------------------------------------
To mirror the pattern of `hook_civicrm_pre` + `hook_civicrm_post` and `hook_civicrm_preProcess` + `hook_civicrm_postProcess` and the existing pattern of `civi.dao.preDelete` + `civi.dao.postDelete` to complete the set with:
* `civi.dao.preInsert` + `civi.dao.postInsert`
* `civi.dao.preUpdate` + `civi.dao.postUpdate`
Comments
----------------------------------------
PR to follow.5.26.0haystackhaystackhttps://lab.civicrm.org/dev/core/-/issues/1634Evaluate if any indexed fields are unused2022-12-04T23:07:09ZeileenEvaluate if any indexed fields are unused
**Proposal** (note this is being updated based on discussion & comments may refer to an earlier version)
1. Remove the following columns from the xml from civicrm_activity
* phone_id
* phone_number
* relationship_id
1. Remove th...
**Proposal** (note this is being updated based on discussion & comments may refer to an earlier version)
1. Remove the following columns from the xml from civicrm_activity
* phone_id
* phone_number
* relationship_id
1. Remove the index from the xml on
* medium_id
* is_deleted
1. During upgrade we drop the above columns, if empty.
**Follow ups to consider**
2. There are other columns in the civicrm_activity table that are case specific - we might consider indexing is_current_revision & original_id only when CiviCase is enabled
3. I've been doing some tests on searches and found that searching is faster if I DROP the contribution_status_id index - it might be interesting to test the activity_type_id index although I suspect it has a much greater cardinality & is more useful
**Impact of the above**
1. Data would not be lost but api fields would no longer access those fields
2. Developers who might be using them outside of core could be impacted - we can mitigate by communicating on the dev list & perhaps putting checks & deprecation notices onto sites with data in the fields for a few months before making any changes.
3. DB size would be reduced. Note that empty fields contribute notably to table size IF they are indexed
4. Dev confusion & efficiency is improved by not having unused stuff in core.
**Background**
Obviously that's not something we should rush into so I'll have to ping the dev list etc
Looking at our civicrm_activity table it appears that each index has a base size - of around a half a gig. From there, the index size increases based on how much data is in the table. So an index on an empty field is around 57% of the size of our largest index.
There are 5 fields that are indexed + empty in our database for the civicrm_activity table (
```
Original id used for CiviCase
Medium id
Phone id
relationship_id
is_deleted
```
Plus - is_current_revision is effectively null
So my first question is are these all used in other databases - e.g when civicase is in use.
I couldn't spot references to phone_id and it feels 'wrong' to me anyway as I think you would want to either link to the contact or have a hard reference. I wonder if some of these fields are quietly obsolete?
It's very unsafe to drop core fields. However, I'm pondering dropping the indexes on these fields
@DaveD I'd appreciate your thoughts....https://lab.civicrm.org/dev/core/-/issues/1633Proposal - add an optional access_arguments key to setting spec2023-02-22T05:03:56ZeileenProposal - add an optional access_arguments key to setting specCurrently when calling Setting.get or Setting.getoptions you need 'administer CiviCRM' permission
I recently found that an angular form was broken for users without this & digging into it I'm using the hook as a short term solution...Currently when calling Setting.get or Setting.getoptions you need 'administer CiviCRM' permission
I recently found that an angular form was broken for users without this & digging into it I'm using the hook as a short term solution - ie
```
if ($entity === 'setting' &&
(($action === 'get' && isset($params['return']) && $params['return'] === 'deduper_equivalent_name_handling')
|| ($action === 'getoptions' && $params['field'] === 'deduper_equivalent_name_handling'))
) {
$permissions['setting']['get'] = [['merge duplicate contacts', 'administer CiviCRM']];
}
```
But this relies on the setting being accessed this way - so it's very brittle. I'm not sold on just lowering the whole permission level.
My feeling is that we should extend the metadata spec - ie add optional key of
'access_arguments' => ['default' => 'be amazing', 'get' => 'be fair to middling']
- this would mean only amazing people can create the setting (when check_permissions are enabled) but fair to middling people could get the setting.
In all cases the default is still 'Administer CiviCRM' if nothing is set
@colemanw @totten @seamuslee @mattwire