CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-06-17T17:35:14Zhttps://lab.civicrm.org/dev/core/-/issues/1799Daily (and other) scheduled jobs slip forward each day until they're eventual...2023-06-17T17:35:14ZJonGoldDaily (and other) scheduled jobs slip forward each day until they're eventually missedIf this subject line looks familiar, [there's a reason](https://issues.civicrm.org/jira/browse/CRM-16276).
This issue was originally raised and [fixed in April 2015](https://github.com/civicrm/civicrm-core/commit/d0be1535d809b206890bcd4...If this subject line looks familiar, [there's a reason](https://issues.civicrm.org/jira/browse/CRM-16276).
This issue was originally raised and [fixed in April 2015](https://github.com/civicrm/civicrm-core/commit/d0be1535d809b206890bcd4f72ebd4ba8fb1c86d). However, a code simplification in December 2015 [removed the fix](https://github.com/civicrm/civicrm-core/commit/bda41fcbeaf535ff04d7e5cfc7145e83f50c17b2).
Many of us are familiar with this issue. Cron runs at 10:00:00, scheduled jobs are executed in order, and one job might not fire until 10:00:03. If the job fires hourly, we would expect the job to fire at 11:00:00, but since an hour hasn't elapsed since 10:00:00, the job doesn't fire until the *next* cron, maybe at 11:15:00. This has repercussions for a number of scheduled job use cases.
After some thought, I realized the correct solution is to record the time the *cron job* fires, not the time the job actually is started. I believe that when the `last_run` field was created, there was no sense that these times might be different, hence the bug.
I wrote a patch (I'm leaving it in production for a few days before submitting a PR) that does two things:
* `last_run` now refers to the time the cron job started, not the scheduled job.
* To avoid race-y conditions on very slow shared hosts, I force the `last_run` time to always have "00" as its seconds. This is a belt-and-suspenders (or belt-and-braces in non-'Merican) approach that reflects that a) shared hosting could be slow enough that this would otherwise be a problem, b) cron doesn't allow scheduled at the seconds level, so this shouldn't impact anyone's scheduling.JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/3152Data stored in universal time does not handle DST consistently2023-11-30T05:00:20ZtottenData stored in universal time does not handle DST consistently[[_TOC_]]
Overview
----------------------------------------
CiviCRM has a number of `TIMESTAMP` columns -- these are stored in universal time (UTC) and displayed in the user's timezone. However, there is a subtle error in handling Dayl...[[_TOC_]]
Overview
----------------------------------------
CiviCRM has a number of `TIMESTAMP` columns -- these are stored in universal time (UTC) and displayed in the user's timezone. However, there is a subtle error in handling Daylight Savings Time (DST): if the current-date and the target-date sit on different sides of the DST-switch, then the time may present as +/- 1 hour.
This bug was one of the major subissues identified in https://lab.civicrm.org/dev/core/-/issues/2122. Although that particular feature was rolled-back/deferred from 5.47, the DST bug still exists -- it's just less obvious.
Example use-case
----------------------------------------
1. Create a contact record.
2. View the contact record. Note the creation time (`civicrm_contact.created_date`).
3. Change the system clock - set to a date where DST differs (eg if today is March 30, then go to December 5).
4. View the contact record. Note the creation time (`civicrm_contact.created_date`).
Current behavior
----------------------------------------
The displayed value of `civicrm_contact.created_date` _appears_ to change by +/- 1 hour, depending on when you view it.
> (Viewed on Mar 31, 2022)
>
> ![Screen_Shot_2022-03-31_at_12.40.53_AM](/uploads/8a2f5a0e4fb6ec884aede78e30d31b1a/Screen_Shot_2022-03-31_at_12.40.53_AM.png)
> (Viewed on Dec 5, 2022)
>
> ![Screen_Shot_2022-12-05_at_12.44.57_AM](/uploads/2df387488ee029a949010da163bf2ea8/Screen_Shot_2022-12-05_at_12.44.57_AM.png)
Why? CiviCRM sends a note to MySQL about the current user's timezone (`SET time_zone = '...'`). However, it doesn't identify the timezone effectively. It gives [the current numeric offset (at the moment of viewing)](https://github.com/civicrm/civicrm-core/blob/5.47.3/CRM/Utils/System/Base.php#L758-L762) - but (in locales with DST) the offsets fluctuate over time.
(_Ex: On Mar 31, the offset in California is `-0700`. Under current/long-standing law, the offset will be `-0800` on Dec 5. Of course, the US Congress is reconsidering this law... so we don't really know what the offset will be!_)
Proposed behavior 1: Fix MySQL timezones
----------------------------------------
CiviCRM should send the timezone as a symbolic name, such as `Europe/Helsinki`, `America/Los_Angeles`, or `Australia/Sydney`. These symbolic-names have an underlying database which allows them adjust automatically based on DST-rules/target-dates/current-law. On the surface, the fix is extremely simple:
```diff
diff --git a/CRM/Utils/System/Base.php b/CRM/Utils/System/Base.php
index a4660834c5..8e40f6da35 100644
--- a/CRM/Utils/System/Base.php
+++ b/CRM/Utils/System/Base.php
@@ -755,10 +755,9 @@ abstract class CRM_Utils_System_Base {
* Set timezone in mysql so that timestamp fields show the correct time.
*/
public function setMySQLTimeZone() {
- $timeZoneOffset = $this->getTimeZoneOffset();
- if ($timeZoneOffset) {
- $sql = "SET time_zone = '$timeZoneOffset'";
- CRM_Core_DAO::executequery($sql);
+ $timeZone = $this->getTimeZoneString();
+ if ($timeZone) {
+ CRM_Core_DAO::executequery('SET time_zone = %1', [1 => [$timeZone, 'String']]);
}
}
```
There are a couple of catches.
* __Timezone rules change__ (occasionally). Any software that supports timezones ultimately needs a _data feed_ with current rules. The good news: IANA publishes a free/open feed (https://www.iana.org/time-zones; aka `tzdata`; aka `zoneinfo`), most Linux/Unix distros have this feed, and MySQL can read it (`mysql_tzinfo_to_sql`). It usually requires one command (which could run during system-config, system-startup, and/or cron):
```bash
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql mysql
```
The problem: we have no measures for (a) how many CiviCRM deployments actually subscribe to this feed and (b) how many could subscribe, if they chose to.
* __Timezone names may be inconsistent__ (occasionally). For example, in different contexts, it's been fashionable to refer to California's timezone as `America/Los_Angeles`, `US/Pacific`, and `PST8PDT`. (The current+official fashion is `America/Los_Angeles` - the others are deprecated.) However, since Civi integrates with various layers (different CMSs; PHP APIs; MySQL APIs), there are edge-case where the layers may choose different names. (*I'm not super-concerned, but we should raise sensible warnings when names are invalid or mismatched.*)
The central issue is - how to cope when data isn't available? This comes to mind:
* (Status check) If the active TZ (`getTimeZoneString()`) has a deprecated name (eg `PST8PDT`) or an offset (eg `-0700`), show a warning.
* (Status check) If the active TZ (`getTimeZoneString()`) isn't supported by MySQL, show a warning.
* (Runtime) If the active TZ (`getTimeZoneString()`) isn't supported by MySQL, fallback to sending offset.
Proposed behavior 2: Change format. Use only PHP TZs.
----------------------------------------
(_This expands on one of @haystack's suggestions in dev/core#2122._)
If you assume that MySQL time services aren't available - what else would you do? You could use PHP time services.
The astute observer will note the status-quo (using both PHP and MySQL time-services) creates two points-of-failure. If either PHP _or_ MySQL has bad/incomplete/old timezone data, then you'll get mis-calculations _somewhere_. Consolidating on PHP time-services would reduce the #dependencies.
Both Drupal and WordPress take this approach. (I suspect this is extremely useful for maximizing compatibility with heterogeneous web-hosts.) They each do it a bit differently, but some central concepts are the same:
* In Drupal, PHP processes read+write temporal data in universal time -- as an `INT` (Unix-style, seconds-since-epoch).
* In WordPress, PHP processes read+write temporal data in universal time -- as a `DATETIME` with a `_gmt` suffix (eg `post_modified_gmt`).
* Hypothetically, you could hardcode MySQL to `SET time_zone='+0:00'`. PHP processes would read+write temporal data in universal time -- as a `TIMESTAMP`.
In all those cases, the onus is on the PHP devs to convert to/from universal-time when implementing functionality (eg "find records from March 1 - March 15" or "find records from this afternoon" or "extract the hour:minute component").
But there is a catch here: Civi already relies on several MySQL time-services. The schema works a certain way; the reports/searches/UIs/APIs expect the schema to work a certain way; etc.
The central issue is - how do you manage/QA all the changes (in schema+logic) required to change time-service?
Comments
----------------------------------------
* I haven't tested, but I'm fairly certain there will be another manifestation in CiviMail scheduling. Ex:
* You live in a timezone where DST changes on March 16.
* On March 10, you schedule a mail-blast for 2:00pm on March 20. It stores the schedule with the wrong offset.
* When March 20 comes, the mailing actually goes out at 3:00pm (or maybe 1:00pm).https://lab.civicrm.org/dev/core/-/issues/4946DB error on gender searches2024-02-01T16:06:14ZyashodhaDB error on gender searchesWe should NOT allow non numeric values for gender option values.
gender_id in the database expects numeric values instead, we should form rule for such entities.We should NOT allow non numeric values for gender option values.
gender_id in the database expects numeric values instead, we should form rule for such entities.yashodhayashodhahttps://lab.civicrm.org/dev/core/-/issues/1486Deadlocks on acl_cache2023-09-12T18:50:06ZeileenDeadlocks on acl_cacheOne of the prevalent deadlocks we see is on the acl_contact_cache. I've found that I can replicate this locally fairly consistently just by running 2 sets of tests at once.
The thing that is somewhat odd about this is that we don't use ...One of the prevalent deadlocks we see is on the acl_contact_cache. I've found that I can replicate this locally fairly consistently just by running 2 sets of tests at once.
The thing that is somewhat odd about this is that we don't use ACLs at all so in fact the table is always empty. Notably the table is also extremely slow to truncate - despite being empty.
**What happens**
The flow we have is
1) api call to create a contact
2) when the contact is created CRM_Contact_BAO_Contact_Utils::clearContactCaches() is called. It hits a deadlock because
a) it has an FK to contact.id (& in our case it has an FK and an index seemingly)
b) it is called a lot
3) it retries and succeeds
4) we then try to add an email - the add function fails on a foreign key constraint because the earlier deadlock rolled the contact create back
**Proposals**
I think there are a number of things we can do & we should do some combo:
1) Make the delete function less expensive & less locky
a) Add an index on modified date - we are constantly deleting where modified_date < x so let's index it
b) remove the index on acl_id - we ALSO have an FK so having both is probably harmless-but-confusing
c) alter the FK on contact ID to be an index not a foreign key. This is perhaps a confusing suggestion but it is also the one that will make a real dent in these deadlocks. Remember this is a cache table not a 'persistent' table. The impact of the rows outliving the existence of the contact for a little bit is basically none because we don't do ACL lookups on non-existent contacts (this is also potentially helpful for better handling 0 or anonymous contact_id where we've had to work around this before
2) Protect against conflict through mysql locks. I think we can use a mysql lock to avoid 2 processes attempting this flush at once. I'll do a PR for this
3) How often do we REALLY need to flush those caches. Maybe we could cache a 'last flushed' key & flush no more than every 30 seconds.
4) Consider checking if the table is empty before attempting to empty it.
5) Review the places where acl cache is called from & decide how necessary there are - here is the list:
**CRM_ACL_BAO_Cache::resetCache**
CRM/Contact/BAO/Contact/Utils.php: CRM_ACL_BAO_Cache::resetCache();
CRM/Core/BAO/Cache.php: CRM_ACL_BAO_Cache::resetCache();
CRM/Utils/System.php: CRM_ACL_BAO_Cache::resetCache();
CRM/ACL/Page/EntityRole.php: CRM_ACL_BAO_Cache::resetCache();
CRM/ACL/Form/ACLBasic.php: CRM_ACL_BAO_Cache::resetCache();
CRM/ACL/Form/EntityRole.php: CRM_ACL_BAO_Cache::resetCache();
CRM/ACL/BAO/ACL.php: CRM_ACL_BAO_Cache::resetCache();
**CRM_Contact_BAO_Contact_Utils::clearContactCaches()**
CRM/Contact/BAO/Contact/Utils.php: public static function clearContactCaches($isEmptyPrevNextTable = FALSE) {
CRM/Contact/BAO/GroupContact.php: CRM_Contact_BAO_Contact_Utils::clearContactCaches();
CRM/Contact/BAO/GroupContact.php: CRM_Contact_BAO_Contact_Utils::clearContactCaches();
CRM/Contact/BAO/Contact.php: CRM_Contact_BAO_Contact_Utils::clearContactCaches();
CRM/Contact/Import/Form/Preview.php: CRM_Contact_BAO_Contact_Utils::clearContactCaches(TRUE);
bin/cli.class.php: CRM_Contact_BAO_Contact_Utils::clearContactCaches();
6) A bigger task perhaps - but permit opportunistic flushing to be turned off in favour of a cron (per group contact cache)
**Deadlock output**
LATEST DETECTED DEADLOCK
------------------------
2019-12-19 11:29:45 0x700010cd2000
*** (1) TRANSACTION:
TRANSACTION 34518715, ACTIVE 1 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 11 lock struct(s), heap size 1136, 4 row lock(s), undo log entries 7
MySQL thread id 571, OS thread handle 123145583353856, query id 709677 localhost 127.0.0.1 drupalcivi_lkznr updating
DELETE
FROM civicrm_acl_cache
WHERE modified_date IS NULL
OR (modified_date <= '20191218222444')
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 143182 page no 3 n bits 80 index PRIMARY of table `drupalcivi_lkznr`.`civicrm_acl_cache` trx id 34518715 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 6; compact format; info bits 32
0: len 4; hex 00003c14; asc < ;;
1: len 6; hex 0000020eb6ba; asc ;;
2: len 7; hex 3b000001442ac5; asc ; D* ;;
3: SQL NULL;
4: len 4; hex 00000002; asc ;;
5: SQL NULL;
*** (2) TRANSACTION:
TRANSACTION 34518714, ACTIVE 1 sec starting index read
mysql tables in use 2, locked 2
66 lock struct(s), heap size 8400, 71 row lock(s), undo log entries 74
MySQL thread id 573, OS thread handle 123145584189440, query id 710022 localhost 127.0.0.1 drupalcivi_lkznr updating
UPDATE civicrm_contact SET contact_type = 'Organization' , contact_sub_type = NULL , email_greeting_id = NULL , postal_greeting_id = NULL , addressee_id = 3 , employer_id = 15055 WHERE ( civicrm_contact.id = 15055 )
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 143182 page no 3 n bits 80 index PRIMARY of table `drupalcivi_lkznr`.`civicrm_acl_cache` trx id 34518714 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;seamusleeseamusleehttps://lab.civicrm.org/dev/core/-/issues/3148Dedupe with multi-select custom fields can trigger IDS2023-03-18T04:20:40ZJonGoldDedupe with multi-select custom fields can trigger IDSWhen deduping contacts that have multi-select custom fields, and selecting to move the custom fields to the new contact, the IDS is triggered.
### Steps to replicate
* Create a custom field that allows saving multiple values (e.g. a che...When deduping contacts that have multi-select custom fields, and selecting to move the custom fields to the new contact, the IDS is triggered.
### Steps to replicate
* Create a custom field that allows saving multiple values (e.g. a checkbox). Note that you need several of these to trigger the "kick" on the IDS (3, I think).
* Create two contacts that are duplicates.
* Find and merge the records.
### Expected result
Contacts merged successfully.
### Actual result
"Your activity is a bit suspicious, hence aborting"
The issue is the POST request, which is passing arguments like `move_custom_12` with the `VALUE_SEPARATOR` control character. This triggers the IDS filter labeled "Detects nullbytes and other dangerous characters".
I'm really not certain what the correct answer is here - I can exempt users from the IDS, and maybe that's the solution to pursue, but it seems like there should be another solution available. Is it possible to exempt certain paths from the IDS, or use an alternate set of rules for a certain path?
Keyword: Intrusion Detection Systemhttps://lab.civicrm.org/dev/core/-/issues/4581Define re-usable idiom for deferrable upgrade steps2023-09-19T13:05:13ZtottenDefine re-usable idiom for deferrable upgrade stepsOverview
----------------------------------------
In managing upgrade-steps, there is some tension between:
* Making the upgrades automatic for typical sysadmins
* Allowing very large deployments to schedule/manage expensive upgrade-st...Overview
----------------------------------------
In managing upgrade-steps, there is some tension between:
* Making the upgrades automatic for typical sysadmins
* Allowing very large deployments to schedule/manage expensive upgrade-steps
This proposes to balance those expectations by defining a _deferrable upgrade task_. Sysadmins can choose to approach them a few ways:
* "Just run the task like every other upgrade step" (*e.g. default; for average and small systems*)
* "Let me decide when to run it... but please don't make me figure out obscure incantations" (*for larger systems*)
* "Let me find a highly-optimized/bespoke way to accomplish the change" (*for extra-large systems with a team of admins*)
Example use-case
----------------------------------------
1. Take a table (like `civicrm_contact`, `civicrm_activity`, `civicrm_mailing_event_queue`) which can be very large on some deployments.
1. Define a schema change (e.g. add a column and an index)
1. Executing this change be quite slow.
1. In the long-term, all systems need the schema-change. If people skip it, then it will lead to problems/confusion/bug-reports/etc.
1. In the near-future, the schema change is not quite mandatory. (The new column isn't actively used; or its usages are limited+guarded.)
A sketch
----------------------------------------
* Designate a place to store a list of deferred upgrade steps (e.g. `civicrm_deferred_upgrade` or `civicrm_queue_item`)
* In the `CRM/Upgrade/` system, add some helpers to conditionally execute or defer a step. Ex:
```php
$this->addDeferrableTask(ts('Update CiviMail tracking'), 'doCiviMailQueueConversion');
```
```php
function addDeferrableTask($title, $funcName, $args) {
if (deferred upgrades enabled) {
// Add $title, $action, $args to a persistent TODO list
}
else {
$this->addTask($title, $funcName, $args);
}
}
```
* Add a system-status-check and post-upgrade message to warn if there are any of these things that need to run
* Add an API/job/command/UI to list/execute/skip/delete deferred stepshttps://lab.civicrm.org/dev/core/-/issues/4636Deleting group invalidates ACL2023-10-05T11:56:00Zaydunsaidan.saunders@squiffle.ukDeleting group invalidates ACLOverview
----------------------------------------
Groups used in ACL's can be deleted without warning resulting in invalid ACL's.
Setup
----------------------------------------
1. Create a group: TestGroup
1. Create an ACL: Role: Every...Overview
----------------------------------------
Groups used in ACL's can be deleted without warning resulting in invalid ACL's.
Setup
----------------------------------------
1. Create a group: TestGroup
1. Create an ACL: Role: Everyone, Operation: View, Type: Group, Group: TestGroup (just created)
1. Note that the ACL display will shows `Type` as 'Group' and shows the group name
1. In SQL, look at the `civicrm_acl` table and note the `object_table` is 'civicrm_group', and `object_id` is the new group id.
Now delete the TestGroup:
- There is no warning that this group is used in an ACL
- In the ACL display the group name is now blank - a situation which cannot be created through the Add or Edit ACL screens.
- In SQL, the `civicrm_acl` table still show the `object_id` as the now non-existent group id.
Prior to https://github.com/civicrm/civicrm-core/pull/27679 this causes DB Syntax Errors when calling `CRM_Contact_BAO_Contact_Permission::allow()`
Expected behaviour
----------------------------------------
Good question! Maybe a warning that the group is being used in an ACL, but what should then happen to the ACL? Disable it?
Environment information
----------------------------------------
* __CiviCRM:__ _Master but may be longstanding_ <!-- If this problem relates to an upgrade, then specify both old and new versions -->https://lab.civicrm.org/dev/core/-/issues/3586Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process ...2023-09-12T02:13:33Zjustinfreeman (Agileware)Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process any emails if a single email is sent to the bounce mailbox with an invalid returnPathCiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service ema...CiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service email" causing all other emails to remain unprocessed. CiviCRM mail reports are therefore incorrect and ultimately the CiviCRM Fetch Bounces scheduled job fails to perform the job it was designed to do.
Example invalid returnPaths as observed in production:
```
x2108-arfsfufwyfgkvni2x7ww5wvac3eifc2wx3vr1fcwbbysieqppis4ytnyip09xcohck9s1vrm77nbg4xh431tthdsmtmfvfpa@rspf.mxthunderstruck.net
```
```
SRS0=3Ypr33=QM=crm.nothappyjohn.org.au=bounce+b.25211.6039846.eb7e7e8ba282f9d0@airquotes.com.au
```
This problem has been highlighted by the introduction of "Scheduled Job Failures" feature which raises the visibility of this type of problem.
Bounce mailboxes have been observed with 10k to 50k unprocessed emails, depending on the time when the "Denial of service email" was received.
Agileware Ref: CIVICRM-1970https://lab.civicrm.org/dev/core/-/issues/3584Denial of service - CiviCRM Mailing Job, if there are 5 errors sequentially w...2023-11-09T13:25:31Zjustinfreeman (Agileware)Denial of service - CiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and mailing will not sendCiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and the mailing will not send.
An invalid email domain returns an SMTP error 451: _SMTP error 451 Unable to ...CiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and the mailing will not send.
An invalid email domain returns an SMTP error 451: _SMTP error 451 Unable to complete command, DNS not available or timed out 451 Domain of sender address does not resolve_
When the CiviCRM Mailing Job executes again, the process repeats, the same invalid email addresses are used, the same errors are returned and the mailing again will not send.
This process will repeat until someone intervenes, cancelling the mailing, locating and removing the contacts which have the invalid email addresses.
The CiviCRM Mailing Job does not currently check the actual SMTP error code and just treats all SMTP errors as "SMTP Connection Errors". Counting up to 5 and then aborting the job when the threshold is reached.
I think it's fair to call this a "denial of service" because a bad actor could sign up a bunch of fake Contacts to a CiviCRM mailing list with either invalid email domains or valid email domains which are then rendered invalid - and thus cause that mailing list to cease processing.
**Possible solutions**
Change the CiviCRM Mailing Job to check the return SMTP error code and only increment the count for valid "SMTP Connection Errors".
Change the CiviCRM Mailing Job to continue with the mailing, regardless of the 5 errors encountered, which will also skip sending the emails to those Contacts with an invalid email domain. As shown in the patch below.
```
Index: CRM/Mailing/BAO/MailingJob.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/CRM/Mailing/BAO/MailingJob.php b/CRM/Mailing/BAO/MailingJob.php
--- a/CRM/Mailing/BAO/MailingJob.php (revision 20a0376709a4616e32689a821a7e95ca255ed85f)
+++ b/CRM/Mailing/BAO/MailingJob.php (date 1620794367592)
@@ -672,20 +672,9 @@
if ($smtpConnectionErrors <= 5) {
$mailer->disconnect();
$retryGroup = TRUE;
+ CRM_Core_Error::debug_log_message("More than 5 consecutive SMTP Socket Errors. Re-starting mailer.");
continue;
}
-
- // seems like we have too many of them in a row, we should
- // write stuff to disk and abort the cron job
- $this->writeToDB(
- $deliveredParams,
- $targetParams,
- $mailing,
- $job_date
- );
-
- CRM_Core_Error::debug_log_message("Too many SMTP Socket Errors. Exiting");
- CRM_Utils_System::civiExit();
}
// Register the bounce event.
Index: ext/flexmailer/src/Listener/DefaultSender.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/ext/flexmailer/src/Listener/DefaultSender.php b/ext/flexmailer/src/Listener/DefaultSender.php
--- a/ext/flexmailer/src/Listener/DefaultSender.php (revision 20a0376709a4616e32689a821a7e95ca255ed85f)
+++ b/ext/flexmailer/src/Listener/DefaultSender.php (date 1620794390628)
@@ -76,15 +76,9 @@
if ($smtpConnectionErrors <= 5) {
$mailer->disconnect();
$retryBatch = TRUE;
+ \CRM_Core_Error::debug_log_message("More than 5 consecutive SMTP Socket Errors. Re-starting mailer.");
continue;
}
-
- // seems like we have too many of them in a row, we should
- // write stuff to disk and abort the cron job
- $job->writeToDB($deliveredParams, $targetParams, $mailing, $job_date);
-
- \CRM_Core_Error::debug_log_message("Too many SMTP Socket Errors. Exiting");
- \CRM_Utils_System::civiExit();
}
else {
$this->recordBounce($job, $task, $result->getMessage());
```
Agileware Ref: CIVICRM-1728https://lab.civicrm.org/dev/core/-/issues/1516Discussion ticket for handling multivalued tokens in pdfs/emails2023-09-25T14:58:11ZDaveDDiscussion ticket for handling multivalued tokens in pdfs/emailsIn particular related to:
* https://github.com/civicrm/civicrm-core/pull/12012
* and its rebased version https://github.com/civicrm/civicrm-core/pull/16200
* https://github.com/civicrm/civicrm-core/pull/16208
There are a number of way...In particular related to:
* https://github.com/civicrm/civicrm-core/pull/12012
* and its rebased version https://github.com/civicrm/civicrm-core/pull/16200
* https://github.com/civicrm/civicrm-core/pull/16208
There are a number of ways to deal with multivalued fields when you need to produce scalar output based on the field.
For example:
* Take the "first" one.
* This has the disadvantage that "first" doesn't really mean anything, especially if the order is not guaranteed to be the same each time.
* Take the "last" one.
* Ditto.
* Combine them separated by commas.
* This has the disadvantage that it may not make any sense or might look silly. Consider something like the "With Contact" on an activity. Suppose there are 3. Suppose you want to include the primary phone of the contact. Your pdf might look like this. It's not terrible, but now think about adding in address too.
* With Contact: John Smith, Mary Brown, Indiana Jones
* Phone: 555-123-2222, , 555-383-8383
* And it's not clear to me that it can guarantee that the phone values are in the same order as the names when the tokens are separate.
* Do as in https://github.com/civicrm/civicrm-core/pull/12012, where you give the template designer the option to specify a numeric suffix on the token, e.g. {activity.target_2_display_name}. The way it's implemented there has some limitations:
* The order is still arbitrary - what does "2" actually mean?
* The template designer doesn't know max(N), so they can't easily do something like concatenate all of them if they want to.
* Provide some shorthand tokens for some common uses, e.g.
* {activity.target_1_phone} means arbitrarily take the first one.
* {activity.target_COMBINE_phone} means combine all with commas.
* {activity.target_ALL_phone} means give an array and I'll do some smarty-based looping in my template to get what I want.
Further, there are some situations where you might not want some recipients to see some of the other values. An example might be activity.case_id as in [16208](https://github.com/civicrm/civicrm-core/pull/16208). There maybe it makes sense to have a token that means "concatenate all the cases where the recipient has a role, and leave out the others". A similar one might apply for activity.target_display_name, where it might mean "concatenate all the with contact display names for activities where the recipient is one of the activity_contacts.
A related note that's just food for thought: In my other life, I always try to set up 2 fields when somebody asks for something multivalued. For example "services performed". Somebody might come to you asking for a couple of the different types of service you offer. You can designate one as the primary service, which is a single-valued field, which then makes reporting work because you don't run into double-counting, and then have a secondary services field that is multi-valued, which you don't use for reporting, and in emails/pdfs you can concatenate it without it being arbitrary because you would also output the primary which is single-valued. That may not be useful right now, but this is a discussion ticket.https://lab.civicrm.org/dev/core/-/issues/5068Do we need the qfkey in group urls?2024-03-07T13:49:27ZAndrew WestDo we need the qfkey in group urls?My users are forever emailing each other broken links to Civi groups, due to the qfkey in the URL. This doesn't seem to be needed in most of the rest of Civi - is it worth my investigating how to remove it? Or would we aim to replace tha...My users are forever emailing each other broken links to Civi groups, due to the qfkey in the URL. This doesn't seem to be needed in most of the rest of Civi - is it worth my investigating how to remove it? Or would we aim to replace that page with a search kit version soon anyway?https://lab.civicrm.org/dev/core/-/issues/3103Document contract for alterMailParams2023-11-23T17:28:30ZDaveDDocument contract for alterMailParamsThe data in $params for alterMailParams is inconsistent, and with recent token changes has changed in some cases.
Copied from https://github.com/civicrm/civicrm-core/pull/22878, which is specific to message template emails (there are al...The data in $params for alterMailParams is inconsistent, and with recent token changes has changed in some cases.
Copied from https://github.com/civicrm/civicrm-core/pull/22878, which is specific to message template emails (there are also other types of emails that trigger alterMailParams):
> Quote
* Add a class CRM_Event_WorkflowMessage_OfflineReceipt. This would look a lot like CRM_Contribute_WorkflowMessage_ContributionOfflineReceipt which describes contribution_offline_receipt. (Also, there's some draft docs in https://lab.civicrm.org/documentation/docs/dev/-/merge_requests/987.)
* Add a variant of hook_alterMailParams which gives the message as an object. Here's a sketch for firing+consuming hook_alterWorkflowMessage [gist](https://gist.github.com/totten/14178dabea79ea2a5a639a5b928262e2).https://lab.civicrm.org/dev/core/-/issues/2966Duplicate contacts when custom field value is 'email', 'phone', etc.2023-03-18T04:08:01Zrita_compucorpDuplicate contacts when custom field value is 'email', 'phone', etc.**Issue**:
when existing contacts are donating (without being logged in) with the same details (email, first name, last name) that are registered in the system, most of the times there is a new contact created instead of merging it to t...**Issue**:
when existing contacts are donating (without being logged in) with the same details (email, first name, last name) that are registered in the system, most of the times there is a new contact created instead of merging it to the existing contact automatically.
**Investigation**:
When a custom field with a checkbox had its title and value set to ‘email’ or 'phone' and the unsupervised dedupe rule was prepared to check for email or phone as well causing the problem. So the problem occurs when:
- the unsupervised dedupe rule has email in it AND
- the custom field’s value is set to ‘email’
However not only the email option can cause duplications, but if you prepare an:
- unsupervised dedupe rule with phone ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24](/uploads/758592a6d5330c1133dcd1206e714daa/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24.png) AND
- the custom field’s value is set to ‘phone’ ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12](/uploads/df40ee69692fda55f51e946b2c841b9d/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12.png)
And even though the clients enter the same email/phone numbers on the donation form, the dedupe rule will be skipped and a new contact with the same information will be created. Probably this can be reproduced with many fields in the system, but only tried these two.
I checked on a few **different civi versions:**
- 4.7.26 - couldn’t reproduce
- 5.35.2 - issue reproducible
- 5.45.alpha1 (core demo civi) - issue reproducible
**Reproduction steps:**
1. create an unsupervised individual rule with the following details: ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24](/uploads/758592a6d5330c1133dcd1206e714daa/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24.png)
2. create a custom field set for Contacts with checkboxes like this: ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12](/uploads/df40ee69692fda55f51e946b2c841b9d/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12.png)
3. add the custom field set that you created in previous step to a profile that can be added to a contribution page
4. create a new (or use an old) contribution page and add the profile that you created in previous step to it in the Profiles tab
5. create a new contact, and make sure it has a First name, Last name and Phone number
6. open the contribution page as anonymous and donate to it, using the exact same First name, Last name and Phone number that you entered in previous step AND select the ‘phone’ option at the custom field you created → _after you submit the donation, a new contact will be created instead of merging it to the existing one, which is wrong behaviour_
7. donate again on the same form, with the same details as before, but instead of selecting ‘phone’ select either the ‘email’ or ‘post’ options → _the contribution will be merged to the original contact, which is the correct behaviour. It is because email and post are not added to the unsupervised dedupe rule in this case._
I think most of the time civi admins create custom fields with values set as numbers, but in some cases they might configure it to be the same as the label name and in that case duplications might occur. I think it might be happening when the value of the custom field is the same as the name of the field in the database (just a thought, can't say for sure).https://lab.civicrm.org/dev/core/-/issues/4457Entity Reference Custom Field not working as filter in "regular" search2023-10-03T21:46:14ZjensschuppeEntity Reference Custom Field not working as filter in "regular" searchFollow-up to #3721. @fabian_SYSTOPIA found out that filtering doesn't work correctly.
Not sure this is to be fixed, as it's related to the "old-style" searches ...
## Steps to reproduce
* Create an Entity Reference Custom Field on the...Follow-up to #3721. @fabian_SYSTOPIA found out that filtering doesn't work correctly.
Not sure this is to be fixed, as it's related to the "old-style" searches ...
## Steps to reproduce
* Create an Entity Reference Custom Field on the *Participant* entity for referencing an *Event* entity
* Edit a participant and fill out the field by selecting an event
* Use the *Find Participants* search and try to limit the search result to participants with the event previously selected in that field
* Notice that all participants are being found, not only that one with the entity reference field being filledhttps://lab.civicrm.org/dev/core/-/issues/4528Entity Reference custom fields: Configurable delete behavior2024-01-17T22:28:23ZjensschuppeEntity Reference custom fields: Configurable delete behavior## Overview
Currently, when entities referenced in a custom field of the type _Entity Reference_ are deleted, the entity reference field gets cleared (set to `NULL`). This is easy and prevents us from database inconsistencies. But this ...## Overview
Currently, when entities referenced in a custom field of the type _Entity Reference_ are deleted, the entity reference field gets cleared (set to `NULL`). This is easy and prevents us from database inconsistencies. But this can lead to problems:
* inconsistencies when the entity reference field is set to be required
* stale/orphaned entities the entity reference field is attached to
* `civicrm_value_*` records with all-empty columns (or are they being taken care of already)
## Example use-case
Imagine a custom entity named _Qualification_ with (among others) an entity reference custom field referencing a _Contact_ entity, which is set to be required; read: A _Contact_ has multiple _Qualifications_.
## Current behaviour
When deleting the contact, the _Qualification_ entity will not be deleted until done so manually. The reference is just being set to `NULL` instead. Those entities will be meaningless without a referenced contact, the required entity reference custom field will be empty, which breaks the semantic contract.
Question: Is the `SET NULL` behavior a FK constraint on the DB table or part of some logic in Core code?
## Proposed behaviour
When deleting the contact, the _Qualification_ entity should be deleted. This is what we'd have configured the entity reference field to behave, as in other scenarios, the current behavior will be totally fine.
So, we'll need a configuration option for _Entity Reference_ custom fields for selecting what should happen upon deletion of the referenced entity. Available options could be (basically MySQL foreign key constraints):
* Remove Reference (`SET NULL`) - current behavior, maybe also the default option
* Cascade Delete (`CASCADE`) - removes the _`civicrm_value_*`_ record and the entity the custom group is attached to
* Restrict Delete (`RESTRICT`) - just as you can't e. g. delete contacts with financial transactions
* Set Default (`SET DEFAULT`) - to be configured on the custom field
* ~~Keep Reference (`NO ACTION`)~~ - might not be a good idea
## Comments
@colemanw and @michaelmcandrew might be interested.
Currently, this can be partly achieved by implementing a `preDelete` event for retrieving and deleting referencing entities.https://lab.civicrm.org/dev/core/-/issues/4332Error in membership status after a) changing membership type followed by b) p...2023-11-23T07:23:38ZJoeMurrayError in membership status after a) changing membership type followed by b) payment failure## Overview
Using a pay later payment when renewing a membership can lead to problems with the membership status, membership end date and membership type being changed at the time of the renewal being initiated ; these fields are update...## Overview
Using a pay later payment when renewing a membership can lead to problems with the membership status, membership end date and membership type being changed at the time of the renewal being initiated ; these fields are updated without a payment being recorded. It is possible that a payment will never be received, or its processing may fail. It is not easy to revert the data to its former state, or what it would have become through time from date of update to when the correction is attempted. For example, a renewal with a delayed payment might change the status from Grace to Current, the End Date from May 14, 2023 to May 14, 2024, and the Membership Type from General to Student.
These are very old problems dating to at least 2013 I believe.
The problem only occurs when both membership types have the same parent organization, and only for paid memberships. It occurs whether the membership period is Rolling or Fixed, and whether a membership type is being changed or not.
## Proposal
Refactor the current implementation so that a second, temporary membership is created that can store the new information without overwriting the old information until a payment is received for it. The new temporary membership would have a status of Pending. The Pending contribution would be related to the temporary rather than existing membership. When the payment is received (status=complete), the Pending membership's information is used to update the permanent membership, and the temporary membership record is deleted.
## Relevant code
In the Contribution Confirmation page postProcess call [legacyProcessMembership](https://github.com/civicrm/civicrm-core/blob/master/CRM/Contribute/Form/Contribution/Confirm.php#L1623), various fields are updated before the doPayment call is processed [here](https://github.com/civicrm/civicrm-core/blob/master/CRM/Contribute/Form/Contribution/Confirm.php#LL1702C42-L1702C51). As a result, the existing membership record is updated with selected membership type/end date/status after user submits a payment but before the code makes payment request, e.g. to a payment processor. This works if the payment went successfully.
In the case of a payment failure such as for IPN payment processors like Paypal Standard (occasionally when there is a delay on getting IPN callback or if the IPN response is not handled properly like https://lab.civicrm.org/dev/core/-/issues/1931) or for a manual Pay Later payment that isn't received, it leaves the selected membership in current/active state with a changed end date and possibly a different membership type. There is no fallback code written to revert the membership state or set it to Pending, and it isn't easy to reconstruct the data.
## New Behaviour
1. When initiating the Payment for Membership Renewal, create a new membership record and link the contribution in pending status to it. Add a new field, renewing_membership_id, to civicrm_membership to hold a reference from this 'temporary' pending membership to the existing membership that is being renewed. The existing membership record remains unchanged.
2. When the contribution status of the related contribution changes to Complete, update the original membership with the information from the temporary membership and delete the temporary membership.
Recommendation: delay the creation of activities for Membership Renewal (id=8), Change Membership Status (id=35) and Change Membership Type (id=36) until the contribution is completed.
Recommendation: create a new Activity Type, Membership Renewal Pending, to be created when the renewal request is received. In its body, provide: "ID of Membership being renewed: xx, Number of Periods: yy, Membership Type: [Label of Membership Type".
## Implementation:
1. Modify [here](https://github.com/civicrm/civicrm-core/blob/master/CRM/Contribute/Form/Contribution/Confirm.php#L2900), CRM_Member_BAO_Membership::getContactMembership and [here](https://github.com/civicrm/civicrm-core/blob/master/CRM/Contribute/Form/Contribution/Confirm.php#L2939).
2. Add / update new / existing unit tests on these new scenarios.EdselopezEdselopezhttps://lab.civicrm.org/dev/core/-/issues/4458Error when viewing contact-info profile without "view deleted contacts" permi...2023-08-07T12:11:55ZcolemanwError when viewing contact-info profile without "view deleted contacts" permissionThe change in b7edabe813db467aff6dd1ea083d798089198655 switched the profile to use APIv4 to fetch email id for constructing an email link. The API call looks like this:
```php
$emailID = Email::get()->setOrderBy(['is_primary' => 'DES...The change in b7edabe813db467aff6dd1ea083d798089198655 switched the profile to use APIv4 to fetch email id for constructing an email link. The API call looks like this:
```php
$emailID = Email::get()->setOrderBy(['is_primary' => 'DESC'])->setWhere([['contact_id', '=', $this->_id], ['email', '=', $email], ['on_hold', '=', FALSE], ['contact_id.is_deceased', '=', FALSE], ['contact_id.is_deleted', '=', FALSE], ['contact_id.do_not_email', '=', FALSE]])->execute()->first()['id'];
```
It was reported on SE that this fails for users without "view deleted contacts", however I'm unable to reproduce.
See https://civicrm.stackexchange.com/questions/45313/invalid-field-contact-id-is-deceased-apiv4
This should have already been double-fixed by:
- [Revert "Add permission metadata to contact is_deleted field" #22203](https://github.com/civicrm/civicrm-core/pull/22203)
- [APIv4 - Silently ignore non-permissioned fields instead of throwing exceptions #20724](https://github.com/civicrm/civicrm-core/pull/20724)https://lab.civicrm.org/dev/core/-/issues/1328Escape-on-Input => Escape-on-Output2022-09-27T23:42:48ZtottenEscape-on-Input => Escape-on-Output# Introduction
In a multi-tier application like CiviCRM, a piece of data (such as a `first_name`) may be passed through several systems and formats (MySQL/PHP/Firefox/etc; SQL/HTML/URL/JSON/CSV/PDF/etc). In each system, this piece of d...# Introduction
In a multi-tier application like CiviCRM, a piece of data (such as a `first_name`) may be passed through several systems and formats (MySQL/PHP/Firefox/etc; SQL/HTML/URL/JSON/CSV/PDF/etc). In each system, this piece of data may be encoded in a different way. The differences are sometimes imperceptible to a casual reader (ex: "banana" looks the same in SQL, HTML, URL, JSON, and CSV), but mistakes and confusion about encoding are still problematic. The impact ranges from quirky (odd characters appearing occasionally in unexpected places) to malicious (security vulnerabilities).
CiviCRM has a long-standing technical debt with respect to encoding. This debt is not specifically a bug, feature, or vulnerability in itself. (If you know a specific piece of data that is misencoded, then that is often a security problem that should be [reported to the security team](https://civicrm.org/security) for discrete resolution.) Rather, this debt is *an unusual convention* ("Escape-on-Input") which has *unintuitive consequences* and thereby invites bugs.
See also: [Dev Guide: Sanitize inputs or outputs?](https://docs.civicrm.org/dev/en/latest/security/#strategy)
At the recent Barcelona sprint, there was a partial meeting of the security team, and attendees agreed that this was the root issue behind multiple problems and merited special attention. Unfortunately, this convention is deeply embedded. Changing it is difficult and risky, so it has lingered for several years without substantive improvement, and (with its current framing/visibility) would remain indefinitely. The aim of this filing is to have a public discussion/record which works out the what/how/why of a (possible) cleanup.
# Definitions
* __Escape-on-Input (aka Esc-In)__: An architectural style in which data is taken as input and immediately escaped; it is then written to disk in escaped (HTML-esque) format. Consequently, subsequent searches+reads yield HTML-esque strings. These can be outputted literally on HTML pages without any extra processing, but (for other media) they require double processing (decode/re-encode).
* __Escape-on-Output (aka Esc-Out)__: An architectural style in which data is is taken as input and stored in its canonical format. When the data is presented as output, it is escaped in a way that matches the current output-format.
* __Canonical (String format)__: An encoding in which strings look... like they should. For example, if a user types `first_name` of `:<`, then the string is `:<`
* __HTML-esque (String format)__: An encoding in which key HTML characters are escaped as entities. For example, canonical string `:<` becomes HTML-esque string `:<`. In HTML-esque, one does not intend to convey HTML tags.
* __O(1) Plan/Task__: A plan/task in which the number of tasks is fixed. (Ex: To change behavior of all "quickforms", you might patch+test the base class `HTML_Quickform` one time.) It may strictly be 1 or 2 or 3 steps... but the number of steps does *not* depend on how many screens/use-cases are impacted.
* __O(n) Plan/Task__: A plan/task in which the number of tasks depends on how many screens/use-cases are impacted. (Ex: To change behavior of all "quickforms", you might patch+test *every subclass*.)
* __Smarty Filter/Autoescape__: A Smarty filter allows one to programmatically change the behavior/content of all Smarty templates. For example, you can write a Smarty prefilter to automatically add HTML "escape" commands to all output.
* __Linter__: A programmer tool which scans source code and checks for compliance. For example, a linter could have the rule "all Smarty variable include the `|escape` or `|noescape` modifier."
# Safe and unsafe situations
The status quo is peculiar. It is loosely safe in some common situations, but it's surprisingly unsafe in other situations. Consider this table:
<table>
<thead>
<tr><th></th><th>Read/write data via SQL/DAO/BAO</th><th>Read/write data via APIv3/v4</th></tr>
<thead>
<tbody>
<tr><th>Process inputs+outputs via HTML_Quickform+Smarty (HTML-only)</th><td>OK</td><td>Caution</td></tr>
<tr><th>Process inputs+outputs via most frameworks (Drupal Form API, AngularJS, DOM API, PHPExcel, etc)</th><td>Caution</td><td>OK</td></tr>
<tr><th>Interchange data between SQL/DAO/BAO and APIv3/v4</th><td>Caution</td><td>Caution</td></tr>
</tbody>
</table>
In "OK" situations, you can take a string and send it to the screen -- and there's a high probability it will end-up encoded correctly without any thinking/effort.
In "Caution" situations, you should treat most strings with suspicion -- many would merit extra encoding/decoding steps.
The general aim of this cleanup issue is to be "OK" on the entire grid.
# Relevant background/reading
* (Jan 2010) [CRM-5667](https://issues.civicrm.org/jira/browse/CRM-5667): Introduction of "Escape-on-Input"
* (Dec 2012) [svn log](https://github.com/civicrm/civicrm-svn/commits/v4.2/CRM/Core/HTMLInputCoder.php): Mitigation for APIv3 consumers
* (Dec 2012) [CRM-11532](https://issues.civicrm.org/jira/browse/CRM-11532): First filing about the problematic status-quo. (Shorter)
* (Apr 2018, et al) [security/core#6](https://lab.civicrm.org/security/core/issues/6): Second filing about the problematic status-quo. (Longer)
* (2018) [Dev Guide: Sanitize inputs or outputs?](https://docs.civicrm.org/dev/en/latest/security/#strategy): Reference material for developers
* (Oct 2019) [POC: Auto-escaping in Smarty v2](https://gist.github.com/totten/5b30e10a21fe626a43a30e21ded26fc4)
* (Oct 2019) [Report: Categorize DB fields by escaping implication](https://gist.github.com/totten/ec78dcb869aa7cbbc95c5f273f7ea30d)
* (Oct 2019) [Report: List of Smarty expressions](https://gist.github.com/totten/9c7176bbdfc63f88d423f026359f50fc)
# Candidate Approaches
* __Do Nothing__:
* __How__: Kick off your shoes. Pull up Netflix.
* __Pro__: Less work
* __Con__: Encourages developers to write buggy code. Feels embarrassing whenever they realize this is a thing.
* __Epic w/Smarty Filter__:
* __Pitch__: Smarty does all HTML-escaping automatically and by default.
* __How__:
* Remove Esc-In policy (`HTMLInputCoder`)
* Write a generic upgrade step to recode all stored strings.
* To restore output-coding, add a Smarty prefilter to auto-escape output.
* __Pro__: Cleans up the database.
* __Con__: The main problem is Smarty templates are not homogeneous - *most* generate HTML, but *some* generate other formats. They *often* pass-through data from the database, but they *also* display data which is computed and/or loaded from a different source. Consequently, there is an O(n) task of re-testing all screens/fields and patching a large number of exceptions.
* __Con__: Additionally, there's the `r-tech` issue - even if we correctly change every screen/field/line in the main application, there is an open-set of third-party integrations. If these use SQL/DAO/BAO and work correctly in the current system, then they'll likely break as soon as the epic change is made, which will create stress for upgrade-scheduling.
* __Con__: Smarty is difficult to write unit tests for.
* __Epic w/DAO Filter__:
* __Pitch__: DAO does all HTML-escaping automatically and by default.
* __How__:
* Remove Esc-In policy (`HTMLInputCoder`)
* Write a generic upgrade step to recode all stored strings.
* To restore output-coding, update DAO/BAO with an optional escaping mode. (Take the current skiplist - everything else defaults to autoescaping.)
* __Pro__: Cleans up the database.
* __Pro__: Looks like an O(1) plan.
* __Pro__: Restoring output coding can be done at a single point - no need to fix existing templates.
* __Pro__: Output coding can be unit-tested.
* __Con__: Only addresses DAO-based query-building (`$d=new DAO_Foo(); $d->bar=123; $d->fetch()`). Does not address `executeQuery('SELECT foo AS bar...')`, `executeQuery('UPDATE foo SET bar...')`, or other query-building. This seems to be left as O(n) task?
* __Question__: Do we get read/write consistency?
* __Wait by Extension Approach__:
* __Pitch__: Grow new UIs on top of APIv3/v4. Obsolete everything all UIs that rely on DAO/BAO/SQL.
* __How__: Do nothing specifically for this problem. Instead, wait for extensions like Afform or Data Processor to gradually replace the HTML_Quickform/Smarty UIs. Once there's a critical mass, remove all old UIs and do an "Epic" switch.
* __Pro__: Don't have to do anything now! Creates pressure to do other useful changes.
* __Con__: You may be waiting a long time.
* __Incremental, Per-Component/Per-Field__:
* __Pitch__: Convert fields incrementally
* __How__: Make a list of all relevant DB fields (~500). In the first month, take a subset (such as "all CiviEvent fields") and transition/test the subset. In the second month, do another subset of fields (such as "all CiviCampaign fields").
* __Pro__: This allows tasks to spread out over time with more manageable/affordable chunks.
* __Con__: This still has the `r-tech` issue for third-party integrations. It creates a long-term drip-drip of compatibility breaks -- every month, some subset of fields have their data-format changed. This will make it quite challenging for extensions/integrations to keep in sync.
* __Comment__: Bespoke screens (e.g. "New participant") are amenable to simple grep+divide+conquer, but we might get boxed-in with how to update metadata-driven screens which tend to treat all strings the same way (e.g. import/export).
* __Incremental, Setting Plus Smarty Linting__:
* __Pitch__: Add toggle to allow old+new storage-formats. Incrementally enhance code to make new-storage-format work.
* __How__:
* Define a new setting to indicate whether strings are stored as "html-esque" (initial-default; long-term-deprecate) or "canonical" (initial-experimental; long-term-goal).
* Define a migration script - which can be run whenever one changes the setting.
* Update Smarty/Quickform/API to check the setting and filter accordingly. Define corresponding Smarty filter. (Naive example: `{$records[$cid]->display_name|crmEscape:'Contact.display_name'}` means "escape this data based on the escaping-rules for the `Contact` field `display_name`)
* Write a linter to report on which Smarty templates use the filter. Over time (month-by-month), incrementally update Smarty templates to use the new filters and re-test those screens.
* __Pro__: This allows tasks to be spread out over time with more manageable/affordable chunks. The linter provides a guide for helping us track "cleared ground".
* __Pro__: Site-implementers and extension/integration-authors will have greater flexibility on scheduling the change - they don't have to update all codes and all databases at the same time.
* __Con__: Requires exposing some new filters/functions/settings.
* __Incremental, Setting Plus DAO Filter__:
* __How__:
* Define a new setting to indicate whether strings are stored as "html-esque" (initial-default; long-term-deprecate) or "canonical" (initial-experimental; long-term-goal).
* Define a migration script - which can be run whenever one changes the setting.
* Update DAO/API to check the setting and filter accordingly.
* __Pro__: At a glance, this looks like an O(1) plan.
* __Pro__: Site-implementers and extension/integration-authors will have greater flexibility on scheduling the change - they don't have to update all codes and all databases at the same time.
* __Con__: Only addresses DAO-based query-building (`$d=new DAO_Foo(); $d->bar=123; $d->fetch()`). Does not address literal queries (`executeQuery('SELECT foo AS bar...')`, `executeQuery('UPDATE foo SET bar...')`) or other forms of query-building (`CRM_Utils_SQL`/APIv4/adhoc PHP). This seems to be left as O(n) task?
* __Question__: Do we get read/write consistency?
* __Incremental, SQL Views with Deprecation Alerts__:
* __Pitch__: Support both formats in MySQL *at the same time*. Borrow the idea of wide-spread VIEWs from multilingual.
* __How__:
* Every entity (eg `Contact`) should have a concrete TABLE and a derived VIEW (eg `civicrm_contact` and `civicrm_contact_new`) which present the same data in different formats (old/HTML-esque and new/canonical).
* `executeQuery()` generates a deprecation warning whenever there's a query that hits old-format table. The warning says something like "`civicrm_contact_old` is deprecated. Please use `civicrm_contact_new` instead. Be sure to update the escaping on any consumers."
* At the mid-point during transition, swap the SQL schema (eg the old-format is given by SQL VIEW and the new-format is given by SQL TABLE).
* Eventually, remove the old-format VIEWs completely.
* __Pro__: Existing (unpatched) consumers in any medium continue to work throughout the transition. But they start to emit warnings immediately.
* __Pro__: Extension/integration-authors (and site-admins/upgraders) have greater flexibility on scheduling changes - they don't have to update all codes and all databases at the same time.
* __Con__: Query performance on the filtered VIEW should be slower than the canonical TABLE. During the transitional process, queries will flop-flop around between faster/slower.https://lab.civicrm.org/dev/core/-/issues/4018Extension upgrades are not run if there are no core upgrades2022-12-02T12:24:19ZherbdoolExtension upgrades are not run if there are no core upgrades@totten recently did: https://github.com/civicrm/civicrm-core/pull/24030 to include extension upgrades in the core upgrades.
I've noticed a couple times, however, that there seem to be extension upgrades still waiting after upgrading Ci...@totten recently did: https://github.com/civicrm/civicrm-core/pull/24030 to include extension upgrades in the core upgrades.
I've noticed a couple times, however, that there seem to be extension upgrades still waiting after upgrading CiviCRM code base and running `cv upgrade:db`. Could it be that if there are no core upgrade hooks waiting that it won't run the extension upgrades? It does seem likely given my chat with @colemanw https://chat.civicrm.org/civicrm/pl/cfd18cjpofg5iesh7o4zqejcbh.https://lab.civicrm.org/dev/core/-/issues/5017Fatal Error with Angular Manager.php after upgrading to CiviCRM 5.69.52024-02-23T15:18:48ZLKuttnerFatal Error with Angular Manager.php after upgrading to CiviCRM 5.69.5After upgrading to CiviCRM 5.69.5 from 5.64.4 we get a WSOD.
`Parse error: syntax error, unexpected '=' in .../modules/civicrm/Civi/Angular/Manager.php on line 98`
`Fatal error: Exception thrown without a stack frame in Unknown on line...After upgrading to CiviCRM 5.69.5 from 5.64.4 we get a WSOD.
`Parse error: syntax error, unexpected '=' in .../modules/civicrm/Civi/Angular/Manager.php on line 98`
`Fatal error: Exception thrown without a stack frame in Unknown on line 0
`
I have not been able to identify what is causing this.
I have disabled all the non-standard CiviCRM extensions, with no change.
This is on Drupal 7.99 with PHP 7.3.33. Thank you for any help you can offer.