CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-09-12T18:50:06Zhttps://lab.civicrm.org/dev/core/-/issues/1486Deadlocks on acl_cache2023-09-12T18:50:06ZeileenDeadlocks on acl_cacheOne of the prevalent deadlocks we see is on the acl_contact_cache. I've found that I can replicate this locally fairly consistently just by running 2 sets of tests at once.
The thing that is somewhat odd about this is that we don't use ...One of the prevalent deadlocks we see is on the acl_contact_cache. I've found that I can replicate this locally fairly consistently just by running 2 sets of tests at once.
The thing that is somewhat odd about this is that we don't use ACLs at all so in fact the table is always empty. Notably the table is also extremely slow to truncate - despite being empty.
**What happens**
The flow we have is
1) api call to create a contact
2) when the contact is created CRM_Contact_BAO_Contact_Utils::clearContactCaches() is called. It hits a deadlock because
a) it has an FK to contact.id (& in our case it has an FK and an index seemingly)
b) it is called a lot
3) it retries and succeeds
4) we then try to add an email - the add function fails on a foreign key constraint because the earlier deadlock rolled the contact create back
**Proposals**
I think there are a number of things we can do & we should do some combo:
1) Make the delete function less expensive & less locky
a) Add an index on modified date - we are constantly deleting where modified_date < x so let's index it
b) remove the index on acl_id - we ALSO have an FK so having both is probably harmless-but-confusing
c) alter the FK on contact ID to be an index not a foreign key. This is perhaps a confusing suggestion but it is also the one that will make a real dent in these deadlocks. Remember this is a cache table not a 'persistent' table. The impact of the rows outliving the existence of the contact for a little bit is basically none because we don't do ACL lookups on non-existent contacts (this is also potentially helpful for better handling 0 or anonymous contact_id where we've had to work around this before
2) Protect against conflict through mysql locks. I think we can use a mysql lock to avoid 2 processes attempting this flush at once. I'll do a PR for this
3) How often do we REALLY need to flush those caches. Maybe we could cache a 'last flushed' key & flush no more than every 30 seconds.
4) Consider checking if the table is empty before attempting to empty it.
5) Review the places where acl cache is called from & decide how necessary there are - here is the list:
**CRM_ACL_BAO_Cache::resetCache**
CRM/Contact/BAO/Contact/Utils.php: CRM_ACL_BAO_Cache::resetCache();
CRM/Core/BAO/Cache.php: CRM_ACL_BAO_Cache::resetCache();
CRM/Utils/System.php: CRM_ACL_BAO_Cache::resetCache();
CRM/ACL/Page/EntityRole.php: CRM_ACL_BAO_Cache::resetCache();
CRM/ACL/Form/ACLBasic.php: CRM_ACL_BAO_Cache::resetCache();
CRM/ACL/Form/EntityRole.php: CRM_ACL_BAO_Cache::resetCache();
CRM/ACL/BAO/ACL.php: CRM_ACL_BAO_Cache::resetCache();
**CRM_Contact_BAO_Contact_Utils::clearContactCaches()**
CRM/Contact/BAO/Contact/Utils.php: public static function clearContactCaches($isEmptyPrevNextTable = FALSE) {
CRM/Contact/BAO/GroupContact.php: CRM_Contact_BAO_Contact_Utils::clearContactCaches();
CRM/Contact/BAO/GroupContact.php: CRM_Contact_BAO_Contact_Utils::clearContactCaches();
CRM/Contact/BAO/Contact.php: CRM_Contact_BAO_Contact_Utils::clearContactCaches();
CRM/Contact/Import/Form/Preview.php: CRM_Contact_BAO_Contact_Utils::clearContactCaches(TRUE);
bin/cli.class.php: CRM_Contact_BAO_Contact_Utils::clearContactCaches();
6) A bigger task perhaps - but permit opportunistic flushing to be turned off in favour of a cron (per group contact cache)
**Deadlock output**
LATEST DETECTED DEADLOCK
------------------------
2019-12-19 11:29:45 0x700010cd2000
*** (1) TRANSACTION:
TRANSACTION 34518715, ACTIVE 1 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 11 lock struct(s), heap size 1136, 4 row lock(s), undo log entries 7
MySQL thread id 571, OS thread handle 123145583353856, query id 709677 localhost 127.0.0.1 drupalcivi_lkznr updating
DELETE
FROM civicrm_acl_cache
WHERE modified_date IS NULL
OR (modified_date <= '20191218222444')
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 143182 page no 3 n bits 80 index PRIMARY of table `drupalcivi_lkznr`.`civicrm_acl_cache` trx id 34518715 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 6; compact format; info bits 32
0: len 4; hex 00003c14; asc < ;;
1: len 6; hex 0000020eb6ba; asc ;;
2: len 7; hex 3b000001442ac5; asc ; D* ;;
3: SQL NULL;
4: len 4; hex 00000002; asc ;;
5: SQL NULL;
*** (2) TRANSACTION:
TRANSACTION 34518714, ACTIVE 1 sec starting index read
mysql tables in use 2, locked 2
66 lock struct(s), heap size 8400, 71 row lock(s), undo log entries 74
MySQL thread id 573, OS thread handle 123145584189440, query id 710022 localhost 127.0.0.1 drupalcivi_lkznr updating
UPDATE civicrm_contact SET contact_type = 'Organization' , contact_sub_type = NULL , email_greeting_id = NULL , postal_greeting_id = NULL , addressee_id = 3 , employer_id = 15055 WHERE ( civicrm_contact.id = 15055 )
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 143182 page no 3 n bits 80 index PRIMARY of table `drupalcivi_lkznr`.`civicrm_acl_cache` trx id 34518714 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;seamusleeseamusleehttps://lab.civicrm.org/dev/core/-/issues/3302Add accessible fieldset for checkboxes and radio buttons2023-11-28T12:49:35ZMonish DebAdd accessible fieldset for checkboxes and radio buttonsAs per https://www.w3.org/WAI/tutorials/forms/grouping , grouped form controls like radio buttons/checkboxes must be enclosed in ```<fieldset>``` and the ```<legend>``` element act as a header to identify the group of elements. Currently...As per https://www.w3.org/WAI/tutorials/forms/grouping , grouped form controls like radio buttons/checkboxes must be enclosed in ```<fieldset>``` and the ```<legend>``` element act as a header to identify the group of elements. Currently, when we review such block with WAVE accessibility tool, it complains about:
![Screen_Shot_2018-06-01_at_6.53.18_PM](/uploads/0782446dde4a776d6ec4a8f5c51edf47/Screen_Shot_2018-06-01_at_6.53.18_PM.png)
## Proposal
1. Enclose the checkbox/radio button block in a fieldset
2. Add a special class to that fieldset say `crm-sr-fieldset` which hides it in front-end forms (public and backoffice forms)
3. Use label of checkbox/radio buttons as its legend header.
Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/719can't delete profile pre- post- field help on multilinguage sites2022-08-26T00:20:44ZJoeMurraycan't delete profile pre- post- field help on multilinguage sitesIf you add field pre and post help to profiles on a single language site, you can then delete it and save successfully. On a site with more than 1 language, trying to delete the help leads to it being resaved (I believe from the alternat...If you add field pre and post help to profiles on a single language site, you can then delete it and save successfully. On a site with more than 1 language, trying to delete the help leads to it being resaved (I believe from the alternative language's copy of the help).
Verified on dmaster 5.12.alpha1.Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/3648SMS limit is hardcoded at 460 but should be changeable somewhere2024-01-29T09:44:22ZjasonobrownSMS limit is hardcoded at 460 but should be changeable somewhereTwillio (and probably other sms providers) allows SMS messages up to 1600 characters, but Civi has the limit hardcoded at 460. I've verified that by simply going to CRM/SMS/Provider.php and changing MAX_SMS_CHAR from 460 to 1600 the sys...Twillio (and probably other sms providers) allows SMS messages up to 1600 characters, but Civi has the limit hardcoded at 460. I've verified that by simply going to CRM/SMS/Provider.php and changing MAX_SMS_CHAR from 460 to 1600 the system now accepts (and sends) messages up to 1600 characters. It seems logical that the character limit should able to be adjusted and stored somewhere rather than set as a static value in the code. We are currently using the larger limit, and would really like to see this implemented asap so that we can stop manually updating the code each time we update civi.https://lab.civicrm.org/dev/core/-/issues/1083Non-deductible amount not working2023-12-21T11:11:54ZJoeMurrayNon-deductible amount not workingOn 5.14 and dmaster 5.16.alpha1 when a price set is configured with an option that includes a non-deductible amount, then a purchase is made of that option, the non-deductible amount is not appearing on the contribution.
Price Set Field...On 5.14 and dmaster 5.16.alpha1 when a price set is configured with an option that includes a non-deductible amount, then a purchase is made of that option, the non-deductible amount is not appearing on the contribution.
Price Set Field with non-deductible $5 on $10 price:
![2019-06-27_14-44-40](/uploads/3954b396bb81d67c5f01cf5c6464b85a/2019-06-27_14-44-40.png)
Contribution for $10 with $0 non-deductible:
![2019-06-27_14-43-59](/uploads/9c43a1a6c2a0381ed94c403901228821/2019-06-27_14-43-59.png)EdselopezEdselopezhttps://lab.civicrm.org/dev/core/-/issues/1191CiviCRM reaching MySQL join limit2023-11-26T16:41:52ZJGauntCiviCRM reaching MySQL join limitI've seen a similar error on a few websites where there is a lot of custom data which can throw up a few issues and it was mentioned I should write up an issue here to see whether it is a common issue in the community.
All current versi...I've seen a similar error on a few websites where there is a lot of custom data which can throw up a few issues and it was mentioned I should write up an issue here to see whether it is a common issue in the community.
All current versions of MySQL and MariaDB have a join limit of 61 and Civi needs to join a lot of tables together (dependant on how much custom data you have). This can cause problems when you have a site with a lot of custom data sets as this join limit is reached which prevents Civi from working as expected.
An example of this would be yesterday where I tried to load up a contact's activities but, because of the amount of custom data sets on the site, I was given an AJAX error and could not view the contact's activities.
After doing a bit of debugging, I found out the reason was because the join limit of 61 had been reached.
I managed to resolve the issue by disabling some unused custom data sets but I am wondering if this is quite a common issue with some bigger sites.
My thinking is that as time goes on, the database will naturally get bigger as more is added to it and the issue will come up more and more.
Looking forward to hearing some thoughts and suggestions on this!https://lab.civicrm.org/dev/core/-/issues/1328Escape-on-Input => Escape-on-Output2022-09-27T23:42:48ZtottenEscape-on-Input => Escape-on-Output# Introduction
In a multi-tier application like CiviCRM, a piece of data (such as a `first_name`) may be passed through several systems and formats (MySQL/PHP/Firefox/etc; SQL/HTML/URL/JSON/CSV/PDF/etc). In each system, this piece of d...# Introduction
In a multi-tier application like CiviCRM, a piece of data (such as a `first_name`) may be passed through several systems and formats (MySQL/PHP/Firefox/etc; SQL/HTML/URL/JSON/CSV/PDF/etc). In each system, this piece of data may be encoded in a different way. The differences are sometimes imperceptible to a casual reader (ex: "banana" looks the same in SQL, HTML, URL, JSON, and CSV), but mistakes and confusion about encoding are still problematic. The impact ranges from quirky (odd characters appearing occasionally in unexpected places) to malicious (security vulnerabilities).
CiviCRM has a long-standing technical debt with respect to encoding. This debt is not specifically a bug, feature, or vulnerability in itself. (If you know a specific piece of data that is misencoded, then that is often a security problem that should be [reported to the security team](https://civicrm.org/security) for discrete resolution.) Rather, this debt is *an unusual convention* ("Escape-on-Input") which has *unintuitive consequences* and thereby invites bugs.
See also: [Dev Guide: Sanitize inputs or outputs?](https://docs.civicrm.org/dev/en/latest/security/#strategy)
At the recent Barcelona sprint, there was a partial meeting of the security team, and attendees agreed that this was the root issue behind multiple problems and merited special attention. Unfortunately, this convention is deeply embedded. Changing it is difficult and risky, so it has lingered for several years without substantive improvement, and (with its current framing/visibility) would remain indefinitely. The aim of this filing is to have a public discussion/record which works out the what/how/why of a (possible) cleanup.
# Definitions
* __Escape-on-Input (aka Esc-In)__: An architectural style in which data is taken as input and immediately escaped; it is then written to disk in escaped (HTML-esque) format. Consequently, subsequent searches+reads yield HTML-esque strings. These can be outputted literally on HTML pages without any extra processing, but (for other media) they require double processing (decode/re-encode).
* __Escape-on-Output (aka Esc-Out)__: An architectural style in which data is is taken as input and stored in its canonical format. When the data is presented as output, it is escaped in a way that matches the current output-format.
* __Canonical (String format)__: An encoding in which strings look... like they should. For example, if a user types `first_name` of `:<`, then the string is `:<`
* __HTML-esque (String format)__: An encoding in which key HTML characters are escaped as entities. For example, canonical string `:<` becomes HTML-esque string `:<`. In HTML-esque, one does not intend to convey HTML tags.
* __O(1) Plan/Task__: A plan/task in which the number of tasks is fixed. (Ex: To change behavior of all "quickforms", you might patch+test the base class `HTML_Quickform` one time.) It may strictly be 1 or 2 or 3 steps... but the number of steps does *not* depend on how many screens/use-cases are impacted.
* __O(n) Plan/Task__: A plan/task in which the number of tasks depends on how many screens/use-cases are impacted. (Ex: To change behavior of all "quickforms", you might patch+test *every subclass*.)
* __Smarty Filter/Autoescape__: A Smarty filter allows one to programmatically change the behavior/content of all Smarty templates. For example, you can write a Smarty prefilter to automatically add HTML "escape" commands to all output.
* __Linter__: A programmer tool which scans source code and checks for compliance. For example, a linter could have the rule "all Smarty variable include the `|escape` or `|noescape` modifier."
# Safe and unsafe situations
The status quo is peculiar. It is loosely safe in some common situations, but it's surprisingly unsafe in other situations. Consider this table:
<table>
<thead>
<tr><th></th><th>Read/write data via SQL/DAO/BAO</th><th>Read/write data via APIv3/v4</th></tr>
<thead>
<tbody>
<tr><th>Process inputs+outputs via HTML_Quickform+Smarty (HTML-only)</th><td>OK</td><td>Caution</td></tr>
<tr><th>Process inputs+outputs via most frameworks (Drupal Form API, AngularJS, DOM API, PHPExcel, etc)</th><td>Caution</td><td>OK</td></tr>
<tr><th>Interchange data between SQL/DAO/BAO and APIv3/v4</th><td>Caution</td><td>Caution</td></tr>
</tbody>
</table>
In "OK" situations, you can take a string and send it to the screen -- and there's a high probability it will end-up encoded correctly without any thinking/effort.
In "Caution" situations, you should treat most strings with suspicion -- many would merit extra encoding/decoding steps.
The general aim of this cleanup issue is to be "OK" on the entire grid.
# Relevant background/reading
* (Jan 2010) [CRM-5667](https://issues.civicrm.org/jira/browse/CRM-5667): Introduction of "Escape-on-Input"
* (Dec 2012) [svn log](https://github.com/civicrm/civicrm-svn/commits/v4.2/CRM/Core/HTMLInputCoder.php): Mitigation for APIv3 consumers
* (Dec 2012) [CRM-11532](https://issues.civicrm.org/jira/browse/CRM-11532): First filing about the problematic status-quo. (Shorter)
* (Apr 2018, et al) [security/core#6](https://lab.civicrm.org/security/core/issues/6): Second filing about the problematic status-quo. (Longer)
* (2018) [Dev Guide: Sanitize inputs or outputs?](https://docs.civicrm.org/dev/en/latest/security/#strategy): Reference material for developers
* (Oct 2019) [POC: Auto-escaping in Smarty v2](https://gist.github.com/totten/5b30e10a21fe626a43a30e21ded26fc4)
* (Oct 2019) [Report: Categorize DB fields by escaping implication](https://gist.github.com/totten/ec78dcb869aa7cbbc95c5f273f7ea30d)
* (Oct 2019) [Report: List of Smarty expressions](https://gist.github.com/totten/9c7176bbdfc63f88d423f026359f50fc)
# Candidate Approaches
* __Do Nothing__:
* __How__: Kick off your shoes. Pull up Netflix.
* __Pro__: Less work
* __Con__: Encourages developers to write buggy code. Feels embarrassing whenever they realize this is a thing.
* __Epic w/Smarty Filter__:
* __Pitch__: Smarty does all HTML-escaping automatically and by default.
* __How__:
* Remove Esc-In policy (`HTMLInputCoder`)
* Write a generic upgrade step to recode all stored strings.
* To restore output-coding, add a Smarty prefilter to auto-escape output.
* __Pro__: Cleans up the database.
* __Con__: The main problem is Smarty templates are not homogeneous - *most* generate HTML, but *some* generate other formats. They *often* pass-through data from the database, but they *also* display data which is computed and/or loaded from a different source. Consequently, there is an O(n) task of re-testing all screens/fields and patching a large number of exceptions.
* __Con__: Additionally, there's the `r-tech` issue - even if we correctly change every screen/field/line in the main application, there is an open-set of third-party integrations. If these use SQL/DAO/BAO and work correctly in the current system, then they'll likely break as soon as the epic change is made, which will create stress for upgrade-scheduling.
* __Con__: Smarty is difficult to write unit tests for.
* __Epic w/DAO Filter__:
* __Pitch__: DAO does all HTML-escaping automatically and by default.
* __How__:
* Remove Esc-In policy (`HTMLInputCoder`)
* Write a generic upgrade step to recode all stored strings.
* To restore output-coding, update DAO/BAO with an optional escaping mode. (Take the current skiplist - everything else defaults to autoescaping.)
* __Pro__: Cleans up the database.
* __Pro__: Looks like an O(1) plan.
* __Pro__: Restoring output coding can be done at a single point - no need to fix existing templates.
* __Pro__: Output coding can be unit-tested.
* __Con__: Only addresses DAO-based query-building (`$d=new DAO_Foo(); $d->bar=123; $d->fetch()`). Does not address `executeQuery('SELECT foo AS bar...')`, `executeQuery('UPDATE foo SET bar...')`, or other query-building. This seems to be left as O(n) task?
* __Question__: Do we get read/write consistency?
* __Wait by Extension Approach__:
* __Pitch__: Grow new UIs on top of APIv3/v4. Obsolete everything all UIs that rely on DAO/BAO/SQL.
* __How__: Do nothing specifically for this problem. Instead, wait for extensions like Afform or Data Processor to gradually replace the HTML_Quickform/Smarty UIs. Once there's a critical mass, remove all old UIs and do an "Epic" switch.
* __Pro__: Don't have to do anything now! Creates pressure to do other useful changes.
* __Con__: You may be waiting a long time.
* __Incremental, Per-Component/Per-Field__:
* __Pitch__: Convert fields incrementally
* __How__: Make a list of all relevant DB fields (~500). In the first month, take a subset (such as "all CiviEvent fields") and transition/test the subset. In the second month, do another subset of fields (such as "all CiviCampaign fields").
* __Pro__: This allows tasks to spread out over time with more manageable/affordable chunks.
* __Con__: This still has the `r-tech` issue for third-party integrations. It creates a long-term drip-drip of compatibility breaks -- every month, some subset of fields have their data-format changed. This will make it quite challenging for extensions/integrations to keep in sync.
* __Comment__: Bespoke screens (e.g. "New participant") are amenable to simple grep+divide+conquer, but we might get boxed-in with how to update metadata-driven screens which tend to treat all strings the same way (e.g. import/export).
* __Incremental, Setting Plus Smarty Linting__:
* __Pitch__: Add toggle to allow old+new storage-formats. Incrementally enhance code to make new-storage-format work.
* __How__:
* Define a new setting to indicate whether strings are stored as "html-esque" (initial-default; long-term-deprecate) or "canonical" (initial-experimental; long-term-goal).
* Define a migration script - which can be run whenever one changes the setting.
* Update Smarty/Quickform/API to check the setting and filter accordingly. Define corresponding Smarty filter. (Naive example: `{$records[$cid]->display_name|crmEscape:'Contact.display_name'}` means "escape this data based on the escaping-rules for the `Contact` field `display_name`)
* Write a linter to report on which Smarty templates use the filter. Over time (month-by-month), incrementally update Smarty templates to use the new filters and re-test those screens.
* __Pro__: This allows tasks to be spread out over time with more manageable/affordable chunks. The linter provides a guide for helping us track "cleared ground".
* __Pro__: Site-implementers and extension/integration-authors will have greater flexibility on scheduling the change - they don't have to update all codes and all databases at the same time.
* __Con__: Requires exposing some new filters/functions/settings.
* __Incremental, Setting Plus DAO Filter__:
* __How__:
* Define a new setting to indicate whether strings are stored as "html-esque" (initial-default; long-term-deprecate) or "canonical" (initial-experimental; long-term-goal).
* Define a migration script - which can be run whenever one changes the setting.
* Update DAO/API to check the setting and filter accordingly.
* __Pro__: At a glance, this looks like an O(1) plan.
* __Pro__: Site-implementers and extension/integration-authors will have greater flexibility on scheduling the change - they don't have to update all codes and all databases at the same time.
* __Con__: Only addresses DAO-based query-building (`$d=new DAO_Foo(); $d->bar=123; $d->fetch()`). Does not address literal queries (`executeQuery('SELECT foo AS bar...')`, `executeQuery('UPDATE foo SET bar...')`) or other forms of query-building (`CRM_Utils_SQL`/APIv4/adhoc PHP). This seems to be left as O(n) task?
* __Question__: Do we get read/write consistency?
* __Incremental, SQL Views with Deprecation Alerts__:
* __Pitch__: Support both formats in MySQL *at the same time*. Borrow the idea of wide-spread VIEWs from multilingual.
* __How__:
* Every entity (eg `Contact`) should have a concrete TABLE and a derived VIEW (eg `civicrm_contact` and `civicrm_contact_new`) which present the same data in different formats (old/HTML-esque and new/canonical).
* `executeQuery()` generates a deprecation warning whenever there's a query that hits old-format table. The warning says something like "`civicrm_contact_old` is deprecated. Please use `civicrm_contact_new` instead. Be sure to update the escaping on any consumers."
* At the mid-point during transition, swap the SQL schema (eg the old-format is given by SQL VIEW and the new-format is given by SQL TABLE).
* Eventually, remove the old-format VIEWs completely.
* __Pro__: Existing (unpatched) consumers in any medium continue to work throughout the transition. But they start to emit warnings immediately.
* __Pro__: Extension/integration-authors (and site-admins/upgraders) have greater flexibility on scheduling changes - they don't have to update all codes and all databases at the same time.
* __Con__: Query performance on the filtered VIEW should be slower than the canonical TABLE. During the transitional process, queries will flop-flop around between faster/slower.https://lab.civicrm.org/dev/core/-/issues/1408The DAO "FreshnessTest" in the jenkins test suite does nothing because setup....2023-01-08T05:19:21ZDaveDThe DAO "FreshnessTest" in the jenkins test suite does nothing because setup.sh runs before itThe test itself works (CRM_Core_CodeGen_FreshnessTest::testDAOs()), but because in jenkins runs setup.sh gets run near the start of the build, a DAO will appear to be updated even if the git tree doesn't have the updated DAO.
The purpos...The test itself works (CRM_Core_CodeGen_FreshnessTest::testDAOs()), but because in jenkins runs setup.sh gets run near the start of the build, a DAO will appear to be updated even if the git tree doesn't have the updated DAO.
The purpose of the freshness test seems to be to make sure you don't forget to include the DAO in PRs when you make changes to an xml schema to add/change/remove a field in core tables. But it's not doing that currently in jenkins runs because it's always comparing against DAOs after setup.sh/GenCode.php has run.
I'm not sure if there's an easy fix, or maybe the test should be removed and replaced with some developer documentation? It wouldn't even work in a local test suite run if you've run GenCode.php, which you would have had to do to physically try out your schema changes.https://lab.civicrm.org/dev/core/-/issues/1437Creating relationship with start date in future incorrectly shows relationshi...2023-01-17T01:36:45Zellen_compucorpCreating relationship with start date in future incorrectly shows relationship as activeOverview
----------------------------------------
Creating relationship with start date in future incorrectly shows relationship as active
Reproduction steps
----------------------------------------
1. Go to a contact record, then go to...Overview
----------------------------------------
Creating relationship with start date in future incorrectly shows relationship as active
Reproduction steps
----------------------------------------
1. Go to a contact record, then go to the relationships tab
1. Create a new relationship of any type
1. Give the relationship a start date that is in the future
1. Save the new relationship
Current behaviour
----------------------------------------
The relationship is saved correctly, but displays within 'current relationships' in the top half of the relationships tab.
![6AD41E8B-5DD7-4497-84A5-96600B621E9A](/uploads/a43484418ac5b913033190c9e3a52f73/6AD41E8B-5DD7-4497-84A5-96600B621E9A.jpg)
Expected behaviour
----------------------------------------
As the relationship has not yet started, the relationship should show as inactive up until the start date; at which point it should show as current.
Environment information
----------------------------------------
* __Browser:__ Chrome
* __CiviCRM:__ 5.19 (also tested on 5.15 on this test site and contact record: https://civicrm.demo.civihosting.com/civicrm/contact/view?reset=1&cid=98)
* __PHP:__ _7.0
* __CMS:__ Drupal
Comments
----------------------------------------
Contact record from my testing on a demo site: https://civicrm.demo.civihosting.com/civicrm/contact/view?reset=1&cid=98
cc @jamienovick1 @shitijghttps://lab.civicrm.org/dev/core/-/issues/1516Discussion ticket for handling multivalued tokens in pdfs/emails2023-09-25T14:58:11ZDaveDDiscussion ticket for handling multivalued tokens in pdfs/emailsIn particular related to:
* https://github.com/civicrm/civicrm-core/pull/12012
* and its rebased version https://github.com/civicrm/civicrm-core/pull/16200
* https://github.com/civicrm/civicrm-core/pull/16208
There are a number of way...In particular related to:
* https://github.com/civicrm/civicrm-core/pull/12012
* and its rebased version https://github.com/civicrm/civicrm-core/pull/16200
* https://github.com/civicrm/civicrm-core/pull/16208
There are a number of ways to deal with multivalued fields when you need to produce scalar output based on the field.
For example:
* Take the "first" one.
* This has the disadvantage that "first" doesn't really mean anything, especially if the order is not guaranteed to be the same each time.
* Take the "last" one.
* Ditto.
* Combine them separated by commas.
* This has the disadvantage that it may not make any sense or might look silly. Consider something like the "With Contact" on an activity. Suppose there are 3. Suppose you want to include the primary phone of the contact. Your pdf might look like this. It's not terrible, but now think about adding in address too.
* With Contact: John Smith, Mary Brown, Indiana Jones
* Phone: 555-123-2222, , 555-383-8383
* And it's not clear to me that it can guarantee that the phone values are in the same order as the names when the tokens are separate.
* Do as in https://github.com/civicrm/civicrm-core/pull/12012, where you give the template designer the option to specify a numeric suffix on the token, e.g. {activity.target_2_display_name}. The way it's implemented there has some limitations:
* The order is still arbitrary - what does "2" actually mean?
* The template designer doesn't know max(N), so they can't easily do something like concatenate all of them if they want to.
* Provide some shorthand tokens for some common uses, e.g.
* {activity.target_1_phone} means arbitrarily take the first one.
* {activity.target_COMBINE_phone} means combine all with commas.
* {activity.target_ALL_phone} means give an array and I'll do some smarty-based looping in my template to get what I want.
Further, there are some situations where you might not want some recipients to see some of the other values. An example might be activity.case_id as in [16208](https://github.com/civicrm/civicrm-core/pull/16208). There maybe it makes sense to have a token that means "concatenate all the cases where the recipient has a role, and leave out the others". A similar one might apply for activity.target_display_name, where it might mean "concatenate all the with contact display names for activities where the recipient is one of the activity_contacts.
A related note that's just food for thought: In my other life, I always try to set up 2 fields when somebody asks for something multivalued. For example "services performed". Somebody might come to you asking for a couple of the different types of service you offer. You can designate one as the primary service, which is a single-valued field, which then makes reporting work because you don't run into double-counting, and then have a secondary services field that is multi-valued, which you don't use for reporting, and in emails/pdfs you can concatenate it without it being arbitrary because you would also output the primary which is single-valued. That may not be useful right now, but this is a discussion ticket.https://lab.civicrm.org/dev/core/-/issues/1576Patches in `composer.json` do not apply cleanly2023-01-13T06:18:05ZtottenPatches in `composer.json` do not apply cleanlyOverview
----------------------------------------
The definition of `composer.json` appears to require certain patches; however, these cannot be applied in projects where `civicrm-core` is loaded as a dependency.
Reproduction steps
----...Overview
----------------------------------------
The definition of `composer.json` appears to require certain patches; however, these cannot be applied in projects where `civicrm-core` is loaded as a dependency.
Reproduction steps
----------------------------------------
```
drush dl drupal-8.7.x
cd drupal-8.7.x-dev
## Tangential workarounds
composer update psr/log
composer require 'pear/pear_exception:1.0.1 as 1.0.0' 'cache/integration-tests:dev-master#b97328797ab199f0ac933e39842a86ab732f21f9'
## Enable patching
## https://github.com/cweagans/composer-patches#allowing-patches-to-be-applied-from-dependencies
composer require cweagans/composer-patches:^1.5.0
composer config extra.enable-patching true
## Add CiviCRM
composer require civicrm/civicrm-core:5.22.x-dev
```
Current behaviour
----------------------------------------
As part of the output from `composer require ...` / `composer install`, we get:
```
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Installing zetacomponents/mail (dev-master 4dc71cc): Cloning 4dc71ccbcc from cache
- Applying patches for zetacomponents/mail
tools/scripts/composer/patches/civicrm-custom-patches-zetacompoents-mail.patch (CiviCRM Custom Patches for ZetaCompoents mail)
Could not apply patch! Skipping. The error was: The "tools/scripts/composer/patches/civicrm-custom-patches-zetacompoents-mail.patch" file could not be downloaded: failed to open stream: No such file or directory
...
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Installing phpoffice/common (0.2.9): Loading from cache
- Applying patches for phpoffice/common
tools/scripts/composer/patches/phpoffice-common-xml-entity-fix.patch (Fix handling of libxml_disable_entity_loader)
Could not apply patch! Skipping. The error was: The "tools/scripts/composer/patches/phpoffice-common-xml-entity-fix.patch" file could not be downloaded: failed to open stream: No such file or directory
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Installing phpoffice/phpword (0.15.0): Loading from cache
- Applying patches for phpoffice/phpword
tools/scripts/composer/patches/phpword-libxml-fix-global-handling.patch (Fix handling of libxml_disable_entity_loader)
Could not apply patch! Skipping. The error was: The "tools/scripts/composer/patches/phpword-libxml-fix-global-handling.patch" file could not be downloaded: failed to open stream: No such file or directory
```
Additionally, if you manually inspect the files `vendor/zetacomponents/mail/src/parser/interfaces/part_parser.php` and `vendor/phpoffice/common/src/Common/XMLReader.php`, you can see that the files appear unpatched.
Expected behaviour
----------------------------------------
The patches should apply.
Alternatively, if the patches are not needed, then perhaps we should remove them?
Environment information
----------------------------------------
* __PHP:__ 7.2.8
* __CMS:__ Drupal 8.7
* __Database:__ MySQL 5.7https://lab.civicrm.org/dev/core/-/issues/1615Migrate installers to "setup" API2024-02-07T01:05:15ZtottenMigrate installers to "setup" APIOverview
----------------------------------------
The aim of this filing is to do a round of consolidation/reconciliation in the Civi installer code: get rid of `install/*.php` and instead use `civicrm-setup`. The latter offers:
* Bett...Overview
----------------------------------------
The aim of this filing is to do a round of consolidation/reconciliation in the Civi installer code: get rid of `install/*.php` and instead use `civicrm-setup`. The latter offers:
* Better APIs and better organization - e.g. you can drill-down on most installation tasks by browsing the [plugins](https://github.com/civicrm/civicrm-setup/tree/master/plugins/) (esp `installFiles` and `installDatabase`); inspect the list of event-listeners; and [programmatically manage plugins](https://github.com/civicrm/civicrm-setup/blob/master/docs/plugins.md).
* More sensible UX - the list of options is more useful for a new admin initializing the system.
![Screen_Shot_2020-02-14_at_6.03.12_PM](/uploads/f107602f3ebd5f60b6870322711629a5/Screen_Shot_2020-02-14_at_6.03.12_PM.png)
Some background: https://civicrm.org/blog/totten/developers-revising-the-civicrm-installer-library-cli-wordpress
Use-cases
----------------------------------------
There are several different use-cases for performing installation; most permutations of the following variables are fair:
1. __UF/CMS__: Civi can be installed on top of Backdrop, Drupal 6/7/8, Joomla, WordPress.
2. __Medium__: Civi can be installed through:
* Web installation screens (e.g. `civicrm/install/index.php` and the Joomla installer)
* CLI installers (e.g. `cv core:install`)
* Scripts/packagers (e.g. Drupal "profiles", `civibuild`, `docker`)
3. __Options__: Civi can be configured with:
* CMS DB or separate DB
* Empty data or demo data
* "Components" (enabled/disabled)
* "Extensions" (enabled/disabled)
* "Settings" (defaults/overrides)
* Path/URL definitions (defaults/overrides)
Current behavior
----------------------------------------
There are separate installers. There are number of arbitrary differences and un-DRY things.
Proposed behavior
----------------------------------------
All installers use a PHP API for Civi installation.
Process
----------------------------------------
There are several things to consolidate here; it's not just one commit or PR. Suggested tasks (in approximate order):
* [x] __Move code from `civicrm-setup.git` into `civicrm-core.git`__ (*done, [#16618](https://github.com/civicrm/civicrm-core/pull/16618)*):
* `civicrm-setup` started as a separate repo so that I could iterate quickly in the early stages. However, as this gets rolled into more use-cases, it's likely to get referenced in more places (e.g. `civicrm-drupal-8.git`, `civicrm-drupal.git`, `civicrm-wordpress.git`, `civicrm-backdrop.git`), and it is also reasonable to expect a few small patches. Unfortunately, juggling small patches in that arrangement would be mentally draining. Migrating the code into `civicrm-core.git` will remove one layer of complexity.
* Like the API framework or DB framework, the *installer* is sufficiently important to `civicrm-core` go into the main repo.
* [x] __Move docs from `civicrm-setup.git` into `civicrm-dev-docs.git`__ (*done, see [Dev Guide: Setup Reference](https://docs.civicrm.org/dev/en/latest/framework/setup/)*)
* [ ] __Update Civi-WordPress__:
* [x] (Phase 1) Switch the default from `install/` UI to `civicrm-setup` UI. (*It's currently available opt-in. Switch to opt-out.*)
* [x] (Phase 2) Update `wp-cli/civicrm.php` so that WP-CLI uses `civicrm-setup` API.
* [ ] (Phase 3) Ensure that all tasks (e.g. registering base-pages or permissions) are called in WP-oriented `*.civi-setup.php` plugin.
* [ ] __Update Civi-D8/D9__
* [x] (Phase 1) Update `civicrm.install` to call `civicrm-setup` APIs (related: https://github.com/civicrm/civicrm-drupal-8/pull/37)
* [ ] (Phase 2) Update `civicrm.drush.inc` so that D7/BD use `civicrm-setup` API.
* [ ] __Update Civi-D7/BD__
* [x] (Phase 1) Update `install/index.php` so that D7/BD default to loading the `civicrm-setup` UI. The sysadmin docs may continue pointing admins to `install/index.php`. Optionally, allow one to use a parameter to opt-out and get the old UI.
* [ ] (Phase 2) Update `civicrm.drush.inc` so that D7/BD use `civicrm-setup` API.
* [x] __Update Civi-Joomla__
* [x] (Phase 2) Update to use `civicrm-setup` API
* [x] __Ignore Civi-D6__
* [ ] __Update civibuild__
* [ ] (Phase 2) Switch implementation from `civicrm_install()` to `civicrm_install_cv()` (related WIP: https://github.com/civicrm/civicrm-buildkit/pull/673)
* [ ] (Phase 3) Remove old implementation
* [ ] (Phase ??) Convert `civicrm_make_test_settings_php()` to a `civicrm-setup` plugin. Document an installer-option to initialize test/dev-harness.
* [ ] __Remove all opt-outs. Final pass to double-check/remove all remaining logic from `install/` folder__
* (*This is the last step, after all the others.*)https://lab.civicrm.org/dev/core/-/issues/1625IDN-Domain Emails - dont pass email check.2022-09-29T14:42:46ZRar9IDN-Domain Emails - dont pass email check.See Issue https://issues.civicrm.org/jira/browse/CRM-15975 and
Email validation is too restrictive (rejects IDN addresses) https://issues.civicrm.org/jira/browse/CRM-16313
Emails with these Characters should pass
https://www.denic.de/...See Issue https://issues.civicrm.org/jira/browse/CRM-15975 and
Email validation is too restrictive (rejects IDN addresses) https://issues.civicrm.org/jira/browse/CRM-16313
Emails with these Characters should pass
https://www.denic.de/en/know-how/idn-domains/idn-character-list/
Last tested with Civicrm 5.22.1 + Drupal 7
Php 7.2x Intl enabled.https://lab.civicrm.org/dev/core/-/issues/1647Make packages and bower_components tokens compataible with the CiviCRM Core R...2022-09-29T17:40:08ZseamusleeMake packages and bower_components tokens compataible with the CiviCRM Core Reources URL settingOne of the things that has been hightlighted by dev/Financial#120 is that since this commit https://github.com/civicrm/civicrm-core/commit/1143a7815b1abbcb7cfc0cab6c754e499d11d064#diff-ff63186dcddd8213d6b992efd57d70a4 from @totten not al...One of the things that has been hightlighted by dev/Financial#120 is that since this commit https://github.com/civicrm/civicrm-core/commit/1143a7815b1abbcb7cfc0cab6c754e499d11d064#diff-ff63186dcddd8213d6b992efd57d70a4 from @totten not all resources urls are treated the same. What that causes is that all bower_components and packages urls now are relative to the CiviCRM Core root path as calculated by getCiviSourceStorage rather than the CiviCRM Core Resources URL setting. Where as assets loaded from the root folder e.g. js/common.js etc would all still continue to respect the setting. This ticket is about unifying them back together in a way that is sensible. Or at least we should update the documentation and other respects to indicate that the setting no longer applies to packages and bower_components urlshttps://lab.civicrm.org/dev/core/-/issues/1726Allow financial types to not have Expense account defined2022-08-26T20:51:52ZJoeMurrayAllow financial types to not have Expense account definedThis is a proposal for discussion and refinement.
Overview
----------------------------------------
Simplify accounting configuration to remove requirement for, and default creation of, widely unused stuff. In particular, don't require ...This is a proposal for discussion and refinement.
Overview
----------------------------------------
Simplify accounting configuration to remove requirement for, and default creation of, widely unused stuff. In particular, don't require Expenses account for every financial type, nor create relations to Expense and Premium accounts by default when creating a financial type.
Example use-case
----------------------------------------
1. Click on **Administer > CiviContribute > Financial Types**.
1. Click **Add Financial Type**.
1. Enter **Name** and click **Save**.
1. In Financial Accounts, there are Banking Fees and Premiums accounts, which is **undesirable**.
1. Click **Accounts** on the new Financial Type row.
1. Beside the 'Expense Account is', click **Delete**, then confirm by clicking **Delete** again.
1. Click on **Contributions > New Contribution**.
1. Select the Financial Type created above that does not have an Expense Account set up for it anymore, fill in **Contributor** and **Total Amount**, and click **Save**.
1. Try to edit the contribution but not in a popup, for example, go to the contact's page, right click on the edit button for the contribution and Copy Link Address, then paste address into a new tab. You'll see "Sorry, due to an error, we are unable to fulfill your request at the moment. You may want to contact your administrator or service provider with more details about what action you were performing when this occurred.
One of parameters (value: ) is not of the type Integer". This is caused by missing Expense account, **even though it is not needed**.
Current behaviour
----------------------------------------
See above.
Proposed behaviour
----------------------------------------
On creation of Financial Type, no Expense or Premiums account relationship would be setup. On editing a contribution (with a line item) with a financial type without an Expense account relationship setup, no error would occur.
Comments
----------------------------------------
The expectation when this was implemented circa 2014 was that payment processors would all soon record banking fees. That hasn't been the case for a variety of reasons.Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/1770Pagination in Smart Group Alphabetic listings2023-03-15T09:15:59ZmarcusjwilsonPagination in Smart Group Alphabetic listingsOverview
----------------------------------------
When creating a Smart Group (in this case, via the Search Builder) and displaying Contacts within the Smart Group, if we choose to view the Contacts by Alphabet letter, and then paginate ...Overview
----------------------------------------
When creating a Smart Group (in this case, via the Search Builder) and displaying Contacts within the Smart Group, if we choose to view the Contacts by Alphabet letter, and then paginate through that letter (i.e. viewing page 2 of the contacts with surnames starting "H"), then, at the end of the list, the contacts from the start of the "H"s are repeated.
For instance, on page 2 of the "H"s in my smart group, I see that it's displaying "Contact 51 - 67 of 67". But after Contact number 67 it starts to display the "H"s from the start again, in positions 68 to 100 (which shouldn't exist in the view).
Reproduction steps
----------------------------------------
1. Create a Smart Group using Search Builder
2. Navigate to Contacts > Manage Groups. Find the Smart Group and view "Contacts"
3. Navigate to a letter of the alphabet for Contacts where the number of results returned > 50, and view the last page of listings.
Current behaviour
----------------------------------------
Smart Groups Contacts filtered by Letter repeating contacts on last page of listings.
Expected behaviour
----------------------------------------
Smart Groups Contacts filtered by Letter only display each contact once.
Environment information
----------------------------------------
CiviCRM 5.24.3
WordPress 5.4.1
Comments
----------------------------------------
I couldn't test this on the demo sites, as there aren't enough contacts in the demo site databases to display > 50 contacts per letter of the alphabet. Can contact data on demo sites be expanded?https://lab.civicrm.org/dev/core/-/issues/1783Wrong Employer Relationship retained on Merge.2023-03-18T06:41:18ZtommyboboWrong Employer Relationship retained on Merge.Overview
----------------------------------------
If Current Employer record is update to a duplicate that is later merged this results in an inactive relationship being the only employer relationship retained.
Tested on 5.25 but I am ...Overview
----------------------------------------
If Current Employer record is update to a duplicate that is later merged this results in an inactive relationship being the only employer relationship retained.
Tested on 5.25 but I am seeing evidence of this issue going back to 2017.
Reproduction steps
----------------------------------------
1. Contact A has a Current Employer of "Organization 1"
1. They make a contribution and set employer to "Organization One"
1. This sets the relationship to 1 to inactive.
1. CiviCRM Admin notices the error and merges "Organization One" into "Organization 1"
1. CiviCRM Merge saves the inactive "Organization 1" relationship deletes the active "Organization One" to prevent a duplicate, but saves "Organization One" to the Current Employer field.
1. Contact Record is left with inactive employer relationship.
Current behavior
----------------------------------------
CiviCRM is giving preference to Relationships in the retained record during a merge.
It should give preference to Active relationships if there is a conflict.
This may be minor, but a Current Employer fields are a significant source of duplicate records.https://lab.civicrm.org/dev/core/-/issues/1792Wrong status message when deleting a contact if undelete/trash is turned off2023-03-22T16:19:02ZDaveDWrong status message when deleting a contact if undelete/trash is turned off1. Go to Administer - System Settings - Misc
1. Set "Contact Trash and Undelete" to "No".
1. Go to any contact's page.
1. Click the Delete button near the top.
1. Confirm the deletion.
1. The contact is (permanently) deleted correctly, b...1. Go to Administer - System Settings - Misc
1. Set "Contact Trash and Undelete" to "No".
1. Go to any contact's page.
1. Click the Delete button near the top.
1. Confirm the deletion.
1. The contact is (permanently) deleted correctly, but two incorrect messages appear:
1. "[Name] has been moved to the trash."
(It should say permanently deleted, since you turned off trash.)
1. "A Contact with that ID does not exist: [id]"
(Because it's redirecting you to the contact view page for the contact that you just deleted.)
Another path to reproduce the problem is instead of step 3 above do Search - Find Contacts, then select one and from the actions dropdown choose Delete Contacts. It will say moved to trash but it should say permanently deleted. (Choosing Delete Permanently works ok with or without trash turned on.)
I think this might be an ok candidate for a newcomer to tackle, because it's pretty self-contained, not too abstract, and there's an existing unit test to update so provides a jumpstart to working with tests locally.
Relevant code is here:
1. [CRM/Contact/Form/Task/Delete.php in postProcess()](https://github.com/civicrm/civicrm-core/blob/e6d6149bad77c02001efceb0ef487f3ed3f7f8f1/CRM/Contact/Form/Task/Delete.php#L215)
2. [CRM/Contact/Form/Task/Delete.php in preProcess()](https://github.com/civicrm/civicrm-core/blob/e6d6149bad77c02001efceb0ef487f3ed3f7f8f1/CRM/Contact/Form/Task/Delete.php#L52)
3. [Existing test](https://github.com/civicrm/civicrm-core/blob/e6d6149bad77c02001efceb0ef487f3ed3f7f8f1/tests/phpunit/CRM/Contact/Form/Task/DeleteTest.php#L152-L155)
Aside: I'm curious how many people turn off trash because the message seems like something people would have noticed by now. I only noticed it while writing the test.https://lab.civicrm.org/dev/core/-/issues/1799Daily (and other) scheduled jobs slip forward each day until they're eventual...2023-06-17T17:35:14ZJonGoldDaily (and other) scheduled jobs slip forward each day until they're eventually missedIf this subject line looks familiar, [there's a reason](https://issues.civicrm.org/jira/browse/CRM-16276).
This issue was originally raised and [fixed in April 2015](https://github.com/civicrm/civicrm-core/commit/d0be1535d809b206890bcd4...If this subject line looks familiar, [there's a reason](https://issues.civicrm.org/jira/browse/CRM-16276).
This issue was originally raised and [fixed in April 2015](https://github.com/civicrm/civicrm-core/commit/d0be1535d809b206890bcd4f72ebd4ba8fb1c86d). However, a code simplification in December 2015 [removed the fix](https://github.com/civicrm/civicrm-core/commit/bda41fcbeaf535ff04d7e5cfc7145e83f50c17b2).
Many of us are familiar with this issue. Cron runs at 10:00:00, scheduled jobs are executed in order, and one job might not fire until 10:00:03. If the job fires hourly, we would expect the job to fire at 11:00:00, but since an hour hasn't elapsed since 10:00:00, the job doesn't fire until the *next* cron, maybe at 11:15:00. This has repercussions for a number of scheduled job use cases.
After some thought, I realized the correct solution is to record the time the *cron job* fires, not the time the job actually is started. I believe that when the `last_run` field was created, there was no sense that these times might be different, hence the bug.
I wrote a patch (I'm leaving it in production for a few days before submitting a PR) that does two things:
* `last_run` now refers to the time the cron job started, not the scheduled job.
* To avoid race-y conditions on very slow shared hosts, I force the `last_run` time to always have "00" as its seconds. This is a belt-and-suspenders (or belt-and-braces in non-'Merican) approach that reflects that a) shared hosting could be slow enough that this would otherwise be a problem, b) cron doesn't allow scheduled at the seconds level, so this shouldn't impact anyone's scheduling.JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/1807on hold is not respected when two contacts have duplicate email2022-10-10T09:16:44ZIan Kellingon hold is not respected when two contacts have duplicate emailOverview
----------------------------------------
If you have group with two contacts that have the same email, a mailing
will only send to one of them. Then, if that email bounces with the
right classification, one of contacts will hav...Overview
----------------------------------------
If you have group with two contacts that have the same email, a mailing
will only send to one of them. Then, if that email bounces with the
right classification, one of contacts will have their email put on
hold. The next mailing will then ignore that status and send an email to
the other contact which has the same email address.
Reproduction steps
----------------------------------------
Create two contacts that both have a email that doesn't exist and will
produce a bounce that civi understands. Create a group with those two
contacts. Send one or more mailings, processing the bounce until the
contact goes on hold. Then send another mailing.
Current behavior
----------------------------------------
The mailing goes out to the email address that is on hold.
Expected behavior
----------------------------------------
The mailing does not go to the on hold email address.
Environment information
----------------------------------------
* __CiviCRM:__ 5.24.3 with patches that don't affect this issue,
available at https://agpl.fsf.org/crm.fsf.org/CURRENT/
Comments
----------------------------------------
We have many duplicate contacts our database, often 10+ duplicates, thus
making bounce processing require 10+ times more bounces, making bounce
processing very broken. Civi allows users to create contacts with
duplicate emails with the right profile settings, etc, so it should be
able to handle the result without breaking bounce processing.
We are using a rule in our email system to detect these emails
and avoid sending them, it checks if this count is greater than 0:
```
select count(*) FROM civicrm_email where email = REPLACEME and on_hold = 1;
```https://lab.civicrm.org/dev/core/-/issues/1926Finish allowing use of SSL to connect to database2023-07-24T03:44:31ZDaveDFinish allowing use of SSL to connect to databaseFollowing on to complete what's left for https://lab.civicrm.org/dev/core/-/issues/1137, there are a couple confusing things so I'm listing them to try to get a handle on them:
| thing | comment
| ---- | ----
| [CRM_Utils_File](https://...Following on to complete what's left for https://lab.civicrm.org/dev/core/-/issues/1137, there are a couple confusing things so I'm listing them to try to get a handle on them:
| thing | comment
| ---- | ----
| [CRM_Utils_File](https://github.com/civicrm/civicrm-core/blob/6eced9e6cedd66769757a30415a4c6c99fd0d231/CRM/Utils/File.php#L327) | This block seems to do almost the same thing as CRM_Core_DAO::init(), but not completely, but I think also has a "bug" where it wipes the sql_mode instead of updating it. This is called from e.g. the upgrade process, but it's not clear yet why that would require its own thing, since civi is already installed. Will test this out and see.
| [CRM_Utils_System::authenticate](https://github.com/civicrm/civicrm-core/blob/da6c68c61ab13ba434a5061701c3acfd8676cc29/CRM/Utils/System.php#L727) | It calls DB::connect assuming it's already available, and then does nothing with it, and then a couple milliseconds later the UF class does it again. Is this just leftover from refactoring, maybe as suggested in the comment a couple lines above. Additionally, only drupal 7 seems to use it, but it's not clear if it's really necessary and why can't it just use cms functions in the cms class, which it does in drupal 8 and the others.
| install/ | Would like to leave this out if the future plan is to move to a civicrm-setup variant. Otherwise it means adding fields to the form for example.https://lab.civicrm.org/dev/core/-/issues/1984Sometimes Custom fields missing on profile2023-06-05T20:22:11ZPradeep Nayakpradpnayak@gmail.comSometimes Custom fields missing on profileCustom fields are not rendered on profile front end pages like contribution or event pages. As soon as civi cache is cleared they start appearing.
```
Php - 7.3
opcahe disabled
Clean-up Temporary Data and Files schedule job executed hou...Custom fields are not rendered on profile front end pages like contribution or event pages. As soon as civi cache is cleared they start appearing.
```
Php - 7.3
opcahe disabled
Clean-up Temporary Data and Files schedule job executed hourly by cron job
```https://lab.civicrm.org/dev/core/-/issues/3313Unable to save a membership with a text price field2023-12-19T22:18:09ZDaveDUnable to save a membership with a text price fieldSame problem in 5.28 and master and can reproduce on dmaster.demo.civicrm.org. Or maybe I'm missing something.
1. Create a price set used for memberships.
1. Set financial type to member dues.
1. Add one text field (the default input fi...Same problem in 5.28 and master and can reproduce on dmaster.demo.civicrm.org. Or maybe I'm missing something.
1. Create a price set used for memberships.
1. Set financial type to member dues.
1. Add one text field (the default input field type).
1. Amount doesn't matter - put in some number.
1. Leave other fields at defaults.
1. Create a new membership.
1. Choose the price set.
1. Put in quantity 1.
1. Can leave the rest at defaults.
1. Click Save.
1. ERROR: Select at least one membership option.
It works with a radio field.https://lab.civicrm.org/dev/core/-/issues/2021Contribution receipt has donor's name printed twice, once with non-printable ...2023-11-23T07:22:03ZBobSContribution receipt has donor's name printed twice, once with non-printable charactersOverview
----------------------------------------
Contribution receipts, notably those created with the Stripe extension, display the donor's information as follows:
```
Jane�Mary�Smith <$first . \x01 . $middle . \x01 . $last>
...Overview
----------------------------------------
Contribution receipts, notably those created with the Stripe extension, display the donor's information as follows:
```
Jane�Mary�Smith <$first . \x01 . $middle . \x01 . $last>
Jane Mary Smith <$first . \x20 . $middle . \x20 . $last>
123 Main St
...
```
The two \x01 characters (ASCII SOH, CRM_Core_DAO::VALUE_SEPARATOR) are displayed as boxes in some email clients, although they aren't displayable in this bug report.
The message template "Contributions - Receipt (on-line)" includes:
```
{if $billingName}
<tr>
<th {$headerStyle}>
{ts}Billing Name and Address{/ts}
</th>
</tr>
<tr>
<td colspan="2" {$valueStyle}>
{$billingName}<br />
{$address|nl2br}<br />
{$email}
</td>
</tr>
{elseif $email}
```
For Stripe contributions, $billingName contains the first/middle/last names separated by SOH characters, as found in civicrm_address.name. The {if $billingName} block therefore gets printed, including both $billingName and the $address variable which contains the second instance of the donor's name, followed by their address.
The SOH characters in the first line of $address (the donor's name) are replaced with spaces in CRM_Core_BAO_Address::addDisplay().
Contributions via Paypal standard, in contrast, do not display the $billingName block as addresses created that way store NULL in civicrm_address.name.
Reproduction steps
----------------------------------------
1. Create a contribution with CiviCRM Stripe extension v. 6.4.2.
1. Observe the Billing Name and Address section of the emailed receipt.
Current behaviour
----------------------------------------
The donor's name is printed twice. Once with SOH separators, and again with spaces. If the middle name is blank, the second instance contains two spaces between first and last name.
Expected behaviour
----------------------------------------
The name should be printed once, with a single space between non-blank name components.
Additional comments
----------------------------------------
1. Perhaps the fix is to simply remove "$billingName\<br \/\>" from the template(s). Ideally, the double spaces in the case of a null middle name should be corrected also.
2. Event and Membership templates seem to have the same structure as the template shown above.
3. The Thank You page displays the donor's name as expected -- only once, with a single \x20 character between first and last name.
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __CiviCRM:__ 5.28.2
* __CiviCRM Stripe extension__ v. 6.4.2
* __mjwshared extension__ v. 0.8.1
* __PHP:__ 7.3
* __CMS:__ Drupal 7.72https://lab.civicrm.org/dev/core/-/issues/2072Why does composer.json contain `"include-path": ["vendor/tecnickcom"]`2023-05-10T12:51:31ZDaveDWhy does composer.json contain `"include-path": ["vendor/tecnickcom"]`As per https://lab.civicrm.org/extensions/cdntaxreceipts/-/issues/110#note_47514 it seems to cause issues with extensions that try to use TCPDF in drupal 8 since composer creates a vendor/composer/include_paths.php file that then makes t...As per https://lab.civicrm.org/extensions/cdntaxreceipts/-/issues/110#note_47514 it seems to cause issues with extensions that try to use TCPDF in drupal 8 since composer creates a vendor/composer/include_paths.php file that then makes the path relative to /vendor/civicrm/civicrm-core, instead of relative to just /vendor.
Maybe the line was from an early iteration of autoloading and using composer?https://lab.civicrm.org/dev/core/-/issues/2114Trigger-based logging doesn't log if just changing a letter to upper/lower case2023-06-29T12:55:38ZDaveDTrigger-based logging doesn't log if just changing a letter to upper/lower case1. Turn on logging. (Admin - System Settings - Misc - Logging).
1. Change the first name of a contact just changing e.g. the first letter from upper to lower case.
1. Either look in the database in log_civicrm_contact or go to CiviReport...1. Turn on logging. (Admin - System Settings - Misc - Logging).
1. Change the first name of a contact just changing e.g. the first letter from upper to lower case.
1. Either look in the database in log_civicrm_contact or go to CiviReports - Contact Reports - Contact Logging.
1. Change has not been logged.
It's because the standard collation for civi is the case-insensitive xxx_unicode_ci or possibly xxx_general_ci. You might have also changed it manually to use the same as your CMS but is likely still case-insensitive. And in mysql 8 the default utf8mb4_0900_ai_ci is also accent-insensitive (i.e. `é` is treated the same as `e`), if you manually changed your civi tables to use that.
The logging triggers should probably use `xxx_bin`. Related but separate: https://github.com/civicrm/civicrm-core/pull/18721https://lab.civicrm.org/dev/core/-/issues/2122Account for time zone on event registration pages2023-11-05T14:13:53ZwmortadaAccount for time zone on event registration pagesOverview
----------------------------------------
There doesn't appear to be a way to account for the time zone on an event registration page.
This is less of an issue for a local event, but is quite a problem for a global online event....Overview
----------------------------------------
There doesn't appear to be a way to account for the time zone on an event registration page.
This is less of an issue for a local event, but is quite a problem for a global online event.
At a minimum I would expect there to be an option to set the time zone for the event and for the time zone to display on the event registration page. (Ideally you'd want the page to show the time in the user's time zone.)
Current behaviour
----------------------------------------
There is no option to set the time zone for an event. (I presume that the system time zone is used but I'm not sure.)
![image](/uploads/6727ccb71e4b3129120a3275240c56cf/image.png)
The time zone isn't displayed on the event registration page.
![image](/uploads/a16d1c595ab9020e2d9e9c2d668e114c/image.png)
Proposed behaviour
----------------------------------------
There is an option to set the time zone when creating an event. This should default to the user's time zone.
The time zone is displayed on the event registration page.
Comments
----------------------------------------
I became aware of this when trying to register for this event: https://civicrm.org/civicrm/event/info?reset=1&id=1553
Probably relates to https://lab.civicrm.org/dev/core/-/issues/441
Agileware Ref: CIVICRM-404 (feature not found)https://lab.civicrm.org/dev/core/-/issues/2157Array to string conversion in DB_DataObject when doing on behalf of and no st...2023-05-26T16:13:04ZDaveDArray to string conversion in DB_DataObject when doing on behalf of and no state/provinceIf you submit a contribution on behalf of an org and leave the state/province blank, you get `Notice: Array to string conversion in DB_DataObject->_build_condition() (line 2903 of .../web/sites/all/modules/civicrm/packages/DB/DataObject....If you submit a contribution on behalf of an org and leave the state/province blank, you get `Notice: Array to string conversion in DB_DataObject->_build_condition() (line 2903 of .../web/sites/all/modules/civicrm/packages/DB/DataObject.php)`
(At the moment to see this you need to make the field not required on the profile, but it happens either way it's just there's currently a separate issue when the field is required and the country has no state/provinces.)
What happens is that at that code line `$this->name` and `$this->abbreviation` contain a big long list of numbers and letters, including, oddly, the strings "_A", "_B", "_C", etc, but ONLY up to "_P".https://lab.civicrm.org/dev/core/-/issues/2163log_date is missing on logging detail report2023-05-26T16:05:01ZDaveDlog_date is missing on logging detail report1. Turn on logging at Admin - System Settings - Misc.
2. Update a contact.
3. Go to the logging summary civireport.
4. Click on "Update" in the row for your update.
5. At the top of the detail report it will say something like `Change to...1. Turn on logging at Admin - System Settings - Misc.
2. Update a contact.
3. Go to the logging summary civireport.
4. Click on "Update" in the row for your update.
5. At the top of the detail report it will say something like `Change to blah made by somebody on :`
6. It's supposed to show the modified date at the end.
It's coming from [this check](https://github.com/civicrm/civicrm-core/blob/4d660b8e1e6ac980a06b096fbc4cde1e1666e0b9/CRM/Report/Form/Contact/LoggingSummary.php#L193) which decides the log date isn't needed. The comment when that check was added says it can make the report less accurate because of some older data that used to be the way things were logged. So I'm hesitant to just remove that check.
Some other options include passing it but with a different name, and then in the detail report know that that new name is just used in intro text. Or looking it up in the detail report.
TBD.
@VangelisP just FYI.https://lab.civicrm.org/dev/core/-/issues/2195Custom fields of type money and drop-down list with round numbers don't updat...2023-07-04T12:55:04ZDaveDCustom fields of type money and drop-down list with round numbers don't update existing values when you change the option valueCame up reviewing https://github.com/civicrm/civicrm-core/pull/18959
For example:
1. Edit or create a custom field group for contacts.
1. Add a custom field of type money and make it a drop-down select list.
1. For the values, make cho...Came up reviewing https://github.com/civicrm/civicrm-core/pull/18959
For example:
1. Edit or create a custom field group for contacts.
1. Add a custom field of type money and make it a drop-down select list.
1. For the values, make choice 1 have value 10 and choice 2 have value 20.
1. Create a contact and pick one of the choices. Save.
1. Note that in the database in the civicrm_value_XX table this has been stored as 10.00, with the decimals. Note that in civicrm_option_value that the choice is stored as 10, without the decimals.
1. Go and edit the option value and change the value to anything else.
1. What's supposed to happen is that the contact's custom field values should be updated to match. But because of the mismatch above it doesn't change it in civicrm_value_XX and it now appears blank when viewing the contact.
I don't think this is recent.https://lab.civicrm.org/dev/core/-/issues/2225Add a contact.checksum_raw token that does not include "cs="2023-09-24T22:53:45ZherbdoolAdd a contact.checksum_raw token that does not include "cs="Unlike all the other tokens which only include the actual data, such as `{contact.contact_id}`, the `{contact.checksum}` also includes the parameter name. There doesn't seem to be any good reason other than that is how it was always done...Unlike all the other tokens which only include the actual data, such as `{contact.contact_id}`, the `{contact.checksum}` also includes the parameter name. There doesn't seem to be any good reason other than that is how it was always done.
I suppose it's not possible to fix this token at this point since lots of existing links rely on it. But perhaps a new token? A bit messy but I need something like `{contact.checksum.raw}`https://lab.civicrm.org/dev/core/-/issues/2237Provide some way for an extension's database tables to automatically get incl...2022-08-24T18:56:45ZDaveDProvide some way for an extension's database tables to automatically get included when a site runs utf8mb4 conversionThere's an option in the api call / api explorer when running the utf8mb4 conversion to specify some tablename patterns, but this requires the user to know what the tables are that all their extensions create.
One idea is a hook that le...There's an option in the api call / api explorer when running the utf8mb4 conversion to specify some tablename patterns, but this requires the user to know what the tables are that all their extensions create.
One idea is a hook that lets an extension advertise the names of tables it creates. If it wants to manage the charset itself for a table then it can just not list it.
Another is a standard naming like civicrm_ext_FOO, which then core knows not to conflict for its own tables, but which starts with civicrm so gets picked up by default.
Open to other ideas. The status quo is also technically an option. And it's only important if there's some connection between text data in the extension's tables and core tables, or it stores emojis and such.https://lab.civicrm.org/dev/core/-/issues/2255Proposal, Allow the CiviCRM CKEditor to be disabled for specific components a...2023-06-07T05:38:36Zjustinfreeman (Agileware)Proposal, Allow the CiviCRM CKEditor to be disabled for specific components as there is only a global on/off capability currently. We need CKEditor to be disabled for Message Templates, but not for Mailings, Events etc.Proposal, Allow the CiviCRM CKEditor to be disabled for specific components as there is only a global on/off capability currently. We need CKEditor to be disabled for Message Templates, but not for Mailings, Events etc.
CKEditor is alwa...Proposal, Allow the CiviCRM CKEditor to be disabled for specific components as there is only a global on/off capability currently. We need CKEditor to be disabled for Message Templates, but not for Mailings, Events etc.
CKEditor is always displayed for custom Message Templates which in our experience reliably mangles the HTML and Smarty Tags. CKEditor is NOT disabled for "System" Message Templates and therefore does not have this issue.
Agileware Ref: CIVICRM-1609 CIVICRM-1608https://lab.civicrm.org/dev/core/-/issues/2266Wording beside on-behalf-of checkbox isn't required, but has no default eithe...2023-11-23T07:25:09ZDaveDWording beside on-behalf-of checkbox isn't required, but has no default either, so appears as a mysterious checkboxI'm not sure yet if this is recent. I remember there being a default but it's not a strong memory.
1. Create a contribution page.
1. On the title page there's a choice to "Allow individuals to contribute and / or signup for membership o...I'm not sure yet if this is recent. I remember there being a default but it's not a strong memory.
1. Create a contribution page.
1. On the title page there's a choice to "Allow individuals to contribute and / or signup for membership on behalf of an organization". Check it.
1. In the section that appears there's a box for a label that has no default and is not required. Leave it blank.
1. Finish off the rest of the settings - doesn't matter too much what.
1. Visit the contribution page. It looks like this:
![Untitled](/uploads/4776b6636a1ea3fb9106face184bd545/Untitled.png)
It's easily fixable by adding words in the label box, but it just a seems a wrong default and I don't remember it coming up as a problem before.
I notice the label box is a textarea for some reason. Maybe it used to be a normal textbox and so the default now isn't getting set correctly?
Also ignore the decimal places in the dollar amounts in the screenshot - that's a known windows issue with civi.https://lab.civicrm.org/dev/core/-/issues/2290Look at REMOVING id column from cache tables2022-12-13T11:58:44ZjitendraLook at REMOVING id column from cache tablesThis is a continuation of the "In Progress" JIRA https://issues.civicrm.org/jira/browse/CRM-19385
which aimed at removing the id from the cache tables but unfortunately was never completed since I see we still have id column in all the ...This is a continuation of the "In Progress" JIRA https://issues.civicrm.org/jira/browse/CRM-19385
which aimed at removing the id from the cache tables but unfortunately was never completed since I see we still have id column in all the cache tables.
```
civicrm_acl_cache
civicrm_acl_contact_cache
civicrm_cache
civicrm_group_contact_cache
civicrm_prevnext_cache
civicrm_relationship_cache
```
Related PRs -
- https://github.com/civicrm/civicrm-core/pull/9743 (Closed)
- https://github.com/civicrm/civicrm-core/pull/9801 (Closed)
- https://github.com/civicrm/civicrm-core/pull/10019 (Merged) - This removed the dependency in the code so think we can start by removing the column directly.
Thoughts @eileen @johnkhttps://lab.civicrm.org/dev/core/-/issues/2357CiviCRM Dedupe Rules, default rules are incorrectly labelled as "and" when ea...2023-03-18T04:18:24Zjustinfreeman (Agileware)CiviCRM Dedupe Rules, default rules are incorrectly labelled as "and" when each rule is actually "or" eg. "Name and Email" should be "Name OR Email". Use of Name is ambiguous. Add "Nickname" to default Organisation dedupe rulesCiviCRM Dedupe Rules, default rules are incorrectly labelled as "and" when each rule is actually "or" eg. "Name and Email" should be "Name OR Email". Use of Name is ambiguous. Add "Nickname" to default Organisation dedupe rules.
Additio...CiviCRM Dedupe Rules, default rules are incorrectly labelled as "and" when each rule is actually "or" eg. "Name and Email" should be "Name OR Email". Use of Name is ambiguous. Add "Nickname" to default Organisation dedupe rules.
Additionally, Name is ambiguous in the context of Household and Organisation Contacts. Should be Household Name and Organisation Name. Organisations for example have Organisation Name and Nickname.
Suggested changes:
1. Change label to use "or" to match the dedupe criteria when or logic applies.
2. Remove ambiguity and be explicit about Name, ie. Household Name, Organisation Name and for Individuals: First Name and Last Name.
3. Add Nickname field to default Organisation dedupe rules.
![Screenshot_20210204_090434](/uploads/0c5b3b687825535aea7e8f0fdc814bf7/Screenshot_20210204_090434.png)
Agileware Ref: CIVICRM-1660https://lab.civicrm.org/dev/core/-/issues/2435Save fields button on export screen is not permissioned2023-05-24T06:21:51ZandyburnsSave fields button on export screen is not permissionedI cannot find in the code where the _Save Fields_ button is and therefore what permission is needed to save export mappings.
![image](/uploads/e9be7ed19215554edc7e6397dde46518/image.png)
Currently there is no core permission on who ca...I cannot find in the code where the _Save Fields_ button is and therefore what permission is needed to save export mappings.
![image](/uploads/e9be7ed19215554edc7e6397dde46518/image.png)
Currently there is no core permission on who can export. There is [this extension](https://github.com/progressivetech/net.ourpowerbase.exportpermission/blob/master/README.md) that hides the seach action of export but does not effect the actual export mappings screen.
Currently, non-admins try to save a field mapping and it just spins causing user confusion. What permission is currently needed to save field mappings? I think the logical solution is to get a permission added on saving field mappings button so users that don't have that permission do not see it.https://lab.civicrm.org/dev/core/-/issues/2467[Meta] XLTS Angular JS longer-term-support2022-12-20T18:24:57Zhomotechsual[Meta] XLTS Angular JS longer-term-support@JoeMurray raised in chat that [AngularJS goes EoL in December](https://chat.civicrm.org/civicrm/pl/cm6jqmcex3n39ekf3zzx4tiree) there is a company [XLTS](https://xlts.dev/angularjs) providing extended LTS support through 2026. I've send ...@JoeMurray raised in chat that [AngularJS goes EoL in December](https://chat.civicrm.org/civicrm/pl/cm6jqmcex3n39ekf3zzx4tiree) there is a company [XLTS](https://xlts.dev/angularjs) providing extended LTS support through 2026. I've send them an enquiry asking how much they are likely to look to charge given the open-source nature of CiviCRM to see if, perhaps, they'd be willing to "donate" the XLTS to us.
We lose nothing by asking!JoeMurrayJoeMurrayhttps://lab.civicrm.org/dev/core/-/issues/2546Add Settings, Disable, Delete buttons to group contacts listing page2023-09-14T15:09:10ZlarsssandergreenAdd Settings, Disable, Delete buttons to group contacts listing pageCurrently, you can only change settings, disable or delete a group from the Manage Groups page. It would make sense to be able to change the settings at least on the page that lists all the contacts in a group (disable and delete are les...Currently, you can only change settings, disable or delete a group from the Manage Groups page. It would make sense to be able to change the settings at least on the page that lists all the contacts in a group (disable and delete are less important, but I think they might as well be added as well). I often find I click into a group when I want to make it a mailing list and then realize I have to go back to the Manage Groups page and click settings.
I think this would make the most sense at the top of the page, maybe right below the "Add Contacts to this group" button or alternately, just above select records. Relatedly, I would also propose to put the "Edit Smart Group Criteria for this group" and "Add Contacts to this group" buttons on the same line, as they are currently on two.
For reference:
![image](/uploads/b7ed25111565401ddf2ae295795f3b8c/image.png)
I'll give this a go if it is supported.https://lab.civicrm.org/dev/core/-/issues/2571Move reCAPTCHA to core extension2022-12-02T09:28:56Zmattwiremjw@mjwconsult.co.ukMove reCAPTCHA to core extensionThis is a tracking issue to track progress.
* https://github.com/civicrm/civicrm-core/pull/19967 - initial extension merged.
* https://github.com/civicrm/civicrm-core/pull/19999 - add recaptcha to distmaker - merged.
* https://github.co...This is a tracking issue to track progress.
* https://github.com/civicrm/civicrm-core/pull/19967 - initial extension merged.
* https://github.com/civicrm/civicrm-core/pull/19999 - add recaptcha to distmaker - merged.
* https://github.com/civicrm/civicrm-core/pull/20011 - move recaptcha library to extension - merged.
* https://github.com/civicrm/civicrm-core/pull/20019 - Use standard function to add reCAPTCHA to PCPAccount form - merged.
Need to cleanup and move all calls to add reCAPTCHA to forms to the extension.https://lab.civicrm.org/dev/core/-/issues/2572Prevent unintentional removal from group for anonymous users filing out profiles2023-09-01T19:44:06ZlarsssandergreenPrevent unintentional removal from group for anonymous users filing out profilesCurrently, if an anonymous contact submits a profile that contains a groups field and the profile submission is deduped on submission with an existing contact, the group selections in the profile overwrite the contact's previous groups, ...Currently, if an anonymous contact submits a profile that contains a groups field and the profile submission is deduped on submission with an existing contact, the group selections in the profile overwrite the contact's previous groups, for the groups that are included in the profile (i.e. the public groups). The result is that if someone is subscribed to a mailing list and fills in a profile anonymously and doesn't check the same group name, they will be unsubscribed from the group. This is definitely not what users expect! I think it's clear that checking the box will add you to the group, but I doubt many expect that they will be removed from a group if they don't check the box that they've checked in the past.
This is also not what happens with other profile fields that are left blank, nor is it what happens if two contacts are merged manually.
I propose to ignore group checkboxes that are not checked in a profile submitted anonymously. No change for profiles submitted with known cids (as these will have the appropriate checkboxes already checked, allowing the contact to unsubscribe by unchecking them if desired). Currently, all group ids in the profile are being [passed here](https://github.com/civicrm/civicrm-core/blob/5db0bc3c1f54eaca4307f103a73bda596ae914d6/CRM/Contact/BAO/GroupContact.php#L534), removing the contact from groups that were unchecked. Will submit PR if supported.
EDIT: Here is some discussion on StackExchange: [Unwanted unsubscribe in profile](https://civicrm.stackexchange.com/questions/39403/unwanted-unsubscribe-in-profile/39406), [Anonymous user filing out a profile unintentionally remove themselves from groups](https://civicrm.stackexchange.com/questions/29337/anonymous-user-filing-out-a-profile-unintentionally-remove-themselves-from-group)https://lab.civicrm.org/dev/core/-/issues/2595Change Log Tab Excrutiatingly Slow - Poorly Performing Query and Fix (from 8 ...2023-08-14T13:09:44ZgordanChange Log Tab Excrutiatingly Slow - Poorly Performing Query and Fix (from 8 minutes down to 4 seconds)The page takes about 8 minutes to return which is absurdly slow for anything expected to be remotely interactive. It is so slow that my client is referring to it as the "Triangle of Doom".
I tracked it down to this query:
```
INSERT IG...The page takes about 8 minutes to return which is absurdly slow for anything expected to be remotely interactive. It is so slow that my client is referring to it as the "Triangle of Doom".
I tracked it down to this query:
```
INSERT IGNORE INTO civicrm_tmp_e_logsummary_ffa3ac146126d178ade367f2a5d17bf5
SELECT activity_id, IF (entity_log_civireport.log_action = 'Insert' AND extra_table.activity_type_id = 51 , GROUP_CONCAT(entity_log_civireport.contact_id), 1) , entity_log_civireport.log_action as log_civicrm_entity_log_action, 'log_civicrm_activity_contact' as log_civicrm_entity_log_type, entity_log_civireport.log_user_id as log_civicrm_entity_log_user_id, entity_log_civireport.log_date as log_civicrm_entity_log_date, modified_contact_civireport.display_name as log_civicrm_entity_altered_contact, modified_contact_civireport.id as log_civicrm_entity_altered_contact_id, entity_log_civireport.log_conn_id as log_civicrm_entity_log_conn_id, modified_contact_civireport.is_deleted as log_civicrm_entity_is_deleted, altered_by_contact_civireport.display_name as altered_by_contact_display_name
FROM staging_civicrm.log_civicrm_activity_contact entity_log_civireport
JOIN civicrm_contact modified_contact_civireport ON (entity_log_civireport.contact_id = modified_contact_civireport.id )
JOIN staging_civicrm.log_civicrm_activity extra_table ON extra_table.id = entity_log_civireport.activity_id
LEFT JOIN civicrm_contact altered_by_contact_civireport ON (entity_log_civireport.log_user_id = altered_by_contact_civireport.id)
WHERE modified_contact_civireport.id = 338520 AND
entity_log_civireport.log_action != 'Initialization'
GROUP BY entity_log_civireport.log_conn_id,
entity_log_civireport.log_user_id,
EXTRACT(DAY_MICROSECOND FROM entity_log_civireport.log_date),
entity_log_civireport.id
ORDER BY entity_log_civireport.log_date DESC;
EXPLAIN shows:
```
```
+------+-------------+-------------------------------+--------+---------------+---------+---------+---------------------------------------------------+---------+-------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+--------+---------------+---------+---------+---------------------------------------------------+---------+-------------------------------------------------+
| 1 | SIMPLE | modified_contact_civireport | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | extra_table | ALL | NULL | NULL | NULL | NULL | 3014020 | |
| 1 | SIMPLE | entity_log_civireport | ALL | NULL | NULL | NULL | NULL | 5518537 | Using where; Using join buffer (flat, BNL join) |
| 1 | SIMPLE | altered_by_contact_civireport | eq_ref | PRIMARY | PRIMARY | 4 | staging_civicrm.entity_log_civireport.log_user_id | 1 | Using where |
+------+-------------+-------------------------------+--------+---------------+---------+---------+---------------------------------------------------+---------+-------------------------------------------------+
```
The fix passes:
First pass:
```
ALTER TABLE log_civicrm_activity_contact ADD INDEX index_activity_id (activity_id);
```
With no further changes, this alone makes the above query go from 8 minutes to 1m45s.
Explain plain:
```
+------+-------------+-------------------------------+--------+-------------------+-------------------+---------+---------------------------------------------------+---------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+--------+-------------------+-------------------+---------+---------------------------------------------------+---------+---------------------------------+
| 1 | SIMPLE | modified_contact_civireport | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | extra_table | ALL | NULL | NULL | NULL | NULL | 3014020 | Using where |
| 1 | SIMPLE | entity_log_civireport | ref | index_activity_id | index_activity_id | 5 | staging_civicrm.extra_table.id | 1 | Using where |
| 1 | SIMPLE | altered_by_contact_civireport | eq_ref | PRIMARY | PRIMARY | 4 | staging_civicrm.entity_log_civireport.log_user_id | 1 | Using where |
+------+-------------+-------------------------------+--------+-------------------+-------------------+---------+---------------------------------------------------+---------+---------------------------------+
```
Second pass:
```
ALTER TABLE log_civicrm_activity ADD INDEX index_id (id);
```
Change the JOIN order explicitly and add a hint for the query optimizer to not re-order the JOINs:
```
SELECT STRAIGHT_JOIN activity_id, IF (entity_log_civireport.log_action = 'Insert' AND extra_table.activity_type_id = 51 , GROUP_CONCAT(entity_log_civireport.contact_id), 1) , entity_log_civireport.log_action as log_civicrm_entity_log_action, 'log_civicrm_activity_contact' as log_civicrm_entity_log_type, entity_log_civireport.log_user_id as log_civicrm_entity_log_user_id, entity_log_civireport.log_date as log_civicrm_entity_log_date, modified_contact_civireport.display_name as log_civicrm_entity_altered_contact, modified_contact_civireport.id as log_civicrm_entity_altered_contact_id, entity_log_civireport.log_conn_id as log_civicrm_entity_log_conn_id, modified_contact_civireport.is_deleted as log_civicrm_entity_is_deleted, altered_by_contact_civireport.display_name as altered_by_contact_display_name
FROM civicrm_contact modified_contact_civireport
JOIN staging_civicrm.log_civicrm_activity_contact entity_log_civireport ON entity_log_civireport.contact_id = modified_contact_civireport.id
JOIN staging_civicrm.log_civicrm_activity extra_table ON extra_table.id = entity_log_civireport.activity_id
LEFT JOIN civicrm_contact altered_by_contact_civireport ON entity_log_civireport.log_user_id = altered_by_contact_civireport.id
WHERE modified_contact_civireport.id = 338520 AND
entity_log_civireport.log_action != 'Initialization'
GROUP BY entity_log_civireport.log_conn_id,
entity_log_civireport.log_user_id,
EXTRACT(DAY_MICROSECOND FROM entity_log_civireport.log_date),
entity_log_civireport.id
ORDER BY entity_log_civireport.log_date DESC;
```
New EXPLAIN:
```
+------+-------------+-------------------------------+--------+-------------------+----------+---------+---------------------------------------------------+---------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+--------+-------------------+----------+---------+---------------------------------------------------+---------+---------------------------------+
| 1 | SIMPLE | modified_contact_civireport | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | entity_log_civireport | ALL | index_activity_id | NULL | NULL | NULL | 5518537 | Using where; Using filesort |
| 1 | SIMPLE | extra_table | ref | index_id | index_id | 5 | staging_civicrm.entity_log_civireport.activity_id | 1 | |
| 1 | SIMPLE | altered_by_contact_civireport | eq_ref | PRIMARY | PRIMARY | 4 | staging_civicrm.entity_log_civireport.log_user_id | 1 | Using where |
+------+-------------+-------------------------------+--------+-------------------+----------+---------+---------------------------------------------------+---------+---------------------------------+
```
This gets it down to 4 seconds!
Since the first index we started with is no longer getting used in the final variant, we can just not add it.
Summary:
To fix "Change Log" tab taking forever to load, the following fix is needed:
Add index:
```
ALTER TABLE log_civicrm_activity ADD INDEX index_id (id);
```
Make the code emit the query modified as follows:
```
INSERT IGNORE INTO civicrm_tmp_e_logsummary_ffa3ac146126d178ade367f2a5d17bf5
SELECT STRAIGHT_JOIN activity_id, IF (entity_log_civireport.log_action = 'Insert' AND extra_table.activity_type_id = 51 , GROUP_CONCAT(entity_log_civireport.contact_id), 1) , entity_log_civireport.log_action as log_civicrm_entity_log_action, 'log_civicrm_activity_contact' as log_civicrm_entity_log_type, entity_log_civireport.log_user_id as log_civicrm_entity_log_user_id, entity_log_civireport.log_date as log_civicrm_entity_log_date, modified_contact_civireport.display_name as log_civicrm_entity_altered_contact, modified_contact_civireport.id as log_civicrm_entity_altered_contact_id, entity_log_civireport.log_conn_id as log_civicrm_entity_log_conn_id, modified_contact_civireport.is_deleted as log_civicrm_entity_is_deleted, altered_by_contact_civireport.display_name as altered_by_contact_display_name
FROM civicrm_contact modified_contact_civireport
JOIN staging_civicrm.log_civicrm_activity_contact entity_log_civireport ON entity_log_civireport.contact_id = modified_contact_civireport.id
JOIN staging_civicrm.log_civicrm_activity extra_table ON extra_table.id = entity_log_civireport.activity_id
LEFT JOIN civicrm_contact altered_by_contact_civireport ON entity_log_civireport.log_user_id = altered_by_contact_civireport.id
WHERE modified_contact_civireport.id = 338520 AND
entity_log_civireport.log_action != 'Initialization'
GROUP BY entity_log_civireport.log_conn_id,
entity_log_civireport.log_user_id,
EXTRACT(DAY_MICROSECOND FROM entity_log_civireport.log_date),
entity_log_civireport.id
ORDER BY entity_log_civireport.log_date DESC;
```
Not only does this speed it up from 8 minutes down to 4 seconds, it also doesn't wreak havoc with row locking where every row scanned gets locked by the transaction engine, potentially resulting in a massive query pile-up in the database.https://lab.civicrm.org/dev/core/-/issues/2597Tabset for manage contribution pages or manage events requires Classname to e...2023-08-07T15:13:57ZlarsssandergreenTabset for manage contribution pages or manage events requires Classname to equal pathIf you add a tab to a tabset for Configure Event, it will work fine with the last word of the class name different from the last word of the path, except for trying to save, which will save, but redirect to Manage Events instead of stayi...If you add a tab to a tabset for Configure Event, it will work fine with the last word of the class name different from the last word of the path, except for trying to save, which will save, but redirect to Manage Events instead of staying on the same tab. This is because [code here](https://github.com/civicrm/civicrm-core/blob/5db0bc3c1f54eaca4307f103a73bda596ae914d6/CRM/Event/Form/ManageEvent.php#L377) expects lastWord from CRM_myExtension_Form_lastWord to be the same as the path, i.e. /civicrm/event/manage/lastword. Everything else works as expected if these are different, except Save.
Looks like this same issue exists for [Contribution Pages](https://github.com/civicrm/civicrm-core/blob/5db0bc3c1f54eaca4307f103a73bda596ae914d6/CRM/Contribute/Form/ContributionPage.php#L427
) and probably elsewhere.
For now, I've added a warning to the docs for [the tabset hook](https://lab.civicrm.org/documentation/docs/dev/-/merge_requests/908). It would be good to fix this so the code linked above uses the path for the url instead of the class, at some point.https://lab.civicrm.org/dev/core/-/issues/2611Change default_value column for custom fields from varchar to text2023-08-20T02:32:58ZlarsssandergreenChange default_value column for custom fields from varchar to textA custom field can be of many types, including note. For a custom field that is a note, you may want to use a default value that is longer than 255 characters, but currently the default value is stored as a varchar(255). It would make se...A custom field can be of many types, including note. For a custom field that is a note, you may want to use a default value that is longer than 255 characters, but currently the default value is stored as a varchar(255). It would make sense to change this column to text, so that longer values can be entered.
In the custom fields UI, this field would be changed to a textarea when note is selected.
For us, the use case would be storing the content of a short waiver, so that the exact value of the waiver at the time of signing can be stored on the participant for an event, but I can imagine there may be other use cases.
Will submit PR if supported.https://lab.civicrm.org/dev/core/-/issues/2641Users affected by ACLs intermittently lose access to contacts2023-08-22T04:01:30ZjhungerfordUsers affected by ACLs intermittently lose access to contactsOverview
----------------------------------------
When a user's permission to view Contacts is affected by multiple ACLs, they will intermittently lose access to some of them, especially directly after editing a record.
Reproduction ste...Overview
----------------------------------------
When a user's permission to view Contacts is affected by multiple ACLs, they will intermittently lose access to some of them, especially directly after editing a record.
Reproduction steps
----------------------------------------
1. Create a test User who is not an Administrator and does not have "view all contacts" or "edit all contacts" permission
2. Create two Groups (of type Access Control List), and add the test User to both groups
3. Create two Groups (not of ACL type) with distinct names like "Contact Group1" and "Contact Group2"
4. Create two Contacts and add one to each of the Groups created in step 3
5. Create two ACLs, each giving users in one of the ACL groups access to one of the Contact groups
6. Log in as the test User and repeatedly edit the Nickname (or any other data) on both of the test Contacts until you encounter a failure ("Error You do not have the necessary permission to view this contact").
It will rarely take more than ten edits before you encounter a failure, and the failures will never occur for one of the ACL groups.
Current behaviour
----------------------------------------
Sometimes it works, sometimes the user loses permission during the save or reload operation, resulting in the error message below. Our users reported that usually the data would be saved, but sometimes not (e.g. when saving some Activities on a Case). I have not been able to replicate the data loss scenario.
```
Error
You do not have the necessary permission to view this contact.
```
Expected behaviour
----------------------------------------
The data should be saved without error messages appearing (and this is what happens some of the time).
Environment information
----------------------------------------
* __Browser:__ Mostly tested on Chromium, most recently version 90.0.4430.93
* __CiviCRM:__ Tested on 5.33.5 and prior ESR versions
* __PHP:__ 7.3
* __CMS:__ Drupal 7.80
* __Database:__ 10.3.27-MariaDB
* __Web Server:__ Apache
Comments
----------------------------------------
We have mitigated the problem with this patch (minus the watchdog line with the FIXME comment):
https://github.com/AsylumSeekersCentre/civicrm-core/commit/030ecb6e31ed17b9fd01716cb772f13bb60bf455
With this in place, I've been unable to reproduce the error in manual testing. If it still happens, it's very rare. However, I'm not sure that this is a very "proper" solution so I'm opening the issue without a pull request for discussion.
The watchdog line from that patch usually produces output like this:
```
( ( `contact_a`.id IN ( SELECT contact_id FROM civicrm_group_contact WHERE group_id IN (5, 6) AND status = 'Added' ) ) ) AND (contact_a.is_deleted = 0)
```
On the occasions when the permission failure occurs, it instead looks like this:
```
( ( `contact_a`.id IN ( SELECT contact_id FROM civicrm_group_contact WHERE group_id IN (5) AND status = 'Added' ) ) ) AND (contact_a.is_deleted = 0)
```
It has somehow lost one of the groups it should be checking for. It always seems to retain the same one, which I think is why it works if the user is only affected by one ACL, and the bug only shows up if users are affected by multiple ACLs.https://lab.civicrm.org/dev/core/-/issues/2759Intermittent cookies error after contribution2023-03-14T07:16:59ZandyburnsIntermittent cookies error after contributionWe experience a cookies error intermittently after submitting a donation. Currently on Civi 5.35.2 and WP 5.7.2. This is multisite so may be multisite related.
![118401618_379155863073198_2489825481780434605_n](/uploads/1125c66666dd23c3...We experience a cookies error intermittently after submitting a donation. Currently on Civi 5.35.2 and WP 5.7.2. This is multisite so may be multisite related.
![118401618_379155863073198_2489825481780434605_n](/uploads/1125c66666dd23c31102ac79e053f06f/118401618_379155863073198_2489825481780434605_n.png)
A contributor does not get to the thank you page but gets the error above. The donation does succeed. This issue has persisted for a long-time (2+ years) and is a probably the worst civi issue I've encountered seeing it has to do with contribution pages and has no pattern that we can find. Tried all the go to solutions posted on SE with no luck. I would estimate that it occurs ~10-20% of the time.
This is related: https://lab.civicrm.org/dev/core/-/issues/517 and says using PHP 7.2 resolved, but were on 7.2 and now on 7.3
Some steps I've done:
- Set `Clean-up Temporary Data and Files` scheduled job run frequency from Hourly to Daily. It is on every site.
- Changed cookie domain constant in wp-config.php
```
// define('COOKIE_DOMAIN', false);
to
define('COOKIE_DOMAIN', $_SERVER\['HTTP_HOST'\] );
```
- Enabled https://lab.civicrm.org/extensions/qfsessionwarning and set timeout at 120 minutes.
`$civicrm_setting['core']['secure_cache_timeout_minutes'] = 120;`
We have considered changing how garbage collection works as sessions will die if garbage collection is run (server does that every 15 minutes).
[This post](https://civicrm.stackexchange.com/questions/39156/payments-fail-with-paypal-standard-and-could-not-find-a-valid-session-key) seems to have found a solution using the Fatal Error Handler. Is this recommended and how do you implement? Seems like a workaround symptom fix 'solution'.
Also getting this for additional event participants all the time (usually) or not at all depending on when I test: https://lab.civicrm.org/dev/core/-/issues/2708
Help?https://lab.civicrm.org/dev/core/-/issues/2775Scheduled reminders fail silently if there is an error in the template2023-03-27T16:26:57ZwmortadaScheduled reminders fail silently if there is an error in the templateOverview
----------------------------------------
We have recently come across an issue whereby scheduled reminders weren't being sent but we were completely unaware this was happening. We managed to track it down to an error in the temp...Overview
----------------------------------------
We have recently come across an issue whereby scheduled reminders weren't being sent but we were completely unaware this was happening. We managed to track it down to an error in the template for one of the event reminders we had set up - the template contained CSS and this caused a fatal error in Smarty.
```
PHP Fatal error: Smarty error: [in string:<meta content="text/html; charset=utf-8" http-equiv="Content-Type" />
<title></title>
<style type="text/css">html { font-size: 100%;}
html, button, input, select, textarea { font-family: sans-serif; }
body, h1, h2, h3, h4, h5, h6, p { margin: 0; }
table { border-collapse: collapse; border-spacing: 0; }
td img { display: block; }
...
```
The issue is that this stopped _all_ scheduled reminders from being sent - not just the event reminder. This meant that none of our membership renewal reminder emails were being sent either. We were completely unaware of this because there was no visible error message in the UI.
Reproduction steps
----------------------------------------
1. Create a scheduled reminder that works correctly.
1. Check that this is sending emails as expected.
1. Create another scheduled reminder with CSS in the template (see above) so that it causes a Smarty error.
1. Note that none of the scheduled reminder emails are being sent.
Current behaviour
----------------------------------------
CiviCRM fails silently when there is an error in a template for a scheduled reminder. This stops all scheduled reminders from being sent.
Expected behaviour
----------------------------------------
There are several things that should happen if there is an error in a Smarty template:
1. CiviCRM alerts the user at the point of creating/editing a scheduled reminder with a Smarty error in the template.
1. CiviCRM alerts the user if there is scheduled reminders fail to be sent - see #1933.
1. CiviCRM should continue to send the other scheduled reminders as normal so only the reminder with an error fails, rather than no reminders being sent at all.
Environment information
----------------------------------------
* __Browser:__ N/A
* __CiviCRM:__ _5.33.5_
* __PHP:__ _7.3_
* __CMS:__ _WordPress 5.4.6_
* __Database:__ _Maria DB 10.3.23._
* __Web Server:__ _Nginx 1.15.0_
Comments
-----------------------------------------
See this post on StackExchange for info about the Smarty error: [SMARTY error with template](https://civicrm.stackexchange.com/questions/31866/smarty-error-with-template).https://lab.civicrm.org/dev/core/-/issues/2788Add tokenlist api2023-09-24T22:53:09ZeileenAdd tokenlist apiWe need an api to call via ajax to determine the available tokens for a given entity. An example is the Scheduled reminders UI which currently presents the tokens for all entities (available or not) but would ideally look up the tokens v...We need an api to call via ajax to determine the available tokens for a given entity. An example is the Scheduled reminders UI which currently presents the tokens for all entities (available or not) but would ideally look up the tokens via ajax once the entity is selected.
I think the hardest part of doing this is agreeing how it should look - eg. entity, action, parameters - eg
```
Civi\Api4\Tokens::list()
->setSchema(['contributionId', 'contactId')
->setWhere('contact_type', '=', 'Organization')
->execute();
```
Possible entity names
- Token
- Tokens
- TokenProcessor
- EntityTokens
Possible actions
-list
- getTokens
- listTokens
Schema - maybe this one is clear cut
And - how do we pass relatively free form tokens such as activity_type_id, contact_type or (possibly in the future) saved_search_id where search kit plays a rolehttps://lab.civicrm.org/dev/core/-/issues/2795PHP8: Use of error-suppression operator (@) behaves differently in error_hand...2023-12-12T19:51:20ZDaveDPHP8: Use of error-suppression operator (@) behaves differently in error_handler functions in php8Parking what's left in https://github.com/civicrm/civicrm-core/pull/21064 here for now since I've cleared out a bunch and am taking a break from it. The gist is that in php8 the error suppression operator (e.g. `@fopen('/nonexistent/file...Parking what's left in https://github.com/civicrm/civicrm-core/pull/21064 here for now since I've cleared out a bunch and am taking a break from it. The gist is that in php8 the error suppression operator (e.g. `@fopen('/nonexistent/file.txt', 'r')` works differently with respect to error_handler functions. In a stock install as of right now you won't notice any difference, just there will be more errors depending on what 3rd party/custom error_handlers you're using and may behave differently on different CMSs in future depending on how their error-handling may change.
### smarty default modifier
The smarty modifier `|default` is a tricky one because its whole purpose is to deal with the situation where the var might be missing, but the way smarty compiles it is something like `<?php $this->assign('to', ((is_array($_tmp=@$this->_tpl_vars['to'])) ? $this->_run_mod_handler('default', true, $_tmp, '_high') : smarty_modifier_default($_tmp, '_high'))); ?>` which uses the @ to try to silence the error. In addition to [templates/CRM/Core/DatePickerRange.tpl and templates/CRM/Core/DatePickerRangeWrapper.tpl](https://github.com/civicrm/civicrm-core/commit/f671f500b658aa772fedc682dbb320cf456bc904), the below are all instances of this. They could all be rewritten to not use the modifier, but the modifier itself is part of smarty so can't easily be prevented from being used in future.
* CRM_Core_InvokeTest::testInvokeDashboardForNonAdmin
* CRM/Contact/Page/DashboardDashlet.tpl - $communityMessages
* CRM_Core_SessionTest::testSetStatusWithTextOnly
* CRM/common/info.tpl
* CRM_Core_SessionTest::testSetStatusWithTitleOnly
* CRM/common/info.tpl
* CRM_Core_SessionTest::testSetStatusWithBoth
* CRM/common/info.tpl
### opendir
```
CRM_Utils_File::cleanDir
if ($dh = @opendir($target)) {
```
### unlink and mkdir
Various calls to `@unlink` and `@mkdir`
### httpclient
```
CRM_Utils_HttpClientTest::testFetchHttp_badOutFile
Exception: fopen(/ba/d/path/too/utput): Failed to open stream: No such file or directory
...\tests\phpunit\CiviTest\CiviUnitTestCase.php:253
...\CRM\Utils\HttpClient.php:79 <---- see here
...\tests\phpunit\CRM\Utils\HttpClientTest.php:77
...\tests\phpunit\CiviTest\CiviUnitTestCase.php:267
...\phpunit8:671
```
### angular changeset
Aside: The code in question here suggests phpQuery has its own debug setting which maybe could be turned on automatically in future if you have civi debugging enabled:
`$return = phpQuery::$debug === 2 ? this->document->loadHTML($markup) : @$this->document->loadHTML($markup);`
```
Civi\Angular\ChangeSetTest::testInsertAfter
Exception: DOMDocument::loadHTML(): htmlParseEntityRef: no name in Entity, line: 2
...\tests\phpunit\CiviTest\CiviUnitTestCase.php:253
...\vendor\electrolinux\phpquery\phpQuery\phpQuery\DOMDocumentWrapper.php:200
...\vendor\electrolinux\phpquery\phpQuery\phpQuery\DOMDocumentWrapper.php:542
...\vendor\electrolinux\phpquery\phpQuery\phpQuery\DOMDocumentWrapper.php:505
...\vendor\electrolinux\phpquery\phpQuery\phpQuery\DOMDocumentWrapper.php:469
...\vendor\electrolinux\phpquery\phpQuery\phpQuery\phpQueryObject.php:2066
...\vendor\electrolinux\phpquery\phpQuery\phpQuery\phpQueryObject.php:2010
...\tests\phpunit\Civi\Angular\ChangeSetTest.php:28
...\Civi\Angular\ChangeSet.php:59
...\Civi\Angular\ChangeSet.php:19
...\tests\phpunit\Civi\Angular\ChangeSetTest.php:39
...\tests\phpunit\CiviTest\CiviUnitTestCase.php:267
...\phpunit8:671
```
* not ok 103 - Error: Civi\Angular\PartialSyntaxTest::testConsistencyExamples with data set 6 `('<div ng-if="a && b"></div>', '<div ng-if="a && b"></div>')`
* not ok 104 - Error: Civi\Angular\PartialSyntaxTest::testAllPartials
* These are the same issue as the changeset tests.
### testing of deprecations
These deliberately test deprecations
* not ok 223 - Error: Civi\Payment\PropertyBagTest::testSetContactIDLegacyWay
* not ok 227 - Error: Civi\Payment\PropertyBagTest::testSetCustomProp
### testGetfieldsHasTitle
```
api_v3_SyntaxConformanceTest::testGetfieldsHasTitle with data set 32 ('Entity')
Failure in api call for Entity getactions: include_once(api/v3/batch_create.php): failed to open stream: No such file or directory
0 ...\api\v3\Entity.php(18): CiviUnitTestCase->letssee(2, 'include_once(ap...', '\path\to\...', 18, Array)
1 ...\api\v3\Entity.php(18): include_once()
2 ...\api\v3\utils.php(2461): _civicrm_api3_entity_deprecation(Array)
3 ...\api\v3\utils.php(251): _civicrm_api3_deprecation_check('Entity', Array)
4 ...\Civi\API\Provider\ReflectionProvider.php(108): civicrm_api3_create_success(Array, Array, 'Entity', 'getactions')
5 ...\Civi\API\Kernel.php(149): Civi\API\Provider\ReflectionProvider->invoke(Array)
6 ...\Civi\API\Kernel.php(81): Civi\API\Kernel->runRequest(Array)
7 ...\api\api.php(22): Civi\API\Kernel->runSafe('Entity', 'getactions', Array)
8 ...\Civi\Test\Api3TestTrait.php(292): civicrm_api('Entity', 'getactions', Array)
9 ...\Civi\Test\Api3TestTrait.php(173): CiviUnitTestCase->civicrm_api('Entity', 'getactions', Array)
10 ...\tests\phpunit\api\v3\SyntaxConformanceTest.php(1662): CiviUnitTestCase->callAPISuccess('Entity', 'getactions', Array)
11 phar://.../phpunit8/phpunit/Framework/TestCase.php(1213): api_v3_SyntaxConformanceTest->testGetfieldsHasTitle('Entity')
12 ...\tests\phpunit\CiviTest\CiviUnitTestCase.php(267): PHPUnit\Framework\TestCase->runTest()
13 phar://.../phpunit8/phpunit/Framework/TestCase.php(889): CiviUnitTestCase->runTest()
14 phar://.../phpunit8/phpunit/Framework/TestResult.php(577): PHPUnit\Framework\TestCase->runBare()
15 phar://.../phpunit8/phpunit/Framework/TestCase.php(663): PHPUnit\Framework\TestResult->run(Object(api_v3_SyntaxConformanceTest))
16 phar://.../phpunit8/phpunit/Framework/TestSuite.php(481): PHPUnit\Framework\TestCase->run(Object(PHPUnit\Framework\TestResult))
17 phar://.../phpunit8/phpunit/Framework/TestSuite.php(481): PHPUnit\Framework\TestSuite->run(Object(PHPUnit\Framework\TestResult))
18 phar://.../phpunit8/phpunit/TextUI/TestRunner.php(462): PHPUnit\Framework\TestSuite->run(Object(PHPUnit\Framework\TestResult))
19 phar://.../phpunit8/phpunit/TextUI/Command.php(129): PHPUnit\TextUI\TestRunner->doRun(Object(PHPUnit\Framework\TestSuite), Array, Array, true)
20 phar://.../phpunit8/phpunit/TextUI/Command.php(101): PHPUnit\TextUI\Command->run(Array, true)
21 ...\phpunit8(671): PHPUnit\TextUI\Command::main()
22 {main}
Failed asserting that a integer is empty.
...\Civi\Test\Api3TestTrait.php:110
...\Civi\Test\Api3TestTrait.php:174
...\tests\phpunit\api\v3\SyntaxConformanceTest.php:1662
...\tests\phpunit\CiviTest\CiviUnitTestCase.php:267
...\phpunit8:671
```
### mailing testConcurrency
This doesn't run properly on windows even without the error_handler, and I don't feel like taking the time right now to deal with that:
not ok 1161 - Error: api_v3_JobProcessMailingTest::testConcurrency with data set #0 (array(20, 3, 10, 4, 1), array(2, 1), 4)https://lab.civicrm.org/dev/core/-/issues/2811Menu Angular error after upgrade2023-01-08T01:26:08ZStoobMenu Angular error after upgradeI do a fair amount of upgrades, and anecdotally after about a _half_ of recent upgrades, the CiviCRM menu fails to load. For half of those, a simple reload/refresh of the browser resolves the issue. For the other half, the civicrm/menu...I do a fair amount of upgrades, and anecdotally after about a _half_ of recent upgrades, the CiviCRM menu fails to load. For half of those, a simple reload/refresh of the browser resolves the issue. For the other half, the civicrm/menu/rebuild cache (or drush cc) must be cleared in order for the menu to appear. It is usually an Angular issue, seen attached.
Two questions, if these steps are required with such frequency can the upgrade process be improved by either:
1. clearing the cache automatically when the upgrade is finished
2. providing instructions and/or link on the upgrade screen to clear cache
![menu-load](/uploads/70b9bc8bcfbae067b5169d330bd16545/menu-load.png)
![upg](/uploads/19aa40afb6bda323d97cafe3c3cec5b3/upg.png)https://lab.civicrm.org/dev/core/-/issues/2863Agree / document how tokens should be rendered from outside core2023-09-24T22:52:19ZeileenAgree / document how tokens should be rendered from outside coreIdeally we would have one recommended api way to do this. This would be a way that is usable via api explorer and all the ways we use the api and core code would be converted to do things using that same api method
There is [a documenta...Ideally we would have one recommended api way to do this. This would be a way that is usable via api explorer and all the ways we use the api and core code would be converted to do things using that same api method
There is [a documentation pr here](https://lab.civicrm.org/documentation/docs/dev/-/merge_requests/962) - at the moment I don't think it meets this need.
**Permissions & non-workflow rendering**
There is a question as to whether the api would support non-workflow messaging (in which case the api would likely be more like `Message::render()`. The argument against is that the permissions are more complex. However some notes on that
1) the most common use case is probably when checkPermissions = FALSE in php code.
2) there is an appropriate permission already in core
```
render templates' => [
$prefix . ts('render templates'),
ts('Render open-ended template content. (Additional constraints may apply to autoloaded records and specific notations.)'),
],
```
3) currently the v3 MessageTemplate.send api allows access to the core function - it required a messageTemplateID to be set - but this can probably be faked to allow 'any text'https://lab.civicrm.org/dev/core/-/issues/2864Meta - token usage 5.43 standardisation effort2022-09-02T01:38:26ZeileenMeta - token usage 5.43 standardisation effortWe are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can ...We are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can be around the one release - documentation is [here](https://lab.civicrm.org/-/ide/project/documentation/docs/sysadmin/tree/case/-/docs/upgrade/version-specific.md/)
The goal is to consolidate such that
1. All token rendering is done through the token processor, token hooks are redundant and deprecated
2. All functions on CRM_Utils_Token are unused in core and deprecated except for those in the legacy BAO mailing classes.
3. We freeze, deprecate and phase out in legacy BAO classes in favour of flexmailer
4. All core classes that render tokens do so using consistent token naming & output conventions (rather than dates being formatted in various ways and fields sometimes outputting labels and other times values or name fields https://lab.civicrm.org/dev/core/-/issues/2650)
5. As much as possible we render via tokens rather than smarty in workflow message templates (that couples them with the entity rather than any form flow)
6. pdf & email & scheduled reminder tasks consistently offer the same entity-specific, contact & domain tokens
7. We can start to define obvious missing tokens such as `{domain.now}` (done), `{contribution.balance}` and formatted addresses
Task list / streams - this gives some idea of where things are progressing or blocked
- [x] standardise contact tokens to support/ advertise label style (e.g `{contact.prefix_id:label}`
- [x] make Contact tokens work with partial contact inputs - currently they only work with no contact array passed in, or an array with all possible values - currently blocked on a caching bug https://github.com/civicrm/civicrm-core/pull/21790
- [x] split contact tokens into own class & extend entity tokens like the other classes(blocked on stdise contact tokens)
- [x] switch contact tokens to use apiv4
- [x] work through standardising location token loading (this could go further - ie I had thought about adding loading for billing
- [x] date formatting standardisation [documentation](https://docs.civicrm.org/user/en/latest/common-workflows/tokens-and-mail-merge/#date) https://lab.civicrm.org/dev/core/-/issues/2746
- [x] locale sensitive standardised money formatting https://lab.civicrm.org/dev/core/-/issues/2638 & [pr](https://github.com/civicrm/civicrm-core/pull/21783)
- [x] switch badge tokens to use token processor
- [x] event tokens cleanup & loading fixes + remove unindexed query, move the `fee_amount` and `balance` tokens to the participant tokens class, fix event class to listen to either 'participantId' or 'eventId'. Participant class should run first & load eventId if required. Events should be cached in a static cache as it would be very rare to message thousands of events in one run.
- [x] Recurring contribution tokens work (currently availble in only the recurring_edit template with a pr to extend to cancelled recurring https://test.civicrm.org/job/CiviCRM-Core-PR/44632/
- [x] pdf letter is sane (https://lab.civicrm.org/dev/core/-/issues/2790) enough to extend to deliver tokens for other entities (participant https://lab.civicrm.org/dev/core/-/issues/780)
- [x] email letter extend to deliver tokens for other entities (membership & participant) (https://lab.civicrm.org/dev/core/-/issues/2862)
- [x] scheduled reminders supports participant tokens https://lab.civicrm.org/dev/core/-/issues/779
- [x] domain tokens are consistently available & render https://lab.civicrm.org/dev/core/-/issues/2838 - done but a gap on case tasks as they prioritise being able to filter tokens by case type
- [x] install flexmailer on new installs https://lab.civicrm.org/dev/core/-/issues/2836
- [ ] fix remaining places that process tokens other than via the payment processor
- [x] eliminate replaceGreeting - currently [blocked on apiv4 caching bug](https://github.com/civicrm/civicrm-core/pull/21790)
- [ ] fix CRM_Contact_BAO_Contact_Utils::updateGreeting to not call `getTokenDetails`
- [ ] confirm civicrm_api3_mailing_preview is replaced by flexmailer
- [x] remove getTokenDetails call from CRM_Contribute_Form_Task_PDFLetter - note this appears to be an expensive way to call the contact api & tokens are unused https://github.com/civicrm/civicrm-core/pull/21805
- [x] remove extra rendering in BAO_Pledge https://github.com/civicrm/civicrm-core/pull/21789
- [x] removed getTokenDetails from updatePledgeStatus https://github.com/civicrm/civicrm-core/pull/21847
- [x] remove getTokenDetails from transitionParticipants
- [ ] remove getTokenDetails from sendCancellation & cancelParticipant in selfsvc flow
- [ ] deliverGroup - replaced by flexmailer?
- [ ] MailingPage::preview - replaced by flexmailer?
- [ ] remove hook:;tokenValues from label task
- [ ] getAnonymousTokenDetails ???
- [ ] confirm BAO_Mailing::compose is only via flexmailer
- [x] communicate rc status via dev list
- [x] communicate changes via blog
- [ ] upgrade message https://lab.civicrm.org/dev/core/-/issues/2871
- [ ] docs update to these instructions / check it can still all be done https://docs.civicrm.org/dev/en/latest/step-by-step/create-custom-case-token/
**Currenly out of scope**
- [ ] Token.getlist api - until we have this we have an issue with audience-filtering and with switching tokens depending on the schema. Also case tasks still have to render tokens weirdly without this because they currently filter by case type with partial success https://lab.civicrm.org/dev/core/-/issues/2788
- [ ] Clarify how extensions should render tokens https://lab.civicrm.org/dev/core/-/issues/2863
- [ ] encourage flexmailer with status checks on existing installs https://lab.civicrm.org/dev/core/-/issues/2836
Analysis of current token rendering it core:
Method | Used in | Status
-- | -- | --
Use the token processor | scheduled reminders, civimail with flexmailer in use, activity pdf task | This is the method we are consolidating all paths to use.
Use the functions in CRM_Utils_Token | PDF tasks other than activity | Migrated (PRs on [membership](https://github.com/civicrm/civicrm-core/pull/21521) and [contribution](https://github.com/civicrm/civicrm-core/pull/21524). Also work on fixing up the [pdf task class](https://lab.civicrm.org/dev/core/-/issues/2790) supporting[add participant tokens](https://lab.civicrm.org/dev/core/-/issues/780) (once we’ve finished [the standardisation of those](https://github.com/civicrm/civicrm-core/pull/21587))
| Email tasks | Migrated to token processor. Added tokens for other entities (participant, membership https://lab.civicrm.org/dev/core/-/issues/1396) [also see](https://lab.civicrm.org/dev/core/-/issues/2862)
| Export tasks (when merging contacts in export) | Migrated to token processor
| Profile link rendering | Migrated to token processor
| Storing contact greetings (eg. email_greeting_display) | In progress. Currently resolving inconsistent token syntax and load issues as this needs to be able to use partial pre-loads
Use random ad hoc code | Event badges | [migrated](https://github.com/civicrm/civicrm-core/pull/21587) - this is the only place outside of scheduled reminders where event or participant tokens were used in core
| Send SMS task (BAO_Activity::sendSMS) | migrated
| Labels task CRM_Contact_Form_Task_Label | argh
| CRM_Event_Form_SelfSvcTransfer | Still pretty argh - but this is a workflow message template so the fix is a bit different
| Transition participants | Also a workflow - possibly redundant as seems to be mostly about contact tokens which are resolved in the send function anyway
| Pledge acknowledgements | Also appear to be redundant
| CiviMail with flexmailer not used (many functions & classes) | Push the switch to flexmailer.
| Various unsubscribe / resubscribe type actions | Probably out of scope for this roundhttps://lab.civicrm.org/dev/core/-/issues/2915Proposal: Remove the Group Visibility terminology "Expose Publicly" as it has...2024-02-08T23:31:11Zjustinfreeman (Agileware)Proposal: Remove the Group Visibility terminology "Expose Publicly" as it has unintended connotations and would be good to standardise the terminology to describe "Group Visibility" and its optionsThis is a proposal to:
1. Remove the **Group Visibility** terminology _Expose Publicly_ as it has unintended connotations (as pointed out to me today by a Lawyer) and
2. Would be good to standardise the terminology to describe **Group Vi...This is a proposal to:
1. Remove the **Group Visibility** terminology _Expose Publicly_ as it has unintended connotations (as pointed out to me today by a Lawyer) and
2. Would be good to standardise the terminology to describe **Group Visibility** and its options. Ideally making it more obvious as to the purpose of each option.
As described on https://docs.civicrm.org/user/en/latest/organising-your-data/groups-and-tags/#visibility
* **Group Visibility**: **Public pages / Expose Publicly** - if you want to allow contacts to join and remove themselves from this group via Registration and Account Profile forms
* **Group Visibility**: **User and User Admin Only** - if membership in this group is controlled only by authorised CiviCRM users
As I understand it, this option _only_ applies to **Mailing List, Group Types**.
# Expose Publicly
![expose-publicly](/uploads/8d44c054f82a86d8fe92f8f7cd535663/expose-publicly.png)
# Group Visibility and options
![visbility](/uploads/7c4069b330d413230996fab273d242f4/visbility.png)https://lab.civicrm.org/dev/core/-/issues/2945Make CRM_Core_Smarty::singleton() use Civi::statics instead of a static var2023-10-13T05:11:14ZDaveDMake CRM_Core_Smarty::singleton() use Civi::statics instead of a static varDuring unit tests, static vars can be problematic because they don't get reset between tests. There's a point within smarty where if a test fails at that point then its secure mode doesn't get reset. It's come up a couple times and then ...During unit tests, static vars can be problematic because they don't get reset between tests. There's a point within smarty where if a test fails at that point then its secure mode doesn't get reset. It's come up a couple times and then all the tests that use smarty on certain forms fail after it. It's not easy to track down if you haven't seen it before.
There's an argument civi code should be smarty secure-mode-compliant instead, but it isn't at the moment.https://lab.civicrm.org/dev/core/-/issues/2966Duplicate contacts when custom field value is 'email', 'phone', etc.2023-03-18T04:08:01Zrita_compucorpDuplicate contacts when custom field value is 'email', 'phone', etc.**Issue**:
when existing contacts are donating (without being logged in) with the same details (email, first name, last name) that are registered in the system, most of the times there is a new contact created instead of merging it to t...**Issue**:
when existing contacts are donating (without being logged in) with the same details (email, first name, last name) that are registered in the system, most of the times there is a new contact created instead of merging it to the existing contact automatically.
**Investigation**:
When a custom field with a checkbox had its title and value set to ‘email’ or 'phone' and the unsupervised dedupe rule was prepared to check for email or phone as well causing the problem. So the problem occurs when:
- the unsupervised dedupe rule has email in it AND
- the custom field’s value is set to ‘email’
However not only the email option can cause duplications, but if you prepare an:
- unsupervised dedupe rule with phone ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24](/uploads/758592a6d5330c1133dcd1206e714daa/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24.png) AND
- the custom field’s value is set to ‘phone’ ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12](/uploads/df40ee69692fda55f51e946b2c841b9d/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12.png)
And even though the clients enter the same email/phone numbers on the donation form, the dedupe rule will be skipped and a new contact with the same information will be created. Probably this can be reproduced with many fields in the system, but only tried these two.
I checked on a few **different civi versions:**
- 4.7.26 - couldn’t reproduce
- 5.35.2 - issue reproducible
- 5.45.alpha1 (core demo civi) - issue reproducible
**Reproduction steps:**
1. create an unsupervised individual rule with the following details: ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24](/uploads/758592a6d5330c1133dcd1206e714daa/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24.png)
2. create a custom field set for Contacts with checkboxes like this: ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12](/uploads/df40ee69692fda55f51e946b2c841b9d/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12.png)
3. add the custom field set that you created in previous step to a profile that can be added to a contribution page
4. create a new (or use an old) contribution page and add the profile that you created in previous step to it in the Profiles tab
5. create a new contact, and make sure it has a First name, Last name and Phone number
6. open the contribution page as anonymous and donate to it, using the exact same First name, Last name and Phone number that you entered in previous step AND select the ‘phone’ option at the custom field you created → _after you submit the donation, a new contact will be created instead of merging it to the existing one, which is wrong behaviour_
7. donate again on the same form, with the same details as before, but instead of selecting ‘phone’ select either the ‘email’ or ‘post’ options → _the contribution will be merged to the original contact, which is the correct behaviour. It is because email and post are not added to the unsupervised dedupe rule in this case._
I think most of the time civi admins create custom fields with values set as numbers, but in some cases they might configure it to be the same as the label name and in that case duplications might occur. I think it might be happening when the value of the custom field is the same as the name of the field in the database (just a thought, can't say for sure).https://lab.civicrm.org/dev/core/-/issues/2990CiviCampaign - Move data into a single table2024-03-11T20:47:07ZcolemanwCiviCampaign - Move data into a single tableBackground
------------
CiviCampaign is an optional component (disabled by default on new installs). It allows contributions, events, etc. to be associated with a campaign.
Campaigns are basically like tags. The campaign itself contain...Background
------------
CiviCampaign is an optional component (disabled by default on new installs). It allows contributions, events, etc. to be associated with a campaign.
Campaigns are basically like tags. The campaign itself contains a bit more data (type, start & end date, description, goal) than a Tag, but the mechanism for linking an entity with a campaign is a very simple foreign key.
How it works now
-------------
"Tagging" an entity with a campaign works via a column on that entity's table. E.g. `civicrm_contribution_page.campaign_id`. There are currently 10 entities with such a column, allowing them to be "tagged" with a campaign_id.
The problem
-----------
This design involves tight coupling between CiviCampaign and other components, as it involves adding a column to their tables. It also limits the possibilities for which entities can have a campaign_id.
The solution
----------
There is another common pattern in CiviCRM, which is to add a small "bridge" table to join two entities, which is a good alternative to adding a column. `civicrm_entity_tag` is one example. It includes `tag_id`, `entity_id` and `entity_table` columns, and an OptionGroup `tag_used_for` which lists all the possible values for `entity_table`. Importantly, it's easily added-on by extensions; if an extension provides a new entity type which should be taggable, is just adds it to the option list.
The migration
------------
If CiviCampaign were a greenfield project designed today, we wouldn't think twice about structuring the data with a bridge table instead of littering our database with `campaign_id` columns. As a brownfield problem, fixing the structure may be more trouble than it's worth, but I thought I'd at least articulate how it might be possible:
- APIv4 has a new concept of "Extra" calculated fields, which could provide a faux `campaign_id` column for backward compatibility.
- CiviCampaign could implement a post-hook to save campaign_id if it's passed into "create" params. This would keep create and update working as if that column still exists.
- APIv3 would require some hacking to make a pseudo-field to fill in for a missing `campaign_id`.
Random musings
-----------
While researching this, I stumbled across a table called `civicrm_campaign_group`. I have no idea what it does but it looks very similar to how I imagined the new bridge table would be. It includes a `campaign_id`, `entity_id` and `entity_table` column. Why does this table exist??https://lab.civicrm.org/dev/core/-/issues/2991Proposal: rework name fields to make name entity2023-12-13T09:20:59ZAndie HuntProposal: rework name fields to make name entity## Overview
A recent [discussion with Guy](https://chat.civicrm.org/civicrm/pl/wwqowz635fbpifio7on4e8r8dw) and #2883, plus the ongoing mayhem of organization names and personal experience with a variety of great and terrible name handli...## Overview
A recent [discussion with Guy](https://chat.civicrm.org/civicrm/pl/wwqowz635fbpifio7on4e8r8dw) and #2883, plus the ongoing mayhem of organization names and personal experience with a variety of great and terrible name handling systems, have led me to think that the solution to names isn't creating more or better-labeled fields for more types of names but rather to make name its own entity akin to phone, email, address, website, etc.
Once name is an entity, each contact could have multiple names, one primary, but with a bunch of possible types:
- legal name
- trade name
- nickname
- former name
- name in other language/culture
- name in an external system
- common mistakes
There could also be a bunch of name renderings - ways the name is displayed or inserted:
- sort name
- display name
- name for greetings
- name for listings
The latter could generate the existing greetings and potentially others. In particular, I think there's a real gap for how you insert someone's name in emails where you don't want "Dear", "Hi", or any other greeting as such.
Finally, the duplicate matching process could look for any of the possible names, cutting down on duplicates.
## Example use-case
### Organization has multiple names
An organization has their corporate name, colloquial name, and the name of their main product. These all refer to the same contact, but employees might fill the current employer field as one of the various ways. As an admin, you want to avoid having to merge them all the time. As a routine user, you will want to know all the various ways an organization is termed. As a bookkeeper, you will want to send invoices, checks, receipts, etc. with the correct entity name.
In this case, there are three name entities:
| Organization name | Name type | Is primary |
| ------------------------ | --------- | ---------- |
| Widget Productions | General | true |
| Widget Productions, Inc. | Legal | false |
| WidgetPro | General | false |
| Widgetizer | Product | false |
There would be a couple of name renderings:
| Rendering type | Value |
| -------------- | ------------------------ |
| Display Name | Widget Productions |
| Sort Name | Widget Productions |
| Billing name | Widget Productions, Inc. |
### Minister has variety of prefixes
Someone's name is styled differently when it is listed versus when they are addressed. In this case, a minister is styled "The Reverend Sally Jones" or "Rev. Sally Jones" when listed somewhere but her salutation is "Ms. Jones".
I think she'd have only one entry as a name entity but several renderings:
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------------|------------|-------------|-----------|--------|-----------|------------|
| Ms. | The Reverend | Sally | Lynn | Jones | *null* | General | true |
These values are going to depend upon your sitewide preferences and choices for this contact, but a reasonable default might produce the following:
| Rendering type | Value |
|-----------------|--------------------------|
| Display Name | Sally Lynn Jones |
| Sort Name | Jones, Sally Lynn |
| Addressee | The Reverend Sally Jones |
| Email greeting | Dear Sally |
| Postal greeting | Dear Ms. Jones |
| Short name | Sally |
### An organization has a different name in another language
This should be self-explanatory:
| Organization name | Name type | Is primary |
|--------------------------|-----------|------------|
| Médecins Sans Frontières | General | true |
| Doctors Without Borders | Localized | false |
| Rendering type | Value |
|----------------|--------------------------|
| Display Name | Médecins Sans Frontières |
| Sort Name | Médecins Sans Frontières |
| English Name | Doctors Without Borders |
### Someone's family name comes before their given name
I'm on the fence about how this should be handled, but all I know is that we currently handle it terribly (except maybe in translations that assume it?) One option might be to call the fields first, middle, and last names according to position (*Ai* being the family name, but it comes first):
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------|------------|-------------|-----------|--------|-----------|------------|
| Mr. | *null* | Ai | *null* | Weiwei | *null* | General | true |
Alternatively, we could name the fields according to function:
| Prefix | Title | Given name | Additional name | Surname | Suffix | Name type | Is primary |
|--------|--------|------------|-----------------|---------|--------|-----------|------------|
| Mr. | *null* | Weiwei | *null* | Ai | *null* | General | true |
In either case, the renderings should be like
| Rendering type | Value |
|-----------------|---------------|
| Display Name | Ai Weiwei |
| Sort Name | Ai Weiwei |
| Addressee | Mr. Ai Weiwei |
| Email greeting | Dear Weiwei |
| Postal greeting | Dear Mr. Ai |
| Short name | Weiwei |
### Clinic needs to retain legal name for insurance billing
Someone's real name doesn't match their legal name, let alone their nickname. While most organizations wouldn't want or need to record a contact's dead name, a clinic might need it for billing correspondence. In this case, a patient is named Stephen, or Steve for short, but his legal name is Barbara.
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------|------------|-------------|-----------|--------|-----------|------------|
| Mr. | *null* | Stephen | Wayne | Smith | *null* | General | true |
| Mr. | *null* | Steve | *null* | Smith | *null* | Nickname | false |
| *null* | *null* | Barbara | Wayne | Smith | *null* | Legal | false |
All of the name renderings would use the primary name or nickname, but the clinic could add an additional name rendering for how they refer to patients when billing to the insurance company.
| Rendering type | Value |
|-----------------|----------------------|
| Display Name | Stephen Wayne Smith |
| Sort Name | Smith, Stephen Wayne |
| Addressee | Mr. Stephen Smith |
| Email greeting | Dear Steve |
| Postal greeting | Dear Mr. Smith |
| Short name | Steve |
| Insurance name | Barbara Smith |
### Household name possibilities
The current household name field befuddles lots of people and, in my opinion, makes households less attractive to use. If a household can have multiple names, it might be more useful.
You might use a Household name field that has everyone all listed out, or you could use the Last name field.
| Household name | Last name | Name type | Is primary |
|-------------------------------------|----------------|-----------|------------|
| Katherine Williams and Kendra Green | *null* | General | true |
| *null* | Williams-Green | General | false |
| Rendering type | Value |
|-----------------|------------------------------------------|
| Display Name | Katherine Williams and Kendra Green |
| Sort Name | Katherine Williams and Kendra Green |
| Addressee | The Williams-Green Household |
| Email greeting | Dear Katherine Williams and Kendra Green |
| Postal greeting | Dear Katherine Williams and Kendra Green |
Or if everyone shares a last name, a single name entity could do it all:
| Given name | Surname | Name type | Is primary |
|---------------|---------|-----------|------------|
| Jack and Jill | Hill | General | true |
(If we don't feel the need to stick with the legacy name field names, there's really no reason that "Given name" couldn't replace Organization name and Household name, too, so this illustrates that.)
| Rendering type | Value |
|-----------------|-------------------------|
| Display Name | Jack and Jill Hill |
| Sort Name | Hill, Jack and Jill |
| Addressee | The Hill Household |
| Email greeting | Dear Jack and Jill |
| Postal greeting | Dear Jack and Jill Hill |
## Current behavior
Right now, each of these situations is pretty annoying. It might require a separate custom field, located away from the name fields. It might make it difficult to automatically and accurately generate greetings for many people.
Currently, there are fields in the `civicrm_contact` table for each name part, plus nickname and legal name.
The only name renderings are display name and sort name (set sitewide), and addressee and email and postal greetings (site-wide presets and defaults, with custom options).
There's no provision for additional renderings or any way to do fallbacks like pick a nickname and fall back to a first name.
There's a limited number of alternative names for a contact.
Alternative names are not compared against primary names when deduping.
## Proposed behavior
### Schema
The name fields would be gone from the contact entity, and a new `civicrm_contact_name` entity is added with one of the following sets of fields:
| Traditional | Radical |
|-------------------|-------------------|
| id | id |
| contact_id | contact_id |
| contact_name_type | contact_name_type |
| is_primary | is_primary |
| first_name | given_name |
| middle_name | additional_name |
| last_name | surname |
| prefix_id | prefix_id |
| suffix_id | suffix_id |
| formal_title | formal_title |
| household_name | |
| organization_name | |
The `contact_name_type` would have reserved options General, Nickname, Legal, and Localized, but more could be added.
A new `civicrm_contact_rendering` entity is added with the following set of fields:
| Fields |
|-------------------------------|
| id |
| contact_id |
| contact_rendering_type |
| contact_rendering_template_id |
| value |
The `contact_rendering_type` would have reserved options Display name, Sort name, Addressee, Email greeting, Postal greeting, and Short name, but more could be added.
The `contact_rendering_template_id` would be akin to the existing `addressee_id`, `email_greeting_id`, and `postal_greeting_id`, while `value` would be the actual generated or custom value.
### Upgrade
On upgrade, the `civicrm_contact_name` would be populated with the main name fields, with nickname and legal name as additional rows. On upgrade, the `civicrm_contact_rendering` would be populated with rows for each contact's display name, sort name, addressee, and greetings.
### Contact rendering templates
The contact rendering templates would have the option to specify a contact name type to provide the default value, with a fallback to the primary contact name. For example, email greeting might default to "Dear {nickname.first_name}", where it picks the value of `first_name` from a contact's name entry with the name type of Nickname, and if that isn't available, it picks the primary name entry's value for `first_name`.
### Dedupe
When matching contacts on any of the name fields, the dedupe process would compare against all rows in the `civicrm_contact_name` field, regardless of `contact_name_type` or if it's the primary name.
## Comments
This is all very early in the process: I think it's important to do something like this, but I don't have my heart on the specific mechanisms here. I think it would need to be evaluated for performance in a variety of ways, but I don't expect it would have too huge of an impact.https://lab.civicrm.org/dev/core/-/issues/2994Support oEmbed for external facing pages2024-03-21T18:50:44ZJoeMurraySupport oEmbed for external facing pagesAn important evolution of CiviCRM is to better support remote sites exposing CiviCRM forms and content.
Many platforms including WordPress and Drupal core since 8.6 support oEmbed, which allows a site to pull in content from a different...An important evolution of CiviCRM is to better support remote sites exposing CiviCRM forms and content.
Many platforms including WordPress and Drupal core since 8.6 support oEmbed, which allows a site to pull in content from a different site that exposes its content as oEmbed.
There is a good list of oEmbed providers, and CiviCRM should aspire to join it: https://oembed.com/providers.json
We should expose the following 'pages' as oEmbed content, using appropriate standard markup for use in WordPress and Drupal 8.6+ sites, and support the workflow for confirmation, thank you, and search results as appropriate:
1. Phase 1
1. Contribution page
2. Event info page
3. Event registration page
2. Phase 2
1. Something like a profile create/edit page that does not involve a payment but allows submission of a form to create a contact and activity and perhaps more (eg parent-child relationship)
2. Search Kit page
3. Petition page
3. Phase 3
1. Personal Campaign Page
2. Pages from extensions, eg Grant Application Page
3. CiviSurvey pages
Note: I haven't done the research to understand the details of oEmbed and potential challenges. I think this could be done through extension(s).
I'm posting this in order to generate discussion for a possible Roadmap item.https://lab.civicrm.org/dev/core/-/issues/3010authx: Users that are disabled/blocked in the cms can still log in, even if t...2023-11-07T10:19:00ZDaveDauthx: Users that are disabled/blocked in the cms can still log in, even if the `XXX_user` setting is set to "require"At https://docs.civicrm.org/dev/en/latest/framework/authx/#settings you have a choice for whether the cms user needs to exist in order to log in, but even if you set it to "require", it will let a disabled/blocked user log in. My expecta...At https://docs.civicrm.org/dev/en/latest/framework/authx/#settings you have a choice for whether the cms user needs to exist in order to log in, but even if you set it to "require", it will let a disabled/blocked user log in. My expectation is that it would work the same way as regular logins if I choose "require" as the policy.
Some cms's may also allow other ways to block users, such as throttling or temporary IP bans. I would argue it should respect those too - i.e. if the cms doesn't allow login then authx shouldn't (when using "require").https://lab.civicrm.org/dev/core/-/issues/3014Proposal: Explore adding hooks to allow plugins to track database queries, e....2023-04-23T20:13:36ZBradley TaylorProposal: Explore adding hooks to allow plugins to track database queries, e.g. to enable integration with plugins like Query MonitorOne of my favourite plugins in the WordPress space is [Query Monitor](https://wordpress.org/plugins/query-monitor/). Among other things it allows you to monitor at a glance what database calls have been made for a specific request, makin...One of my favourite plugins in the WordPress space is [Query Monitor](https://wordpress.org/plugins/query-monitor/). Among other things it allows you to monitor at a glance what database calls have been made for a specific request, making it easy to spot inefiencient code and queries.
Currently CiviCRM does not have anything analogous to this. The [CiviCRM docs recommend using the MySQL query log](https://docs.civicrm.org/dev/en/latest/tools/debugging/#viewing-a-query-log-from-mysql), but this isn't easy to view at a glance, and it doesn't make it easy to correlate queries to sepecific web requests. CiviCRM's database queries don't show in Query Monitor because CiviCRM does not use WordPress' `WPDB` class. Therefore, I've been looking to see if it would be possible to extend Query Monitor to know about CiviCRM's database queries. Its reasonably easy to make this work if you patch the `CRM_Core_DAO::query` method to the following:
```
public function query($query, $i18nRewrite = TRUE) {
// rewrite queries that should use $dbLocale-based views for multi-language installs
global $wpdb, $dbLocale, $_DB_DATAOBJECT;
if ( defined( 'SAVEQUERIES' ) && SAVEQUERIES ) {
$wpdb->timer_start();
}
if (empty($_DB_DATAOBJECT['CONNECTIONS'][$this->_database_dsn_md5])) {
// Will force connection to be populated per CRM-20541.
new CRM_Core_DAO();
}
$conn = &$_DB_DATAOBJECT['CONNECTIONS'][$this->_database_dsn_md5];
$orig_options = $conn->options;
$this->_setDBOptions($this->_options);
if ($i18nRewrite and $dbLocale) {
$query = CRM_Core_I18n_Schema::rewriteQuery($query);
}
$ret = parent::query($query);
if ( defined( 'SAVEQUERIES' ) && SAVEQUERIES ) {
$wpdb->num_queries++;
$wpdb->queries[] = array(
$query,
$wpdb->timer_stop(),
$wpdb->get_caller(),
$wpdb->time_start,
array(),
'trace' => new QM_Backtrace( array(
'ignore_frames' => 1,
)),
'result' => is_countable($ret) ? count($ret) : 1
);
}
$this->_setDBOptions($orig_options);
return $ret;
}
```
Esentially the `$wpdb->queries` array is being populated with knowledge of the CiviCRM database queries, which query monitor can then read.
With this in place you can start to see some interesting insights. For example, we can see that the new mailing screen contains a number of database queries that are reapeated 40 times each, which is probably ripe for optimisation:
![Screenshot_2022-01-02_at_14.13.41](/uploads/117036cbf7cb970e93c7399eeb436f64/Screenshot_2022-01-02_at_14.13.41.png)
The WordPress admin bar shows the total count of queries (243) combined from both CiviCRM and WordPress.
Hacking the core files works for my purposes, but obviously doesn't allow for this to be packaged into a distributable plugin. Therefore, I think it'd be cool if there were a generic hook at the end of `CRM_Core_DAO::query` which could be used to save the query information in a format ameanable to Query Monitor. Alternatively CiviCRM could copy WordPress idea of the `$wpdb->queries` array, creating an array of queries which plugins like Query Monitor could make use of (in WordPress this is behind a flag so there is no performance hit when not in a development environment).
I've not given too much thought to exactly how this hook should look, but before I take this further (e.g. to the Pull Request state) it'd be good to get some views from the community:
- Is this something others in the community would find useful if this were distributed as a plugin?
- Does anyone know of plugins similar to Query Monitor in the Drupal or Joomla space which may benifit from this type of solution?
- Would people find this sort of hook useful for other use cases - for example, for writing debugging tools native to CiviCRM.
Keen to get others views on this one. I'll be interested to know what people think.https://lab.civicrm.org/dev/core/-/issues/3027If you don't have permission to add contacts don't show ways to add contacts2022-11-15T17:49:39ZandyburnsIf you don't have permission to add contacts don't show ways to add contactsIn various places in Civi, you can either search for or add a contact when adding a contribution, participant, activity, etc. This works if you have the `add contacts` permission but not if you don't. The error is misleading too.
![add_...In various places in Civi, you can either search for or add a contact when adding a contribution, participant, activity, etc. This works if you have the `add contacts` permission but not if you don't. The error is misleading too.
![add_contact_form](/uploads/7d307dea4e7643e33db812398bc745e4/add_contact_form.png)
https://lab.civicrm.org/dev/core/-/blob/master/js/Common.js#L811https://lab.civicrm.org/dev/core/-/issues/30315.44+ Unsubscribe from Smart Group form generates excessive process load2023-01-31T13:58:20Ztommybobo5.44+ Unsubscribe from Smart Group form generates excessive process loadOverview
----------------------------------------
The recent change to email unsubscribe from Smart Group is overloading mysql. [This change ](https://github.com/civicrm/civicrm-core/pull/21176) by @mattwire is providing a check to see w...Overview
----------------------------------------
The recent change to email unsubscribe from Smart Group is overloading mysql. [This change ](https://github.com/civicrm/civicrm-core/pull/21176) by @mattwire is providing a check to see which smart group an unsubscribing user is in before unsubscribing them. On a large mailing, site with a lot of smart groups, or even one complex smart this code is a time bomb. Especially as we see a rise in automated link checking bots on large email providers.
Current behavior
----------------------------------------
When a email recipient clicks unsubscribe all smart groups are regenerated. When CRM_Contact_BAO_GroupContactCache is called all smart groups are regenerated regardless of the setting of cache time. Also this is running even if the email doesn't contain any smart groups. Then the cache table is Left Joined onto the query for a search. Which also can be refreshed by another user clicking the link in the 5-10 second process time of a large site with large smart groups.
This is an incredibly high process load before any action is actually taken in order to display a group title on page that 9 times out of 10 is a bot not an end user.
Proposed behavior
----------------------------------------
Members do not need to see the Group labels from which they are unsubscribing, and if they are unsubscribing from this Email list then it is reasonable to mark them as unsubscribed from a smart group used in the mailing even if they are not in it. If they in the future become a member of the smart group, they probably don't want email about it.
It is reasonable to run this process AFTER the unsubscribe button is clicked by the user. Running it on page load is just wasting a ton resources on bot traffic.
Comments
----------------------------------------
@mattwire 's solution is reasonable for many, but it just isn't scaling well on larger civicrm installs. We have either reverted the code or switched to Opt Out of all emails for those larger clients, but those are temporary solutions.https://lab.civicrm.org/dev/core/-/issues/3040UI allows creation of unresolvable failure for status check "Relationship Typ...2023-11-16T16:09:55ZAllenShawUI allows creation of unresolvable failure for status check "Relationship Type Internal Name Duplicates"Admittedly, "unresolvable" is a bit of an overstatement, but you can create this situation in the UI, and then the UI provides no way to resolve it (it is resolvable via the API).
**To reproduce:**
1. Visit https://dmaster.demo.civicrm....Admittedly, "unresolvable" is a bit of an overstatement, but you can create this situation in the UI, and then the UI provides no way to resolve it (it is resolvable via the API).
**To reproduce:**
1. Visit https://dmaster.demo.civicrm.org/civicrm/admin/reltype?reset=1
1. Create a new relationship type with Label A "Foobar" and Label B "Employee Of"
1. Visit https://dmaster.demo.civicrm.org/civicrm/a/#/status and observe error: "Relationship Type Internal Name Duplicates;
Relationship type Employee of has the same internal machine name as another type.": ![error](/uploads/6d2eb70ff9aac5ce16ca948faee6d077/error.png)
1. Visit https://dmaster.demo.civicrm.org/civicrm/admin/reltype?reset=1 and edit the offending relationship (created in step 1); you can edit the labels but not the name. Best you can do is change "Employee of" to something else, like "Foobar of".
1. Visit https://dmaster.demo.civicrm.org/civicrm/a/#/status and observe that the error persists, exactly the same as in Step 3.
There's no way to change the name in the UI; you must use the API to change it.
**Proposed solution:**
* Upon creating a relationship type (other than reserved types created upon installation), use centralized logic to derive unique name from label; i.e., opt-in to the new behavior provided by https://github.com/civicrm/civicrm-core/pull/22508.
Thoughts?https://lab.civicrm.org/dev/core/-/issues/3067Improving performance of retreiving display names for multivalued custom data2022-11-25T15:39:20ZMichael McAndrewImproving performance of retreiving display names for multivalued custom data.Have been taking a look at https://github.com/civicrm/civicrm-core/pull/20764/files#diff-36ab608da0b5718996afd18c10a8c12653c2a1675e81fd3c4dc3e40c2ebb25d2R1040.
My hunch is that for a query that returns 50,000 rows, the 'group_concat fi....Have been taking a look at https://github.com/civicrm/civicrm-core/pull/20764/files#diff-36ab608da0b5718996afd18c10a8c12653c2a1675e81fd3c4dc3e40c2ebb25d2R1040.
My hunch is that for a query that returns 50,000 rows, the 'group_concat find_in_set' subquery is executed 50,000 times even if pagination is turned on since the limit is applied after it does all the subqueries
If you turn that subquery into a 'superquery', i.e. you do the 'group_concat find_in_set' hydration after the limit, it imrpoves the performance a lot.
I did a proof of concept below. For a query with 500 results it went from 37 seconds to 1.3 seconds.
And for 34,000 it went from never completing to 1.0 seconds.
Not sure how easy it would be to acheive this within the constraints of searchkit @colemanw but it would be great to get your initial thoughts on this.
Proof of concept changes to the query below...
Changing this query:
```sql
SELECT `a`.`id`, (
SELECT GROUP_CONCAT(
`display_name`
ORDER BY FIND_IN_SET(`civicrm_contact`.`id`, REPLACE(`Case_of_abuse_details_1`.`perpetrator_143`, '�', ','))
SEPARATOR '�'
)
FROM `civicrm_contact`
WHERE FIND_IN_SET(`civicrm_contact`.`id`, REPLACE(`Case_of_abuse_details_1`.`perpetrator_143`, '�', ','))
) AS `Case_of_abuse_details.Perpetrator.display_name`
FROM civicrm_activity a
LEFT JOIN `civicrm_value_case_of_abuse_68` `Case_of_abuse_details_1` ON a.id = Case_of_abuse_details_1.entity_id
WHERE ((`a`.`activity_type_id` IS NULL OR (`a`.`activity_type_id` IN (1, 2, 3, 4, 5, 9, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 40, 44, 45, 48, 49, 50, 51, 52, 53, 55, 57, 59, 61, 63, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74)))) AND (`a`.`activity_type_id` = "68") AND (`a`.`is_test` = "0") AND (`a`.`is_deleted` = "0")
GROUP BY `a`.`id`
LIMIT 50
OFFSET 0
```
into this query
```sql
SELECT id,(SELECT GROUP_CONCAT(
`display_name`
ORDER BY FIND_IN_SET(`civicrm_contact`.`id`, REPLACE(`perpetrator_143`, '�', ','))
SEPARATOR '�'
)
FROM `civicrm_contact`
WHERE FIND_IN_SET(`civicrm_contact`.`id`, REPLACE(`perpetrator_143`, '�', ','))
) AS `Case_of_abuse_details.Perpetrator.display_name` FROM (SELECT `a`.`id`, `Case_of_abuse_details_1`.`perpetrator_143`
FROM civicrm_activity a
LEFT JOIN `civicrm_value_case_of_abuse_68` `Case_of_abuse_details_1` ON a.id = Case_of_abuse_details_1.entity_id
WHERE ((`a`.`activity_type_id` IS NULL OR (`a`.`activity_type_id` IN (1, 2, 3, 4, 5, 9, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 40, 44, 45, 48, 49, 50, 51, 52, 53, 55, 57, 59, 61, 63, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74)))) AND (`a`.`activity_type_id` = "68") AND (`a`.`is_test` = "0") AND (`a`.`is_deleted` = "0")
GROUP BY `a`.`id`
LIMIT 50
OFFSET 0) AS alias
```https://lab.civicrm.org/dev/core/-/issues/3070Placeholders in forms2023-12-06T15:27:43ZshaneonabikePlaceholders in formsOverview
----------------------------------------
_Please describe your improvement in detail._
Example use-case
----------------------------------------
1. View a Contribution form
1. Fields such as _First name_ do not have a placehold...Overview
----------------------------------------
_Please describe your improvement in detail._
Example use-case
----------------------------------------
1. View a Contribution form
1. Fields such as _First name_ do not have a placeholder and only a label
Current behaviour
----------------------------------------
I think that in general providing placeholders is a good practice for all fields for the following reasons:
+ More accessible
+ Simplifies design (hidden labels) reduces the size of forms
Proposed behaviour
----------------------------------------
I was thinking there could be two ways to make this happen to start.
+ Provide placeholders that match the labels (Phase 1)
+ Provide ability on all fields with text entry to have a customizable placeholder (Phase 2)
Comments
----------------------------------------
Would there be any problem with me writing a patch that would allow the creation of placeholders for text holders that presently uses the label and copies that into a placeholder field?https://lab.civicrm.org/dev/core/-/issues/3191SearchKit not functional on Joomla 42023-05-24T05:20:40Zjoshjosh@civicrm.orgSearchKit not functional on Joomla 4The past two release of CiviCRM have resulted in SearchKit not functioning on Joomla 4. When browsing to SearchKit, the user is presented with what appears to be unrendered code:
![Screenshot_2022-02-21_12.13.32](/uploads/d52471e665bbfc...The past two release of CiviCRM have resulted in SearchKit not functioning on Joomla 4. When browsing to SearchKit, the user is presented with what appears to be unrendered code:
![Screenshot_2022-02-21_12.13.32](/uploads/d52471e665bbfcaf82f934243c6c9c19/Screenshot_2022-02-21_12.13.32.jpg)
Console presents the following error:
```
Error: [$injector:unpr] http://errors.angularjs.org/1.8.2/$injector/unpr?p0=savedSearchesProvider%20%3C-%20savedSearches%20%3C-%20searchList
at angular.min.js:7:168
at angular.min.js:46:468
at Object.d [as get] (angular.min.js:44:197)
at angular.min.js:47:29
at d (angular.min.js:44:197)
at e (angular.min.js:44:438)
at Object.instantiate (angular.min.js:45:333)
at angular.min.js:99:267
at Object.link (angular-modules.d0a616a75686afef23d3772fcee92a7f.js:5648:217)
at angular.min.js:17:134 '<div ng-view="" class="ng-scope">'
(anonymous) @ angular.js:15697
```
All other aspects of CiviCRM appear to be functional. The issue can be reproduced at https://cividemo.com and on fresh installs of just Joomla and CiviCRM (no extensions installed on either systems).Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/3089Meta: Create list of items for moving a core component to an extension2024-03-08T05:07:35ZDaveDMeta: Create list of items for moving a core component to an extensionIt's never really been fully done before. Some of the Grant things are likely to come up again if it's not documented. Partial list just quickly compiled from tickets. Doesn't include the things that did work (which would still need doin...It's never really been fully done before. Some of the Grant things are likely to come up again if it's not documented. Partial list just quickly compiled from tickets. Doesn't include the things that did work (which would still need doing), and might be missing some PRs that had no tickets.
* https://lab.civicrm.org/dev/core/-/issues/3069
* https://lab.civicrm.org/dev/core/-/issues/3076
* https://lab.civicrm.org/dev/core/-/issues/3056
* https://lab.civicrm.org/dev/core/-/issues/3057
* https://lab.civicrm.org/dev/core/-/issues/3087
* https://lab.civicrm.org/dev/core/-/issues/3093
* https://lab.civicrm.org/dev/core/-/issues/3100
* https://github.com/civicrm/civicrm-buildkit/pull/677
* https://lab.civicrm.org/dev/core/-/issues/3101
* https://github.com/colemanw/webform_civicrm/pull/719
* extendedreports
* https://github.com/eileenmcnaughton/nz.co.fuzion.extendedreport/commit/6184de98c5b9f40475ab4164c993b19d68561240
* https://github.com/eileenmcnaughton/nz.co.fuzion.extendedreport/pull/503
* https://github.com/eileenmcnaughton/nz.co.fuzion.extendedreport/pull/506
* civicrm_entity
* https://github.com/eileenmcnaughton/civicrm_entity/pull/364
* an unresolved cache(?) issue
* https://github.com/twomice/com.joineryhq.jsumfields/pull/23
* https://github.com/civicrm/civicrm-drupal/pull/653
* https://github.com/civicrm/civicrm-drupal/pull/656
* https://github.com/civicrm/civicrm-core/pull/22905
* https://lab.civicrm.org/dev/core/-/issues/3118
* https://lab.civicrm.org/dev/core/-/issues/3119
* https://lab.civicrm.org/dev/core/-/issues/3159
* https://github.com/civicrm/civicrm-core/pull/23116
* https://github.com/civicrm/civicrm-core/pull/23118
* https://github.com/civicrm/civicrm-core/pull/23115
* https://lab.civicrm.org/dev/core/-/issues/3161
* https://github.com/civicrm/civicrm-core/pull/23336
* https://lab.civicrm.org/dev/core/-/issues/3485
* https://lab.civicrm.org/dev/core/-/issues/3492
* https://lab.civicrm.org/dev/core/-/issues/3503
* https://github.com/civicrm/civicrm-core/pull/24191
* https://github.com/civicrm/civicrm-core/pull/26118
Round 2:
* https://github.com/civicrm/civicrm-core/pull/26497
* https://github.com/civicrm/civicrm-core/pull/26499
Late to the party:
* https://lab.civicrm.org/dev/core/-/issues/5075https://lab.civicrm.org/dev/core/-/issues/3096Marking historical contact data - add on hold, hold date, reset date columns ...2023-01-06T22:02:03ZandyburnsMarking historical contact data - add on hold, hold date, reset date columns to civicrm_address and civicrm_phone to match civicrm_emailCurrently only civicrm_email has a mechanism to mark a bad email. This data model should be mirrored over to civicrm_address and civicrm_phone. We currently use the lightweight "location type" approach where we have an Invalid location t...Currently only civicrm_email has a mechanism to mark a bad email. This data model should be mirrored over to civicrm_address and civicrm_phone. We currently use the lightweight "location type" approach where we have an Invalid location type to designate a bad email/phone/address. However, it is limiting as you can only have 1 address per location type for a contact.
This was previously brought up here but was not pursued: https://issues.civicrm.org/jira/browse/CRM-13777.
The rationale is storing older data on these 3 entities helps prevent the creation of duplicates when bringing external sources of data (that may be older as well) upon import and therefore need to remain in the respective db table and not only sent to a log table.
The limitation of 1 address per location type would not apply to on hold addresses, therefore creating a history of addresses.
Setting to on hold would automatically set the hold date.
On hold entities would be shown in a "Former Communications Data" tab much like this extension (https://civicrm.org/extensions/former-communication-data) to prevent cluttering the Contact Summary Screen by only querying the entities that are on hold (and conversely, the contact's summary screen would only show those not on hold) and showing them in descending On Hold date order. Allow filtering by email/phone/address.
On hold entities would not be able to used throughout CiviCRM. E.g. a on hold mobile phone would not show Outbound SMS action, an on hold address would be added as a criteria to this checkbox upon export:
![image](/uploads/d481ef9470febc1620b895af2fb1baff/image.png)
There is obviously overlap with what detailed logging does and this so I'd greatly appreciate anyone's thoughts on how to approach / refine this feature request. We are willing to fund a solution.
Ref:
- https://civicrm.stackexchange.com/questions/681/what-are-best-practices-for-bad-postal-addresses
- https://civicrm.stackexchange.com/questions/20584/need-historical-address-informationhttps://lab.civicrm.org/dev/core/-/issues/3103Document contract for alterMailParams2023-11-23T17:28:30ZDaveDDocument contract for alterMailParamsThe data in $params for alterMailParams is inconsistent, and with recent token changes has changed in some cases.
Copied from https://github.com/civicrm/civicrm-core/pull/22878, which is specific to message template emails (there are al...The data in $params for alterMailParams is inconsistent, and with recent token changes has changed in some cases.
Copied from https://github.com/civicrm/civicrm-core/pull/22878, which is specific to message template emails (there are also other types of emails that trigger alterMailParams):
> Quote
* Add a class CRM_Event_WorkflowMessage_OfflineReceipt. This would look a lot like CRM_Contribute_WorkflowMessage_ContributionOfflineReceipt which describes contribution_offline_receipt. (Also, there's some draft docs in https://lab.civicrm.org/documentation/docs/dev/-/merge_requests/987.)
* Add a variant of hook_alterMailParams which gives the message as an object. Here's a sketch for firing+consuming hook_alterWorkflowMessage [gist](https://gist.github.com/totten/14178dabea79ea2a5a639a5b928262e2).https://lab.civicrm.org/dev/core/-/issues/3121The mysql session timezone that civi sets does not observe daylight saving (a...2023-12-09T05:04:30ZDaveDThe mysql session timezone that civi sets does not observe daylight saving (aka The Contact Logging civireport can show the wrong time)Pulling this out as a separate issue from #2122. It's also different from that issue in that here the value stored in the database is correct, just the display is wrong.
Mysql/mariadb often do not ship with named timezones populated (i....Pulling this out as a separate issue from #2122. It's also different from that issue in that here the value stored in the database is correct, just the display is wrong.
Mysql/mariadb often do not ship with named timezones populated (i.e. the mysql.time_zone table and related tables are empty). So when civi sets the mysql timezone for the session it calculates an offset instead (e.g. '+01:00'), based on "now". Offsets don't know about DST rules, so when it's converted for display based on that "now" offset, it will be wrong if "now" is on the other side of DST from the date that was stored in the database. Example:
Suppose log_civicrm_contact has a log_date with a value of March 8, 2022 12:34, created in timezone America/New_York. When viewed on March 9, 10, 11, etc it displays correctly.
When viewed on, say, March 16, civi will set the mysql timezone as '-04:00', which is different from what it was on March 8 ('-05:00'). Therefore when it displays, it shows 13:34 instead of 12:34.
This only affects timestamp columns. Datetime columns don't do any conversion based on session timezone.
The code where this happens is at https://github.com/civicrm/civicrm-core/blob/1754e1752eb8e516ec846a438005f6fcb55be247/CRM/Utils/System/Base.php#L760
Note also that getTimeZoneString() does different things for different CMSs, so you may experience the issue slightly differently depending on the install.https://lab.civicrm.org/dev/core/-/issues/3130Lack of hooks to detect when an Attachment is deleted2023-12-07T08:27:58ZhaystackLack of hooks to detect when an Attachment is deletedOverview
----------------------------------------
At present, when an Attachment is deleted via the CiviCRM UI (e.g. on "Edit Activity" dialogs/screens) only two CiviCRM hooks fire: the `civi.dao.preDelete` and `civi.dao.postDelete` Symf...Overview
----------------------------------------
At present, when an Attachment is deleted via the CiviCRM UI (e.g. on "Edit Activity" dialogs/screens) only two CiviCRM hooks fire: the `civi.dao.preDelete` and `civi.dao.postDelete` Symfony events triggered by `CRM_Core_BAO_EntityTag::del()`. All other deletions are done via direct SQL queries.
It would be great to:
* Have `hook_civicrm_pre` and `hook_civicrm_post` fire
* Have some way of retrieving the Attachment data prior to deletion
Reproduction steps
----------------------------------------
1. Set up CiviCRM to log all hooks
1. Click the "bin/trash" icon next to an Attachment (as per the screenshot below)
1. Inspect logged callbacks
![attachments-accordion](/uploads/9abded9e0e7f9cc602213b8b20de345d/attachments-accordion.png)
Current behaviour
----------------------------------------
The following callbacks are the only ones that will be seen (data trimmed for concision)
#### `civi.dao.preDelete`
```
[event] => Civi\Core\DAO\Event\PreDelete Object
(
[object] => CRM_Core_BAO_EntityTag Object
(
[id] =>
[entity_table] => civicrm_file
[entity_id] => 6
[tag_id] =>
)
)
```
#### `civi.dao.postDelete`
```
[event] => Civi\Core\DAO\Event\PostDelete Object
(
[object] => CRM_Core_BAO_EntityTag Object
(
[id] =>
[entity_table] => civicrm_file
[entity_id] => 6
[tag_id] =>
)
[result] => 0
)
```
It is possible to use the `entity_id` to retrieve the "File" data via the CiviCRM API. However, because of the order of deletions in [the `deleteEntityFile()` method](https://github.com/civicrm/civicrm-core/blob/6769e8bf8556701b81f914d30fc1bb913f8ed2ce/CRM/Core/BAO/File.php#L250), it is _not_ possible to retrieve the compound "Attachment" data via the API (because the "Entity File" data has already been deleted) and it is therefore not possible to find out which Entity/Entities the File was attached to.
It should also be noted that when the "Delete All Attachment(s)" checkbox is used, the `civi.dao.preDelete` and `civi.dao.postDelete` Symfony events fire *before* `hook_civicrm_pre` is fired for the Activity. To me, it would make more sense for `hook_civicrm_pre` to fire *before* the File deletion process is initiated so that `Activity.pre` and `Activity.post` "wrap" the entire process.
Expected behaviour
----------------------------------------
I would expect to be able to receive callbacks from `hook_civicrm_pre` and `hook_civicrm_post`, where `$op = "delete"` and `$objectName = "File"` along with some more contextual data about the "File" and the "Attachment" being deleted.
Comments
----------------------------------------
I'm going to open a PR that addresses the issue of retrieving the "Attachment" data via the API when using the Symfony hook, since that's pretty straightforward.
I'd appreciate any guidance from those more familiar with this class on the most sensible place(s) to add hooks during the deletion process - or whether deletion should more properly be done via `BAO` objects instead of direct queries.https://lab.civicrm.org/dev/core/-/issues/3139Badgelayouts cannot be edited with PHP warning2022-09-01T13:49:35ZBradley TaylorBadgelayouts cannot be edited with PHP warning_Reproduced on dmaster and locally on WordPress_
**Steps to reproduce**
1. Navigate to "Administer CiviCRM", "Event Name Badge Layouts".
2. Create a new name badge
3. Edit the newly created name badge.
**Expected outcode**
The edit scr..._Reproduced on dmaster and locally on WordPress_
**Steps to reproduce**
1. Navigate to "Administer CiviCRM", "Event Name Badge Layouts".
2. Create a new name badge
3. Edit the newly created name badge.
**Expected outcode**
The edit screen should be pre-filled with the values entered initially.
**Actual outcome**
Each field is blank, a PHP warning is shown:
![Screenshot_2022-03-27_at_10.18.33](/uploads/105dc6046cbb9b569207500da40cf77f/Screenshot_2022-03-27_at_10.18.33.png)
**Technical explanation**
The bug was introduced in https://github.com/civicrm/civicrm-core/commit/873bfeb503caa413f17460dbe450b74fac3d6dbf.
The commit above added a new tokens:
```
'{event.start_date|crmDate:"%B %E%f"}' => ts('Event Start Date'),
'{event.end_date|crmDate:"%B %E%f"}' => ts('Event End Date'),`
```
The data for badge layouts is stored as encoded JSON. This means that the quote marks in these two tokens are being wrapped in double-quotes for the string, causing something like `"{event.start_date|crmDate:"%B %E%f"}"`. As such the JSON is not valid and cannot be `json_decode`ed.
The actual fix could be straightforward enough: Switch the tokens to use single instead of double quotes. However, I'm not sure what the correct solution is for any broken JSON which is now stored in CiviCRM databases. Some sort of upgrade script might be required to find/replace the known broken JSON.
Pinging @eileen who did a lot of work on tokens last year.https://lab.civicrm.org/dev/core/-/issues/3148Dedupe with multi-select custom fields can trigger IDS2023-03-18T04:20:40ZJonGoldDedupe with multi-select custom fields can trigger IDSWhen deduping contacts that have multi-select custom fields, and selecting to move the custom fields to the new contact, the IDS is triggered.
### Steps to replicate
* Create a custom field that allows saving multiple values (e.g. a che...When deduping contacts that have multi-select custom fields, and selecting to move the custom fields to the new contact, the IDS is triggered.
### Steps to replicate
* Create a custom field that allows saving multiple values (e.g. a checkbox). Note that you need several of these to trigger the "kick" on the IDS (3, I think).
* Create two contacts that are duplicates.
* Find and merge the records.
### Expected result
Contacts merged successfully.
### Actual result
"Your activity is a bit suspicious, hence aborting"
The issue is the POST request, which is passing arguments like `move_custom_12` with the `VALUE_SEPARATOR` control character. This triggers the IDS filter labeled "Detects nullbytes and other dangerous characters".
I'm really not certain what the correct answer is here - I can exempt users from the IDS, and maybe that's the solution to pursue, but it seems like there should be another solution available. Is it possible to exempt certain paths from the IDS, or use an alternate set of rules for a certain path?
Keyword: Intrusion Detection Systemhttps://lab.civicrm.org/dev/core/-/issues/3152Data stored in universal time does not handle DST consistently2023-11-30T05:00:20ZtottenData stored in universal time does not handle DST consistently[[_TOC_]]
Overview
----------------------------------------
CiviCRM has a number of `TIMESTAMP` columns -- these are stored in universal time (UTC) and displayed in the user's timezone. However, there is a subtle error in handling Dayl...[[_TOC_]]
Overview
----------------------------------------
CiviCRM has a number of `TIMESTAMP` columns -- these are stored in universal time (UTC) and displayed in the user's timezone. However, there is a subtle error in handling Daylight Savings Time (DST): if the current-date and the target-date sit on different sides of the DST-switch, then the time may present as +/- 1 hour.
This bug was one of the major subissues identified in https://lab.civicrm.org/dev/core/-/issues/2122. Although that particular feature was rolled-back/deferred from 5.47, the DST bug still exists -- it's just less obvious.
Example use-case
----------------------------------------
1. Create a contact record.
2. View the contact record. Note the creation time (`civicrm_contact.created_date`).
3. Change the system clock - set to a date where DST differs (eg if today is March 30, then go to December 5).
4. View the contact record. Note the creation time (`civicrm_contact.created_date`).
Current behavior
----------------------------------------
The displayed value of `civicrm_contact.created_date` _appears_ to change by +/- 1 hour, depending on when you view it.
> (Viewed on Mar 31, 2022)
>
> ![Screen_Shot_2022-03-31_at_12.40.53_AM](/uploads/8a2f5a0e4fb6ec884aede78e30d31b1a/Screen_Shot_2022-03-31_at_12.40.53_AM.png)
> (Viewed on Dec 5, 2022)
>
> ![Screen_Shot_2022-12-05_at_12.44.57_AM](/uploads/2df387488ee029a949010da163bf2ea8/Screen_Shot_2022-12-05_at_12.44.57_AM.png)
Why? CiviCRM sends a note to MySQL about the current user's timezone (`SET time_zone = '...'`). However, it doesn't identify the timezone effectively. It gives [the current numeric offset (at the moment of viewing)](https://github.com/civicrm/civicrm-core/blob/5.47.3/CRM/Utils/System/Base.php#L758-L762) - but (in locales with DST) the offsets fluctuate over time.
(_Ex: On Mar 31, the offset in California is `-0700`. Under current/long-standing law, the offset will be `-0800` on Dec 5. Of course, the US Congress is reconsidering this law... so we don't really know what the offset will be!_)
Proposed behavior 1: Fix MySQL timezones
----------------------------------------
CiviCRM should send the timezone as a symbolic name, such as `Europe/Helsinki`, `America/Los_Angeles`, or `Australia/Sydney`. These symbolic-names have an underlying database which allows them adjust automatically based on DST-rules/target-dates/current-law. On the surface, the fix is extremely simple:
```diff
diff --git a/CRM/Utils/System/Base.php b/CRM/Utils/System/Base.php
index a4660834c5..8e40f6da35 100644
--- a/CRM/Utils/System/Base.php
+++ b/CRM/Utils/System/Base.php
@@ -755,10 +755,9 @@ abstract class CRM_Utils_System_Base {
* Set timezone in mysql so that timestamp fields show the correct time.
*/
public function setMySQLTimeZone() {
- $timeZoneOffset = $this->getTimeZoneOffset();
- if ($timeZoneOffset) {
- $sql = "SET time_zone = '$timeZoneOffset'";
- CRM_Core_DAO::executequery($sql);
+ $timeZone = $this->getTimeZoneString();
+ if ($timeZone) {
+ CRM_Core_DAO::executequery('SET time_zone = %1', [1 => [$timeZone, 'String']]);
}
}
```
There are a couple of catches.
* __Timezone rules change__ (occasionally). Any software that supports timezones ultimately needs a _data feed_ with current rules. The good news: IANA publishes a free/open feed (https://www.iana.org/time-zones; aka `tzdata`; aka `zoneinfo`), most Linux/Unix distros have this feed, and MySQL can read it (`mysql_tzinfo_to_sql`). It usually requires one command (which could run during system-config, system-startup, and/or cron):
```bash
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql mysql
```
The problem: we have no measures for (a) how many CiviCRM deployments actually subscribe to this feed and (b) how many could subscribe, if they chose to.
* __Timezone names may be inconsistent__ (occasionally). For example, in different contexts, it's been fashionable to refer to California's timezone as `America/Los_Angeles`, `US/Pacific`, and `PST8PDT`. (The current+official fashion is `America/Los_Angeles` - the others are deprecated.) However, since Civi integrates with various layers (different CMSs; PHP APIs; MySQL APIs), there are edge-case where the layers may choose different names. (*I'm not super-concerned, but we should raise sensible warnings when names are invalid or mismatched.*)
The central issue is - how to cope when data isn't available? This comes to mind:
* (Status check) If the active TZ (`getTimeZoneString()`) has a deprecated name (eg `PST8PDT`) or an offset (eg `-0700`), show a warning.
* (Status check) If the active TZ (`getTimeZoneString()`) isn't supported by MySQL, show a warning.
* (Runtime) If the active TZ (`getTimeZoneString()`) isn't supported by MySQL, fallback to sending offset.
Proposed behavior 2: Change format. Use only PHP TZs.
----------------------------------------
(_This expands on one of @haystack's suggestions in dev/core#2122._)
If you assume that MySQL time services aren't available - what else would you do? You could use PHP time services.
The astute observer will note the status-quo (using both PHP and MySQL time-services) creates two points-of-failure. If either PHP _or_ MySQL has bad/incomplete/old timezone data, then you'll get mis-calculations _somewhere_. Consolidating on PHP time-services would reduce the #dependencies.
Both Drupal and WordPress take this approach. (I suspect this is extremely useful for maximizing compatibility with heterogeneous web-hosts.) They each do it a bit differently, but some central concepts are the same:
* In Drupal, PHP processes read+write temporal data in universal time -- as an `INT` (Unix-style, seconds-since-epoch).
* In WordPress, PHP processes read+write temporal data in universal time -- as a `DATETIME` with a `_gmt` suffix (eg `post_modified_gmt`).
* Hypothetically, you could hardcode MySQL to `SET time_zone='+0:00'`. PHP processes would read+write temporal data in universal time -- as a `TIMESTAMP`.
In all those cases, the onus is on the PHP devs to convert to/from universal-time when implementing functionality (eg "find records from March 1 - March 15" or "find records from this afternoon" or "extract the hour:minute component").
But there is a catch here: Civi already relies on several MySQL time-services. The schema works a certain way; the reports/searches/UIs/APIs expect the schema to work a certain way; etc.
The central issue is - how do you manage/QA all the changes (in schema+logic) required to change time-service?
Comments
----------------------------------------
* I haven't tested, but I'm fairly certain there will be another manifestation in CiviMail scheduling. Ex:
* You live in a timezone where DST changes on March 16.
* On March 10, you schedule a mail-blast for 2:00pm on March 20. It stores the schedule with the wrong offset.
* When March 20 comes, the mailing actually goes out at 3:00pm (or maybe 1:00pm).https://lab.civicrm.org/dev/core/-/issues/3154Custom tokens not working in Scheduled Reminders2022-08-25T21:35:03ZmartyCustom tokens not working in Scheduled RemindersOverview
----------------------------------------
Custom tokens are not evaluated for Scheduled Reminders when initiated by Cron Job. The tokens are evaluated properly when the Scheduled Reminders job is run manually using the Execute No...Overview
----------------------------------------
Custom tokens are not evaluated for Scheduled Reminders when initiated by Cron Job. The tokens are evaluated properly when the Scheduled Reminders job is run manually using the Execute Now option.
Reproduction steps
----------------------------------------
1. Create a custom token using hook_civicrm_container() and implement the civi.token.list and civi.token.eval event listeners.
1. Add a new Scheduled Reminder (I'm using membership end date) and include the custom token in the email message.
1. Create a Cron Job to run civicrm/bin/cron.php periodically (I run every 15 minutes).
1. Enable the Send Scheduled Reminders job and set to run Always.
1. Trigger the reminder appropriately (I create a new membership and set the end date to trigger).
1. Note the custom token is __not__ included in the resulting email after the cron run.
1. Now trigger a new reminder and click Execute Now on the Scheduled Reminders job (before the next cron run).
1. Note the custom token __is evaluated properly__ and included in the resulting email message.
Current behaviour
----------------------------------------
Custom token not included in Scheduled Reminder email when initiated by Cron Job
Expected behaviour
----------------------------------------
Custom token should be included in Scheduled Reminder email when initiated by Cron Job.
Environment information
----------------------------------------
* __CiviCRM:__ _5.47.2_
* __PHP:__ _7.4.28_
* __CMS:__ _WordPress 5.9.2_
* __Database:__ _MySQL_
* __Web Server:__ _Apache_https://lab.civicrm.org/dev/core/-/issues/3789Site language set back to English after update to 5.51.02023-06-16T02:58:58Zpal_urSite language set back to English after update to 5.51.0Overview
----------------------------------------
I've got a CiviCRM site running on Drupal7, with Hungarian language settings.
There are two available languages, English and Hungarian.
I updated my CiviCRM 5.50.3 to 5.51.0, than to ...Overview
----------------------------------------
I've got a CiviCRM site running on Drupal7, with Hungarian language settings.
There are two available languages, English and Hungarian.
I updated my CiviCRM 5.50.3 to 5.51.0, than to 5.52.0, and I lost my site language: it's set back to English, and I'm not able to set back to Hungarian. My translation file is in /sites/all/modules/civicrm/l10n/hu_HU, and the site language is set to hungarian, and in the `civicrm_setting` table there are the following values: `contact_default_language`: `s:5:"hu_HU";`, `lcMessages` `s:5:"hu_HU";`, `uiLanguages` `a:2:{i:0;s:5:"en_US";i:1;s:5:"hu_HU";}`, `format_locale` `s:5:"hu_HU";`.
Reproduction steps
----------------------------------------
Site works with English language.
Current behaviour
----------------------------------------
Expected behaviour
----------------------------------------
Because of the settings I'd like to use my site in Hungarian.
Environment information
----------------------------------------
* __CiviCRM:__ _Master/5.51.0/5.52.0_
* __PHP:__ _PHP 7.4.30 (cli)__
* __CMS:__ Drupal 7.91_
* __Database:__ _10.3.34-MariaDB-0+deb10u1 - Debian 10_
* __Web Server:__ _Apache/2.4.38 (Debian)_
Comments
----------------------------------------https://lab.civicrm.org/dev/core/-/issues/3185Tags for attachments are not properly assigned to the attachment2023-10-07T21:15:50ZDaveDTags for attachments are not properly assigned to the attachmentI'm not sure if this is recent or where it's happening. What happens is that civicrm_entity_tag.entity_id is one higher than the appropriate id from civicrm_file, e.g.
civicrm_file:
| id | file_type_id | mime_type | uri ...I'm not sure if this is recent or where it's happening. What happens is that civicrm_entity_tag.entity_id is one higher than the appropriate id from civicrm_file, e.g.
civicrm_file:
| id | file_type_id | mime_type | uri |
|------|--------------|-------------------|-------------------------------------------------|
|__28__| NULL | text/plain | abc_aef1644a7b96451b6c15b7e34b862f5d.txt |
civicrm_entity_tag:
| id | entity_table | entity_id | tag_id |
|----|------------------|---------------|--------|
| 57 | civicrm_file | --> __29__ <--| 31 |
1. Create a tag set for attachments
1. Create some tags for the set
1. Create an activity
1. In the attachments section add a file and choose a tag
1. When you go back to view or edit the activity, the tag isn't displayed. Check the db and you'll see the entity_id doesn't match up.
1. Manually edit the entity_id in the db and then go back to view the activity. Now the tag is displayed.https://lab.civicrm.org/dev/core/-/issues/3440Check for matching contact on contact add form sends hardcoded fields to dupl...2023-07-06T07:25:40ZdarrickCheck for matching contact on contact add form sends hardcoded fields to duplicatecheck api callOverview
----------------------------------------
If a custom Supervised rule is created using any field not in ['first_name', 'last_name', 'nick_name', 'household_name', 'organization_name', 'email'] then clicking on the **Check for Mat...Overview
----------------------------------------
If a custom Supervised rule is created using any field not in ['first_name', 'last_name', 'nick_name', 'household_name', 'organization_name', 'email'] then clicking on the **Check for Matching Contact** button will return no results.
Reproduction steps
----------------------------------------
1. Click on **Contacts -> Find and Merge Duplicate Contacts**.
2. Click on **Add Individual Rule**
3. Add a rule with field phone, weight 10, threshold 10
4. Click on **Change rule** and set to **Supervised**
5. Click on **Contacts -> New Individual**
6. Add a contact with First Name: Bob, Last Name: Dobbs and phone 666.666.6666
7. Save the contact.
8. Click on **Contacts -> New Individual**
9. Add a contact with First Name: Bob, Last Name: Dobbs and phone 666.666.6666
10. Click on **Check for Matching Contact**
Current behaviour
----------------------------------------
A popup displays "Similar contact if found" after entering the First Name and after entering the Last Name in step 9.
After entering the duplicate phone number nothing happens.
After clicking on **Check for Matching Contact** nothing happens.
Expected behavior
----------------------------------------
After entering either First Name or Last Name nothing should happen unless the entered field is included in the Supervised rule.
A popup displaying "Similar contact if found" should happen after the phone number is entered and also after clicking on **Check for Matching Contact**
Comments
----------------------------------------
I ran across this while looking to see if I could fix any other outstanding bugs related dedupe. Was working on this one: (https://lab.civicrm.org/dev/core/-/issues/2966) I wasn't able to reproduce their issue.
It may still be useful to hard code those fields so by default the form always matches on name or email when entering those fields but then the additional fields will also be searched when added using the custom rule. As any fields not needed for the custom rule will just be ignored.https://lab.civicrm.org/dev/core/-/issues/3441When generating emails from Search results, activity is recorded as a Print P...2023-03-06T20:39:37ZStoobWhen generating emails from Search results, activity is recorded as a Print PDF DocumentSteps to reproduce:
1. use Search Kit to generate results as _Contributions_
2. choose to send emails receipts
3. emails are sent, but activity is recorded "Print/Merge Document" rather than Type "Email".
#searchkitSteps to reproduce:
1. use Search Kit to generate results as _Contributions_
2. choose to send emails receipts
3. emails are sent, but activity is recorded "Print/Merge Document" rather than Type "Email".
#searchkithttps://lab.civicrm.org/dev/core/-/issues/3586Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process ...2023-09-12T02:13:33Zjustinfreeman (Agileware)Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process any emails if a single email is sent to the bounce mailbox with an invalid returnPathCiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service ema...CiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service email" causing all other emails to remain unprocessed. CiviCRM mail reports are therefore incorrect and ultimately the CiviCRM Fetch Bounces scheduled job fails to perform the job it was designed to do.
Example invalid returnPaths as observed in production:
```
x2108-arfsfufwyfgkvni2x7ww5wvac3eifc2wx3vr1fcwbbysieqppis4ytnyip09xcohck9s1vrm77nbg4xh431tthdsmtmfvfpa@rspf.mxthunderstruck.net
```
```
SRS0=3Ypr33=QM=crm.nothappyjohn.org.au=bounce+b.25211.6039846.eb7e7e8ba282f9d0@airquotes.com.au
```
This problem has been highlighted by the introduction of "Scheduled Job Failures" feature which raises the visibility of this type of problem.
Bounce mailboxes have been observed with 10k to 50k unprocessed emails, depending on the time when the "Denial of service email" was received.
Agileware Ref: CIVICRM-1970https://lab.civicrm.org/dev/core/-/issues/3584Denial of service - CiviCRM Mailing Job, if there are 5 errors sequentially w...2023-11-09T13:25:31Zjustinfreeman (Agileware)Denial of service - CiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and mailing will not sendCiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and the mailing will not send.
An invalid email domain returns an SMTP error 451: _SMTP error 451 Unable to ...CiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and the mailing will not send.
An invalid email domain returns an SMTP error 451: _SMTP error 451 Unable to complete command, DNS not available or timed out 451 Domain of sender address does not resolve_
When the CiviCRM Mailing Job executes again, the process repeats, the same invalid email addresses are used, the same errors are returned and the mailing again will not send.
This process will repeat until someone intervenes, cancelling the mailing, locating and removing the contacts which have the invalid email addresses.
The CiviCRM Mailing Job does not currently check the actual SMTP error code and just treats all SMTP errors as "SMTP Connection Errors". Counting up to 5 and then aborting the job when the threshold is reached.
I think it's fair to call this a "denial of service" because a bad actor could sign up a bunch of fake Contacts to a CiviCRM mailing list with either invalid email domains or valid email domains which are then rendered invalid - and thus cause that mailing list to cease processing.
**Possible solutions**
Change the CiviCRM Mailing Job to check the return SMTP error code and only increment the count for valid "SMTP Connection Errors".
Change the CiviCRM Mailing Job to continue with the mailing, regardless of the 5 errors encountered, which will also skip sending the emails to those Contacts with an invalid email domain. As shown in the patch below.
```
Index: CRM/Mailing/BAO/MailingJob.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/CRM/Mailing/BAO/MailingJob.php b/CRM/Mailing/BAO/MailingJob.php
--- a/CRM/Mailing/BAO/MailingJob.php (revision 20a0376709a4616e32689a821a7e95ca255ed85f)
+++ b/CRM/Mailing/BAO/MailingJob.php (date 1620794367592)
@@ -672,20 +672,9 @@
if ($smtpConnectionErrors <= 5) {
$mailer->disconnect();
$retryGroup = TRUE;
+ CRM_Core_Error::debug_log_message("More than 5 consecutive SMTP Socket Errors. Re-starting mailer.");
continue;
}
-
- // seems like we have too many of them in a row, we should
- // write stuff to disk and abort the cron job
- $this->writeToDB(
- $deliveredParams,
- $targetParams,
- $mailing,
- $job_date
- );
-
- CRM_Core_Error::debug_log_message("Too many SMTP Socket Errors. Exiting");
- CRM_Utils_System::civiExit();
}
// Register the bounce event.
Index: ext/flexmailer/src/Listener/DefaultSender.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/ext/flexmailer/src/Listener/DefaultSender.php b/ext/flexmailer/src/Listener/DefaultSender.php
--- a/ext/flexmailer/src/Listener/DefaultSender.php (revision 20a0376709a4616e32689a821a7e95ca255ed85f)
+++ b/ext/flexmailer/src/Listener/DefaultSender.php (date 1620794390628)
@@ -76,15 +76,9 @@
if ($smtpConnectionErrors <= 5) {
$mailer->disconnect();
$retryBatch = TRUE;
+ \CRM_Core_Error::debug_log_message("More than 5 consecutive SMTP Socket Errors. Re-starting mailer.");
continue;
}
-
- // seems like we have too many of them in a row, we should
- // write stuff to disk and abort the cron job
- $job->writeToDB($deliveredParams, $targetParams, $mailing, $job_date);
-
- \CRM_Core_Error::debug_log_message("Too many SMTP Socket Errors. Exiting");
- \CRM_Utils_System::civiExit();
}
else {
$this->recordBounce($job, $task, $result->getMessage());
```
Agileware Ref: CIVICRM-1728https://lab.civicrm.org/dev/core/-/issues/3559CiviCRM bounce processing, processed emails are never deleted from the mailbo...2023-09-08T22:34:57Zjustinfreeman (Agileware)CiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little valueCiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little value.
Possible solutions:
1. Once a bounce email has been processed it should be deleted from the mailbox,...CiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little value.
Possible solutions:
1. Once a bounce email has been processed it should be deleted from the mailbox, or
1. A new Scheduled Job process should be implemented which deletes all processed email which is older than 30 days
Relates to https://lab.civicrm.org/dev/mail/-/issues/108
Agileware Ref: CIVICRM-1977https://lab.civicrm.org/dev/core/-/issues/3714CiviCRM Contribution Page, total amount calculation is sometimes off by one o...2023-07-18T05:40:43Zjustinfreeman (Agileware)CiviCRM Contribution Page, total amount calculation is sometimes off by one or two cents, which causes incorrect amount to be paid or transaction to fail depending on the Payment Processor being usedWhen using a **CiviCRM Contribution Page** the total amount calculation is sometimes off by one or two cents, which causes incorrect amount to be paid or transaction to fail depending on the Payment Processor being used.
In the case of ...When using a **CiviCRM Contribution Page** the total amount calculation is sometimes off by one or two cents, which causes incorrect amount to be paid or transaction to fail depending on the Payment Processor being used.
In the case of Stripe, the _correct amount_ is _pre-authorised_. When the payment is _confirmed_ the _incorrect amount_ is used and because the two amounts do not match, the **transaction fails**. As a result this bug prevents payments from working with Stripe as a payment processor for certain amounts.
# Steps to reproduce
1. Enable tax and invoicing
2. Add e.g. a 10% GST account to Member Dues
3. Create a membership prices set with one field that has three checkbox options:
4. 318.18182 (350 incl tax)
5. 22.72727 (25 incl tax)
6. 24.54545 (27 after tax - note that this would actually work out to 27.01 with the older premature rounding based off-by one issue)
7. Use this price set on a contribution page
8. Submit a test using this contribution page.
9. Note in the summary that the line totals are 350.00, 25.00, and 27.00
## Expected result
Total should be **$402**
## Actual result
Total is **$401.99** - which is one cent less than the expect value, and two cents less than the value you would get with premature rounding.
## Tested environment
Bug has been confirmed using **CiviCRM 5.52.alpha1** on https://wpmaster.demo.civicrm.org/
## Workaround
Adjust the amount to account for the rounding error. In this case, change the amount: 24.54545 to 24.5363636364 which then changes the total to $401.99. This is then matches the total on the Thank You page.
![image](/uploads/ddf3c7cbeaf3cdc4700079f79b01c7d6/image.png)
![image](/uploads/5eb33f84dfa49cad5697570a019c6936/image.png)
# Contribution page with membership options selected - correct total is shown
![image](/uploads/f35b520763e5e9f3545585f27ff42812/image.png)
# Thank you page - incorrect total is shown
![image](/uploads/d6fa209c43cbf2d721d4cf085882b1f8/image.png)
# Set up - Membership Types
![image](/uploads/01450b1f2a777f3c279e694d5ea7e007/image.png)
# Set up - Priceset
![image](/uploads/672f5c3de5b72e7817dc063388f08c38/image.png)
Agileware Ref: CIVICRM-2007https://lab.civicrm.org/dev/core/-/issues/3740Set default lock level to READ COMMITTED2023-10-20T07:57:25ZJoeMurraySet default lock level to READ COMMITTEDDrupal recently changed the default lock level slightly down from REPEATABLE READ to READ COMMITTED, which has the effect of avoiding some db locks. See https://www.drupal.org/node/3269885 We have seen over recent years some issues with ...Drupal recently changed the default lock level slightly down from REPEATABLE READ to READ COMMITTED, which has the effect of avoiding some db locks. See https://www.drupal.org/node/3269885 We have seen over recent years some issues with deadlocks, primarily on rebuilding group contact cache when there are hierarchies of smart groups. Should CiviCRM be more explicit about what isolation level we expect in order to get more repeatable results, and should we follow Drupal in its move to READ COMMITTED? I tend to think so.
See responses in MM chat from @demeritcowboy
https://chat.civicrm.org/civicrm/pl/gnmhfdxes7rgb8k7ppoaezo45ahttps://lab.civicrm.org/dev/core/-/issues/3759Logging table schemas should be updated when extension table schemas are updated2022-08-24T19:17:07ZJonGoldLogging table schemas should be updated when extension table schemas are updatedI just ran into an issue while enabling Geocoder on a relatively fresh Civi install. `install.sql` generated a table, then `hook_civicrm_managed` tried adding all those record in `.mgd.php`, which failed because `log_civicrm_geocoder` d...I just ran into an issue while enabling Geocoder on a relatively fresh Civi install. `install.sql` generated a table, then `hook_civicrm_managed` tried adding all those record in `.mgd.php`, which failed because `log_civicrm_geocoder` didn't exist.
I know this isn't a simple lift - but given the recent conversations that `civix` might phase out the generation of `.sql` files, this seems like the opportune moment to raise the issue.https://lab.civicrm.org/dev/core/-/issues/3769Change the default font family to sans-serif in print.css2024-03-19T20:19:43ZwmortadaChange the default font family to sans-serif in print.cssThe font family is set as `DejaVu Sans` but the fallback is `serif` in `print.css`. Surely this should be `sans-serif`?
See https://github.com/civicrm/civicrm-core/blob/master/css/print.css#L23
```
#crm-container {
overflow: visible ...The font family is set as `DejaVu Sans` but the fallback is `serif` in `print.css`. Surely this should be `sans-serif`?
See https://github.com/civicrm/civicrm-core/blob/master/css/print.css#L23
```
#crm-container {
overflow: visible !important;
font-family: DejaVu Sans, serif;
margin: 0px 10px 0px 10px;
}
```https://lab.civicrm.org/dev/core/-/issues/3813Performance issue on Redis with arrays2022-11-15T04:01:55ZeileenPerformance issue on Redis with arraysBoth WMF and AUG have achieved a significant performance improvement with [this patch](https://github.com/civicrm/civicrm-core/pull/24156) which removes unnecessary calls to the underlying cache provider. However, in both cases we are us...Both WMF and AUG have achieved a significant performance improvement with [this patch](https://github.com/civicrm/civicrm-core/pull/24156) which removes unnecessary calls to the underlying cache provider. However, in both cases we are using `Redis` and `Redis` itself is performing well.
I'm left with the conclusion that the issue is the way in which the php layer is handling arrays when using `redis`.
On digging we are specifically calling [`serialize`](https://github.com/civicrm/civicrm-core/blob/53a48c5aacea52dd6507a45e111a00ccbed0fdaa/CRM/Utils/Cache/Redis.php#L116) and [`unserialize`](https://github.com/civicrm/civicrm-core/blob/53a48c5aacea52dd6507a45e111a00ccbed0fdaa/CRM/Utils/Cache/Redis.php#L138) in `CRM_Utils_Cache_Redis`.
Discussion on the interweb thingee suggests that `json_encode` & `json_decode` *may* be faster but that is a moving target over php versions .... https://stackoverflow.com/questions/22718903/storing-an-array-of-data-using-redis-from-laravel
Another option appears to be the `h` methods within the `Redis php` class - I note in our `PrevNextRedisCache` implementation we do use some of these hash specific methods and I wonder if using them here would help
https://stackoverflow.com/questions/22718903/storing-an-array-of-data-using-redis-from-laravelhttps://lab.civicrm.org/dev/core/-/issues/3817Question/Discussion: Inconsistencies between "access CiviCRM" and "access AJA...2022-09-01T11:12:34ZJohn TwymanQuestion/Discussion: Inconsistencies between "access CiviCRM" and "access AJAX API" permission grants?Overview
----------------------------------------
We are developing client applications that integrate with CiviCRM via its API and the AuthX extension. This allows us to query the API as the user, rather than as the client applications....Overview
----------------------------------------
We are developing client applications that integrate with CiviCRM via its API and the AuthX extension. This allows us to query the API as the user, rather than as the client applications.
Users of our client applications do not need, and should not have, the "access CiviCRM" permission. So we have been building our apps on the basis of using the "access AJAX API" permission instead.
Unfortunately, we have discovered that almost every API call involving core entities assumes "access CiviCRM" as the baseline permission for use. `Group.get`, `Participant.get`, etc., [as defined in CRM/Core/Permission.php](https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/Permission.php#L968).
Perhaps we have mistakenly assumed that "access AJAX API" was designed as a functionally equivalent permission to "access CiviCRM", minus the CiviCRM UI access.
Should the "access AJAX API" permission have the same baseline (API) permissions as "access CiviCRM"?
(I see a bigger challenge for us here in terms of the range of permissions required for certain calls, eg. Participant.get requires 'access civicrm', 'access civievents', view all participants', but one problem at a time)
Reproduction steps
----------------------------------------
1. Set up a user with an API key, etc., and a role that does _not_ have the `access civicrm` permission but does have the `acess ajax api` one
2. Query the API v3 REST endpoint with the user's credentials; eg. Group.get, Contact.get, etc.
3. Get an API permissions error response
Current behaviour
----------------------------------------
Many/most API calls (at least Entity.get calls) made by users with only the 'access ajax api' call return a permissions denied error: 'require "access civicrm"
Expected behaviour
----------------------------------------
Many/most API calls made by these users should return results
Environment information
----------------------------------------
* __CiviCRM:__ 5.49.5
* __PHP:__ 7.4/
* __CMS:__ Drupal 7.91/
* __Database:__ MariaDB 10.4.21_https://lab.civicrm.org/dev/core/-/issues/3818Trigger based logging - improve archivability2022-08-30T10:59:52ZeileenTrigger based logging - improve archivabilityWe would like to time-limit our database logging but there is a challenge when deleting old rows from the `log_` tables.
Example rows from `log_civicrm_contact`
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ ...We would like to time-limit our database logging but there is a challenge when deleting old rows from the `log_` tables.
Example rows from `log_civicrm_contact`
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ |
| 2015-11-09 | Initialize |8|Bob|
| 2018-09-09 | Update |8|Robert|
| 2022-10-09 | Update |8|John|
In each case the rows are logged to have the status of the updated value. So on 2022-10-09 the value in `civicrm_contact` is "John"
If the change was made in error the change can be rolled back to the last change - ie 'Robert'
However, if we say that we don't want to retain logging data older than 4 years and we have just deleted all rows older than than the after the change the table looks like this
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ |
| 2022-10-09 | Update |8|John|
And we can no longer roll back the change
The options seem to be
- Have an archive routine that adds an `initialization ` of current value when we truncate
- Switch the mechanism to log the PREVIOUS value not the current value on each update (this means the live table would need to be considered when calculating any diffs)https://lab.civicrm.org/dev/core/-/issues/3819Contribution type specific custom data not shown on Pledge Payment form2022-08-30T09:25:36ZyashodhaContribution type specific custom data not shown on Pledge Payment formSteps to replicate :
- Create custom data for entity Contribution of financial type Donation.
![custom_fields](/uploads/cc517d654e574b63686b40c640306c0b/custom_fields.png)
- Check when you create a contribution and set the financial t...Steps to replicate :
- Create custom data for entity Contribution of financial type Donation.
![custom_fields](/uploads/cc517d654e574b63686b40c640306c0b/custom_fields.png)
- Check when you create a contribution and set the financial type Donation, the field is loaded on form.
![donation_only](/uploads/2cf7222d745b53268d403ab4b082fe75/donation_only.png)
- Create a pledge of financial type Donation.
- Record pledge payment and it should take to contribution with financial type Donation pre-filled
![new_pledge_payment](/uploads/c6952707b12344500e3252281f687a11/new_pledge_payment.png)
- The custom data fields (donation only) are missing though on load. On change of financial type, the custom data loads. It should work on on load as well.https://lab.civicrm.org/dev/core/-/issues/3832PHP 8.2 utf8_encode and utf8_decode functions deprecated2024-01-11T18:31:16ZBradley TaylorPHP 8.2 utf8_encode and utf8_decode functions deprecatedIn PHP 8.2 (planned to be released this November) `utf8_encode` and `utf8_decode` functions will be deprecated.
https://wiki.php.net/rfc/remove_utf8_decode_and_utf8_encode
I don't see this causing too many problems for CiviCRM, and mos...In PHP 8.2 (planned to be released this November) `utf8_encode` and `utf8_decode` functions will be deprecated.
https://wiki.php.net/rfc/remove_utf8_decode_and_utf8_encode
I don't see this causing too many problems for CiviCRM, and most uses are in upstream packages and libraries. However there are some uses in core itself (especially `utf8_decode`) which should be replaced.https://lab.civicrm.org/dev/core/-/issues/3833PHP 8.2 Dynamic Properties are deprecated2024-03-22T12:22:59ZBradley TaylorPHP 8.2 Dynamic Properties are deprecatedSee https://wiki.php.net/rfc/deprecate_dynamic_properties.
I see this as a big one, which could require a significant number of fixes for CiviCRM. The RFC has a good example showing what has changed:
```
class User {
public $name;
...See https://wiki.php.net/rfc/deprecate_dynamic_properties.
I see this as a big one, which could require a significant number of fixes for CiviCRM. The RFC has a good example showing what has changed:
```
class User {
public $name;
}
$user = new User;
// Assigns declared property User::$name.
$user->name = "foo";
// Oops, a typo:
$user->nane = "foo";
// PHP <= 8.1: Silently creates dynamic $user->nane property.
// PHP 8.2: Raises deprecation warning, still creates dynamic property.
// PHP 9.0: Throws Error exception.
```
There are many uses of dynamic properties in CiviCRM core which fall into multiple categories:
1. Known undeclared properties. For example `CRM_Activity_Form_Task_Batch` repeatedly references `$_fields` which should just be defined.
2. Dynamic properties. The main example of this is `CRM_Core_DAO`. PHP allows for this use-case by either extending `stdClass` or using the new ` #[AllowDynamicProperties]` attribute. Where possible we should prefer extending `stdClass` which is likely to have better long-term support (i.e. into the PHP 9.x series)
3. Typo's etc. Cases where the property is defined, but later referenced with a different name. These are easy; we just fix the typo (whilst ensuring we don't break any backwards compatiability for plugins using the wrong spelling)
I suggest CiviCRM developers should start to be made aware of this now, to avoid making the problem worse. Incrementally starting to fix the issue, a class or two at a time, seems like a good idea.
In many cases it's not clear why dynamic properties (or declared properties for that matter) are being used instead of a standard variable. This might be a good chance to start annotating properties as either:
- `@api`; designed to be used within hooks, on singletons etc by CiviCRM extensions
- `@internal`; not designed to be used by extensions, and liable to be removed without deprecation.https://lab.civicrm.org/dev/core/-/issues/3835Proposal: Deprecate profile standalone listing mode2023-11-30T17:03:40ZJonGoldProposal: Deprecate profile standalone listing modeProfile standalone listing code is deeply entangled with the query BAOs, and produces weird and wonderful bugs like #3834.
I don't think they're widely used in modern CiviCRM, they seem to have been a stopgap before Views integration, a...Profile standalone listing code is deeply entangled with the query BAOs, and produces weird and wonderful bugs like #3834.
I don't think they're widely used in modern CiviCRM, they seem to have been a stopgap before Views integration, and later for non-Drupal users to get a Views-esque experience. But Search Kit has effectively replaced them - they have feature parity and then some.
I think we can improve the query code if we no longer needed to worry about standalone listing mode.
Proposal:
* Remove the ability to create standalone listings on new installations of CiviCRM.
* Create a check for profiles using standalone listing mode, warning admins if they use it.
We can deprecate this over a long period of time, so the impact should be gentle.
If this proposal is approved, we may also want to consider a function that converts profile listings to SK searches. This will have added utility as we deprecate other uses of profiles (e.g. once SK replaces Advanced Search, Search Profiles will go away).https://lab.civicrm.org/dev/core/-/issues/3836Searching by phone in Advanced Search with a non-default Search Profile retur...2022-11-14T19:16:04ZbrienneSearching by phone in Advanced Search with a non-default Search Profile returns DB Error: no such fieldOverview
----------------------------------------
If a user tries to search by a phone number via Advanced Search and uses a non-default Search Profile that includes a phone number and at least one location field (i.e. city), then the se...Overview
----------------------------------------
If a user tries to search by a phone number via Advanced Search and uses a non-default Search Profile that includes a phone number and at least one location field (i.e. city), then the search produces an error with the message `DB error: no such field`.
Reproduction steps
----------------------------------------
1. Go to **Administer > Custom Data and Screens > Profiles**
1. Click **Add Profile**
1. Enter a Profile Name and check the box next/to the left of *Search Views*
1. Add fields to your Profile. At a minimum, be sure to add **Contacts > Phone > Primary > Phone** and one address field, such as **Contacts > City > Primary**. (I also included first and last name as fields)
1. Go to **Administer > Custom Data and Screens > Search Preferences**. Select the name of your test Profile from the drop down menu for the **Default Contact Search Profile** (roughly halfway down on the settings page).
1. Hover over **Search** on the navbar and click **Advanced Search**
1. Enter a phone number (or even just a number/partial number) and click **Search**
1. At this point, the user will receive an error that states `DB error: no such field`
Current behaviour
----------------------------------------
Instead of returning the results when a user searches by phone number when a non-default Search Profile includes at least one address field, the user receives an error message that the field does not exist. From the SQL displayed in the error message, this happens because the query does not include a statement to join on the address table, but references it in another join statement in the `ON` clause.
```
| Type | DB_Error |
|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Code | -19 |
| Message | DB Error: no such field |
| Mode | 16 |
| UserInfo | SELECT DISTINCT LEFT(contact_a.sort_name, 1) as sort_name FROM civicrm_contact contact_a LEFT JOIN civicrm_phone `1-phone-1` ON contact_a.id = `1-phone-1`.contact_id AND ( `1-phone-1`.phone_type_id = '1' OR `1-phone-1`.phone_type_id IS NULL ) AND ( `1-phone-1`.is_primary = 1 ) LEFT JOIN civicrm_phone ON (contact_a.id = civicrm_phone.contact_id AND civicrm_phone.is_primary = 1) LEFT JOIN civicrm_location_type `1-location_type` ON ( ( `1-address`.location_type_id = `1-location_type`.id ) ) WHERE ( civicrm_phone.phone_numeric LIKE '%301%' ) AND ( 1 ) AND (contact_a.is_deleted = 0) [nativecode=1054 ** Unknown column '1-address.location_type_id' in 'on clause'] |
| DebugInfo | SELECT DISTINCT LEFT(contact_a.sort_name, 1) as sort_name FROM civicrm_contact contact_a LEFT JOIN civicrm_phone `1-phone-1` ON contact_a.id = `1-phone-1`.contact_id AND ( `1-phone-1`.phone_type_id = '1' OR `1-phone-1`.phone_type_id IS NULL ) AND ( `1-phone-1`.is_primary = 1 ) LEFT JOIN civicrm_phone ON (contact_a.id = civicrm_phone.contact_id AND civicrm_phone.is_primary = 1) LEFT JOIN civicrm_location_type `1-location_type` ON ( ( `1-address`.location_type_id = `1-location_type`.id ) ) WHERE ( civicrm_phone.phone_numeric LIKE '%301%' ) AND ( 1 ) AND (contact_a.is_deleted = 0) [nativecode=1054 ** Unknown column '1-address.location_type_id' in 'on clause'] |
```
Expected behaviour
----------------------------------------
When a user searches by phone number via Advanced Search they should not receive an error and instead be directed to the results page of the search.
Environment information
----------------------------------------
* __CiviCRM:__ _Master/5.54.alpha1
Comments
----------------------------------------
[PR #24450](https://github.com/civicrm/civicrm-core/pull/24450) proposes a fix for this issuehttps://lab.civicrm.org/dev/core/-/issues/3816Saving event intermittently throws 500 error2023-02-08T21:37:50ZBobSSaving event intermittently throws 500 errorOverview
----------------------------------------
Saving an event intermittently results in a hang with:
* Greyed out tab contents
* Spinning cursor
* Error in the browser console: "VM6153:1 POST https://dmaster.demo.civicrm.org/civicrm/...Overview
----------------------------------------
Saving an event intermittently results in a hang with:
* Greyed out tab contents
* Spinning cursor
* Error in the browser console: "VM6153:1 POST https://dmaster.demo.civicrm.org/civicrm/ajax/inline?class_name=CRM_UF_Form_Inline_PreviewById&id=14&snippet=json&crmAngularModules=crmApp,crmProfileUtils,crmResource,crmUi,crmUtil,ngRoute,ngSanitize,volunteer 500 (Internal Server Error)"
Upon a page reload following a hang, the following popup is displayed:
```We can't load the requested web page. This page requires cookies to be enabled in your browser settings. Please check this setting and enable cookies (if they are not enabled). Then try again. If this error persists, contact the site administrator for assistance.<br /><br />Site Administrators: This error may indicate that users are accessing this page using a domain or URL other than the configured Base URL. EXAMPLE: Base URL is http://example.org, but some users are accessing the page via http://www.example.org or a domain alias like http://myotherexample.org.<br /><br />Error type: Could not find a valid session key.```
Problem confirmed on https://dmaster.demo.civicrm.org.
Reproduction steps
----------------------------------------
1. Go to https://dmaster.demo.civicrm.org
1. Click **Events** | **Manage Events**.
1. **Rain-forest Cup Youth Soccer Tournament**: Click **Configure** | **Online Registration**.
1. Click **Save**
Current behaviour
----------------------------------------
About 50% of the time it will hang as described above.
Oddly, the failures do not seem to be randomly distributed. It seems that it will tend to fail consistently for several minutes, and then to save successfully for several minutes. Or maybe I'm doing something subtly different that I have not noticed which makes the problem appear.
Problem has been observed upon saving various tabs (Info and Settings, Event Location, Online Registration)
Problem has not been observed upon clicking __Save and Done__.
__Stress Testing__
After a page load and successful save, it appears that every subsequent save will be successful. To stress test, therefore, the following sequence must be repeated:
1. Refresh page
1. Save
__GET/POST Requests__
When saving the Online Registration tab, the following sequence is observed in the browser's Network tab for a successful save:
1. POST https://dmaster.demo.civicrm.org/civicrm/event/manage/registration?action=update&id=3&component=event&qfKey=CRMEventFormManageEventRegistrationlkkd3ym1nsgogsggw8scokgo8kw04w0cw00wsg8csgwkc4gcw_4298&snippet=json&crmAngularModules=crmApp,crmProfileUtils,crmResource,crmUi,crmUtil,ngRoute,ngSanitize,volunteer
1. GET https://dmaster.demo.civicrm.org/civicrm/event/manage/registration?reset=1&action=update&id=3&component=event&qfKey=CRMEventFormManageEventRegistrationlkkd3ym1nsgogsggw8scokgo8kw04w0cw00wsg8csgwkc4gcw_4298&snippet=json&crmAngularModules=crmApp,crmProfileUtils,crmResource,crmUi,crmUtil,ngRoute,ngSanitize,volunteer
1. GET https://dmaster.demo.civicrm.org/civicrm/ajax/inline?class_name=CRM_UF_Form_Inline_PreviewById&id=12&snippet=json&crmAngularModules=crmApp,crmProfileUtils,crmResource,crmUi,crmUtil,ngRoute,ngSanitize,volunteer
When the save fails, however, only the third request is observed, and it throws a 500 error.
__Civi Log__
```
Aug 20 14:07:08 [error]
$Fatal Error Details = array:3 [
"message" => "We can't load the requested web page. This page requires cookies to be enabled in your browser settings. Please check this setting and enable cookies (if they are not enabled). Then try again. If this error persists, contact the site administrator for assistance.<br /><br />Site Administrators: This error may indicate that users are accessing this page using a domain or URL other than the configured Base URL. EXAMPLE: Base URL is http://example.org, but some users are accessing the page via http://www.example.org or a domain alias like http://myotherexample.org.<br /><br />Error type: Could not find a valid session key."
"code" => null
"exception" => CRM_Core_Exception {#1777
-errorData: array:1 [
"error_code" => 0
]
#cause: null
-_trace: null
#message: "We can't load the requested web page. This page requires cookies to be enabled in your browser settings. Please check this setting and enable cookies (if they are not enabled). Then try again. If this error persists, contact the site administrator for assistance.<br /><br />Site Administrators: This error may indicate that users are accessing this page using a domain or URL other than the configured Base URL. EXAMPLE: Base URL is http://example.org, but some users are accessing the page via http://www.example.org or a domain alias like http://myotherexample.org.<br /><br />Error type: Could not find a valid session key."
#code: 0
#file: "/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php"
#line: 861
trace: {
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php:861 {
› $msg = ts("We can't load the requested web page. This page requires cookies to be enabled in your browser settings. Please check this setting and enable cookies (if they are not enabled). Then try again. If this error persists, contact the site administrator for assistance.") . '<br /><br />' . ts('Site Administrators: This error may indicate that users are accessing this page using a domain or URL other than the configured Base URL. EXAMPLE: Base URL is http://example.org, but some users are accessing the page via http://www.example.org or a domain alias like http://myotherexample.org.') . '<br /><br />' . ts('Error type: Could not find a valid session key.');
› throw new CRM_Core_Exception($msg);
› }
}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php:856 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php:316 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php:193 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller/Simple.php:49 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Utils/Wrapper.php:62 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Page/AJAX.php:63 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php:285 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php:69 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php:36 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/drupal/civicrm.module:471 { …}
/var/www/clients/client1/web3/web/drupal/includes/menu.inc:527 { …}
/var/www/clients/client1/web3/web/drupal/index.php:197 { …}
}
}
]
Aug 20 14:07:08 [debug] $backTrace = #0 /var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Error.php(441): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 /var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php(39): CRM_Core_Error::handleUnhandledException(Object(CRM_Core_Exception))
#2 /var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/drupal/civicrm.module(471): CRM_Core_Invoke::invoke((Array:3))
#3 /var/www/clients/client1/web3/web/drupal/includes/menu.inc(527): civicrm_invoke("ajax", "inline")
#4 /var/www/clients/client1/web3/web/drupal/index.php(197): menu_execute_active_handler()
#5 {main}
```
__Possibly Related__
Upon loading any of the event tabs, the following error is always displayed in the console:
```
angular.js:15697 TypeError: Cannot read properties of undefined (reading 'id')
at angular-modules.c3b61f5213a83292c272ff9114e7a62a.js?rgx685:2098:66
at m.$digest (angular.js:19270:23)
at angular.js:19562:15
at Yg.completeTask (angular.js:21403:7)
at angular.js:6879:7
```
Expected behaviour
----------------------------------------
The Save should complete without error.
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ _Chrome 104.0.5112.101_
* __CiviCRM:__ _5.52_, _5.54.alpha1_
* __PHP:__ _7.4_
* __CMS:__ _Drupal 7.91_
* __Database:__ _MariaDB 10.4_
* __Web Server:__ _Apache 2.4_
Comments
----------------------------------------
_Anything else you would like the reviewer to note._seamusleeseamusleehttps://lab.civicrm.org/dev/core/-/issues/3930Managed entity: creating custom fields with same label causes fatal error bec...2022-10-24T09:18:16ZherbdoolManaged entity: creating custom fields with same label causes fatal error because of db index.Overview
----------------------------------------
It seems a bit over the top to have fatal errors when custom field labels are not unique. This would (likely) only appear if someone had a managed entity for the custom field. Or if they...Overview
----------------------------------------
It seems a bit over the top to have fatal errors when custom field labels are not unique. This would (likely) only appear if someone had a managed entity for the custom field. Or if they were using the API directly.
Example use-case
----------------------------------------
1. Create a custom group. Add a custom field.
2. Go to API4, choose CustomField, create. Create a new custom field with the same label.
Current behaviour
----------------------------------------
> DB Error: already exists
Proposed behaviour
----------------------------------------
Remove the index from the database for the label. The custom field index on the name should be enough. The UI will still have validation to prevent duplicate labels, but that's necessary because there's no way to specify the label and name separately in the UI.https://lab.civicrm.org/dev/core/-/issues/4018Extension upgrades are not run if there are no core upgrades2022-12-02T12:24:19ZherbdoolExtension upgrades are not run if there are no core upgrades@totten recently did: https://github.com/civicrm/civicrm-core/pull/24030 to include extension upgrades in the core upgrades.
I've noticed a couple times, however, that there seem to be extension upgrades still waiting after upgrading Ci...@totten recently did: https://github.com/civicrm/civicrm-core/pull/24030 to include extension upgrades in the core upgrades.
I've noticed a couple times, however, that there seem to be extension upgrades still waiting after upgrading CiviCRM code base and running `cv upgrade:db`. Could it be that if there are no core upgrade hooks waiting that it won't run the extension upgrades? It does seem likely given my chat with @colemanw https://chat.civicrm.org/civicrm/pl/cfd18cjpofg5iesh7o4zqejcbh.https://lab.civicrm.org/dev/core/-/issues/4234Change Dedupe Rule Usage Dialog Is Not Displayed2023-05-15T08:23:09ZLKuttnerChange Dedupe Rule Usage Dialog Is Not DisplayedReproduction steps
1. Go to Find and Merge Duplicate Contacts
2. Click _Edit Rule_ for a General Unreserved rule.
3. In Usage: Currently set to: click the _Change Usage_ button which is NOT disabled.
Current behaviour
The Change Usage ...Reproduction steps
1. Go to Find and Merge Duplicate Contacts
2. Click _Edit Rule_ for a General Unreserved rule.
3. In Usage: Currently set to: click the _Change Usage_ button which is NOT disabled.
Current behaviour
The Change Usage dialog does not appear and nothing at all happens.
Expected behaviour
The Change Usage dialog appears and the rule can be set as Supervised or Unsupervised.
Environment information
- CiviCRM 5.50.4
- Drupal 9.3.13
- PHP 7.4.19https://lab.civicrm.org/dev/core/-/issues/4253cURL error when trying to update action-provider, data processor or CiviRules...2024-01-21T12:44:06ZMariaVcURL error when trying to update action-provider, data processor or CiviRules via UIOverview
----------------------------------------
When trying to update action provider or data processor via UI the following error occurs:
`(cURL error 28: Operation timed out after 5000 milliseconds with 142514 out of 408357 bytes rec...Overview
----------------------------------------
When trying to update action provider or data processor via UI the following error occurs:
`(cURL error 28: Operation timed out after 5000 milliseconds with 142514 out of 408357 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)`
Reproduction steps
----------------------------------------
1. Click on _Extensions_
2. Upgrade action-provider or data processor or CiviRules
3. Error (see above)
Current behaviour
----------------------------------------
Using wget via terminal the download for action provider takes 15 seconds.
Data processor download takes 1m 52s via wget.
It seems that lab.civicrm.org is quite slow.
Expected behaviour
----------------------------------------
Successful Update via UI
Possible solution: time out increasehttps://lab.civicrm.org/dev/core/-/issues/4265Checksum sent to Organization will not fill in "on behalf of" profile fields2023-11-23T07:47:01ZStoobChecksum sent to Organization will not fill in "on behalf of" profile fieldsScope the different [checksum token](https://docs.civicrm.org/user/en/latest/common-workflows/tokens-and-mail-merge/) behavior 'on behalf of' Org contribution page for
- an Individual related to an Organization https://civiteacher.com/c...Scope the different [checksum token](https://docs.civicrm.org/user/en/latest/common-workflows/tokens-and-mail-merge/) behavior 'on behalf of' Org contribution page for
- an Individual related to an Organization https://civiteacher.com/civicrm/contribute/transact?reset=1&id=9&cs=e8ecaad83d04439f381e4f2bf35d9322_1682576643_504&cid=102&mid=
- that Organization itself https://civiteacher.com/civicrm/contribute/transact?reset=1&id=9&cs=45ce7efc7d44ed07e8fd4097357ddc74_1682576099_504&cid=112&mid= with no contact info in org profile yet placing the org email in the individual email field
Should it not be the case that a checksum sent to an organization should fill out the 'on behalf of' section and leave the individual contact info blank to be filled out later? There are two good use cases where this may occur:
1. emails sent to the primary member organization _and_ the related contact individual(s) for maximum chance of renewal and/or in case contact person has quit
2. the organization primary member does not have a related contact indvidual at all. maybe it was entered incomplete or org imported without contact individual
But I cannot tell if it is a regression or if it has always been this way. @AllenShaw thoughts?