CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-11-04T05:03:19Zhttps://lab.civicrm.org/dev/core/-/issues/3009authx: Proposal: Make "require" the default user policy instead of "optional"2023-11-04T05:03:19ZDaveDauthx: Proposal: Make "require" the default user policy instead of "optional"If you look at https://docs.civicrm.org/dev/en/latest/framework/authx/#settings the default setting for the `XXX_user` settings is "optional" which means a corresponding cms user doesn't need to exist in order to log in. This seems unint...If you look at https://docs.civicrm.org/dev/en/latest/framework/authx/#settings the default setting for the `XXX_user` settings is "optional" which means a corresponding cms user doesn't need to exist in order to log in. This seems unintuitive as a default. My expectation is that it would work the same way as regular logins by default, so should default to "require".https://lab.civicrm.org/dev/core/-/issues/3003Preserve previous tab when navigating to and from contact page2022-05-05T14:57:50ZBradley TaylorPreserve previous tab when navigating to and from contact pageThe contact page is structured around a series of tabs: Summary, Contributions, Membership etc. There are often times when you click through from a contact to a related entity (say a related contact from the relationships tab). It would ...The contact page is structured around a series of tabs: Summary, Contributions, Membership etc. There are often times when you click through from a contact to a related entity (say a related contact from the relationships tab). It would be nice if using the browser's back button returned you to the tab you were previously looking at rather than returning to the summary tab each time.
This is a bigger issue for users with the "Enable Popup Forms" option unticked, but all users are affected by this.
This shouldn't be a massive technical change, but I do think this is one of those little annoyances, where fixing should improve the overall usability of the CRM for those who need to use it on a daily basis.
**Implementation details**
It is already possible to link to a specific tab with the `selectedChild` query parameter. I propose that whenever the active tab is changed, the [replaceState](https://developer.mozilla.org/en-US/docs/Web/API/History/replaceState) API is used to amend the `selectedChild` parameter in the URL.5.49.0https://lab.civicrm.org/dev/core/-/issues/2994Support oEmbed for external facing pages2024-03-21T18:50:44ZJoeMurraySupport oEmbed for external facing pagesAn important evolution of CiviCRM is to better support remote sites exposing CiviCRM forms and content.
Many platforms including WordPress and Drupal core since 8.6 support oEmbed, which allows a site to pull in content from a different...An important evolution of CiviCRM is to better support remote sites exposing CiviCRM forms and content.
Many platforms including WordPress and Drupal core since 8.6 support oEmbed, which allows a site to pull in content from a different site that exposes its content as oEmbed.
There is a good list of oEmbed providers, and CiviCRM should aspire to join it: https://oembed.com/providers.json
We should expose the following 'pages' as oEmbed content, using appropriate standard markup for use in WordPress and Drupal 8.6+ sites, and support the workflow for confirmation, thank you, and search results as appropriate:
1. Phase 1
1. Contribution page
2. Event info page
3. Event registration page
2. Phase 2
1. Something like a profile create/edit page that does not involve a payment but allows submission of a form to create a contact and activity and perhaps more (eg parent-child relationship)
2. Search Kit page
3. Petition page
3. Phase 3
1. Personal Campaign Page
2. Pages from extensions, eg Grant Application Page
3. CiviSurvey pages
Note: I haven't done the research to understand the details of oEmbed and potential challenges. I think this could be done through extension(s).
I'm posting this in order to generate discussion for a possible Roadmap item.https://lab.civicrm.org/dev/core/-/issues/2993Add option to delete old log files during rotation2023-11-02T05:03:25ZJoeMurrayAdd option to delete old log files during rotationOverview
----------------------------------------
_When creating a new log file, optionally remove old ones like logrotate allows (https://linux.die.net/man/8/logrotate)._
Example use-case
----------------------------------------
CiviCR...Overview
----------------------------------------
_When creating a new log file, optionally remove old ones like logrotate allows (https://linux.die.net/man/8/logrotate)._
Example use-case
----------------------------------------
CiviCRM's simplistic log creation does not actually rotate, it just makes more files for the logs and never deletes any (https://github.com/civicrm/civicrm-core/blob/5433c81f5648bf484ef16debeca3c7e6e98d1a11/CRM/Core/Error.php#L684). This can be dangerous for systems where there is little to no active monitoring of logs, eg small orgs without a provider, or on a shoestring budget. It is also dangerous in cases where a change starts to generate high volumes of error messages, eg a client recently misconfigured a civirule and GBs of log were produced every hour. Running out of disk space causes hard crashes of sites.
It is standard practice to have a process in place to ensure that logs do not end up taking too much disk space. We propose allowing a setting to be added to civicrm.settings.php (eg NUM_LOG_FILES_TO_RETAIN) that would allow the number of logs to be retained to be specified. The default setting would be 0, meaning an infinite number, which would retain current behaviour. We would also be happy to make it another number such as 6, which would be 6 months or more than 1.5GB.
My understanding is that settings that can be added to civicrm.settings.php should be included as comments with explanation that can be uncommented, so we will do this.
Current behaviour
----------------------------------------
_CiviCRM generates log files with random hash names, and creates new ones every month or when the file exceeds 256MB. Log file space grows continuously unless log files are manually deleted._
Proposed behaviour
----------------------------------------
_When a new log file is created, if a maximum number of log files has been specified, then delete those over that number, starting with oldest first_
Comments
----------------------------------------
_To reduce the complexity of the settings pages, we propose that this be implemented as a setting that could be optionally specified in civicrm.settings.php._Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/2991Proposal: rework name fields to make name entity2023-12-13T09:20:59ZAndie HuntProposal: rework name fields to make name entity## Overview
A recent [discussion with Guy](https://chat.civicrm.org/civicrm/pl/wwqowz635fbpifio7on4e8r8dw) and #2883, plus the ongoing mayhem of organization names and personal experience with a variety of great and terrible name handli...## Overview
A recent [discussion with Guy](https://chat.civicrm.org/civicrm/pl/wwqowz635fbpifio7on4e8r8dw) and #2883, plus the ongoing mayhem of organization names and personal experience with a variety of great and terrible name handling systems, have led me to think that the solution to names isn't creating more or better-labeled fields for more types of names but rather to make name its own entity akin to phone, email, address, website, etc.
Once name is an entity, each contact could have multiple names, one primary, but with a bunch of possible types:
- legal name
- trade name
- nickname
- former name
- name in other language/culture
- name in an external system
- common mistakes
There could also be a bunch of name renderings - ways the name is displayed or inserted:
- sort name
- display name
- name for greetings
- name for listings
The latter could generate the existing greetings and potentially others. In particular, I think there's a real gap for how you insert someone's name in emails where you don't want "Dear", "Hi", or any other greeting as such.
Finally, the duplicate matching process could look for any of the possible names, cutting down on duplicates.
## Example use-case
### Organization has multiple names
An organization has their corporate name, colloquial name, and the name of their main product. These all refer to the same contact, but employees might fill the current employer field as one of the various ways. As an admin, you want to avoid having to merge them all the time. As a routine user, you will want to know all the various ways an organization is termed. As a bookkeeper, you will want to send invoices, checks, receipts, etc. with the correct entity name.
In this case, there are three name entities:
| Organization name | Name type | Is primary |
| ------------------------ | --------- | ---------- |
| Widget Productions | General | true |
| Widget Productions, Inc. | Legal | false |
| WidgetPro | General | false |
| Widgetizer | Product | false |
There would be a couple of name renderings:
| Rendering type | Value |
| -------------- | ------------------------ |
| Display Name | Widget Productions |
| Sort Name | Widget Productions |
| Billing name | Widget Productions, Inc. |
### Minister has variety of prefixes
Someone's name is styled differently when it is listed versus when they are addressed. In this case, a minister is styled "The Reverend Sally Jones" or "Rev. Sally Jones" when listed somewhere but her salutation is "Ms. Jones".
I think she'd have only one entry as a name entity but several renderings:
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------------|------------|-------------|-----------|--------|-----------|------------|
| Ms. | The Reverend | Sally | Lynn | Jones | *null* | General | true |
These values are going to depend upon your sitewide preferences and choices for this contact, but a reasonable default might produce the following:
| Rendering type | Value |
|-----------------|--------------------------|
| Display Name | Sally Lynn Jones |
| Sort Name | Jones, Sally Lynn |
| Addressee | The Reverend Sally Jones |
| Email greeting | Dear Sally |
| Postal greeting | Dear Ms. Jones |
| Short name | Sally |
### An organization has a different name in another language
This should be self-explanatory:
| Organization name | Name type | Is primary |
|--------------------------|-----------|------------|
| Médecins Sans Frontières | General | true |
| Doctors Without Borders | Localized | false |
| Rendering type | Value |
|----------------|--------------------------|
| Display Name | Médecins Sans Frontières |
| Sort Name | Médecins Sans Frontières |
| English Name | Doctors Without Borders |
### Someone's family name comes before their given name
I'm on the fence about how this should be handled, but all I know is that we currently handle it terribly (except maybe in translations that assume it?) One option might be to call the fields first, middle, and last names according to position (*Ai* being the family name, but it comes first):
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------|------------|-------------|-----------|--------|-----------|------------|
| Mr. | *null* | Ai | *null* | Weiwei | *null* | General | true |
Alternatively, we could name the fields according to function:
| Prefix | Title | Given name | Additional name | Surname | Suffix | Name type | Is primary |
|--------|--------|------------|-----------------|---------|--------|-----------|------------|
| Mr. | *null* | Weiwei | *null* | Ai | *null* | General | true |
In either case, the renderings should be like
| Rendering type | Value |
|-----------------|---------------|
| Display Name | Ai Weiwei |
| Sort Name | Ai Weiwei |
| Addressee | Mr. Ai Weiwei |
| Email greeting | Dear Weiwei |
| Postal greeting | Dear Mr. Ai |
| Short name | Weiwei |
### Clinic needs to retain legal name for insurance billing
Someone's real name doesn't match their legal name, let alone their nickname. While most organizations wouldn't want or need to record a contact's dead name, a clinic might need it for billing correspondence. In this case, a patient is named Stephen, or Steve for short, but his legal name is Barbara.
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------|------------|-------------|-----------|--------|-----------|------------|
| Mr. | *null* | Stephen | Wayne | Smith | *null* | General | true |
| Mr. | *null* | Steve | *null* | Smith | *null* | Nickname | false |
| *null* | *null* | Barbara | Wayne | Smith | *null* | Legal | false |
All of the name renderings would use the primary name or nickname, but the clinic could add an additional name rendering for how they refer to patients when billing to the insurance company.
| Rendering type | Value |
|-----------------|----------------------|
| Display Name | Stephen Wayne Smith |
| Sort Name | Smith, Stephen Wayne |
| Addressee | Mr. Stephen Smith |
| Email greeting | Dear Steve |
| Postal greeting | Dear Mr. Smith |
| Short name | Steve |
| Insurance name | Barbara Smith |
### Household name possibilities
The current household name field befuddles lots of people and, in my opinion, makes households less attractive to use. If a household can have multiple names, it might be more useful.
You might use a Household name field that has everyone all listed out, or you could use the Last name field.
| Household name | Last name | Name type | Is primary |
|-------------------------------------|----------------|-----------|------------|
| Katherine Williams and Kendra Green | *null* | General | true |
| *null* | Williams-Green | General | false |
| Rendering type | Value |
|-----------------|------------------------------------------|
| Display Name | Katherine Williams and Kendra Green |
| Sort Name | Katherine Williams and Kendra Green |
| Addressee | The Williams-Green Household |
| Email greeting | Dear Katherine Williams and Kendra Green |
| Postal greeting | Dear Katherine Williams and Kendra Green |
Or if everyone shares a last name, a single name entity could do it all:
| Given name | Surname | Name type | Is primary |
|---------------|---------|-----------|------------|
| Jack and Jill | Hill | General | true |
(If we don't feel the need to stick with the legacy name field names, there's really no reason that "Given name" couldn't replace Organization name and Household name, too, so this illustrates that.)
| Rendering type | Value |
|-----------------|-------------------------|
| Display Name | Jack and Jill Hill |
| Sort Name | Hill, Jack and Jill |
| Addressee | The Hill Household |
| Email greeting | Dear Jack and Jill |
| Postal greeting | Dear Jack and Jill Hill |
## Current behavior
Right now, each of these situations is pretty annoying. It might require a separate custom field, located away from the name fields. It might make it difficult to automatically and accurately generate greetings for many people.
Currently, there are fields in the `civicrm_contact` table for each name part, plus nickname and legal name.
The only name renderings are display name and sort name (set sitewide), and addressee and email and postal greetings (site-wide presets and defaults, with custom options).
There's no provision for additional renderings or any way to do fallbacks like pick a nickname and fall back to a first name.
There's a limited number of alternative names for a contact.
Alternative names are not compared against primary names when deduping.
## Proposed behavior
### Schema
The name fields would be gone from the contact entity, and a new `civicrm_contact_name` entity is added with one of the following sets of fields:
| Traditional | Radical |
|-------------------|-------------------|
| id | id |
| contact_id | contact_id |
| contact_name_type | contact_name_type |
| is_primary | is_primary |
| first_name | given_name |
| middle_name | additional_name |
| last_name | surname |
| prefix_id | prefix_id |
| suffix_id | suffix_id |
| formal_title | formal_title |
| household_name | |
| organization_name | |
The `contact_name_type` would have reserved options General, Nickname, Legal, and Localized, but more could be added.
A new `civicrm_contact_rendering` entity is added with the following set of fields:
| Fields |
|-------------------------------|
| id |
| contact_id |
| contact_rendering_type |
| contact_rendering_template_id |
| value |
The `contact_rendering_type` would have reserved options Display name, Sort name, Addressee, Email greeting, Postal greeting, and Short name, but more could be added.
The `contact_rendering_template_id` would be akin to the existing `addressee_id`, `email_greeting_id`, and `postal_greeting_id`, while `value` would be the actual generated or custom value.
### Upgrade
On upgrade, the `civicrm_contact_name` would be populated with the main name fields, with nickname and legal name as additional rows. On upgrade, the `civicrm_contact_rendering` would be populated with rows for each contact's display name, sort name, addressee, and greetings.
### Contact rendering templates
The contact rendering templates would have the option to specify a contact name type to provide the default value, with a fallback to the primary contact name. For example, email greeting might default to "Dear {nickname.first_name}", where it picks the value of `first_name` from a contact's name entry with the name type of Nickname, and if that isn't available, it picks the primary name entry's value for `first_name`.
### Dedupe
When matching contacts on any of the name fields, the dedupe process would compare against all rows in the `civicrm_contact_name` field, regardless of `contact_name_type` or if it's the primary name.
## Comments
This is all very early in the process: I think it's important to do something like this, but I don't have my heart on the specific mechanisms here. I think it would need to be evaluated for performance in a variety of ways, but I don't expect it would have too huge of an impact.https://lab.civicrm.org/dev/core/-/issues/2979Proposal to remove the limit of 15 max values for multiple values can also be...2021-12-09T16:28:04ZyashodhaProposal to remove the limit of 15 max values for multiple values can also be retrieved from url in reportsWe send the parameters as string so that multiple values can also be retrieved from url in reports
for e.g url like - "memtype_in=in&memtype_value=1,2,3
However, there is a restriction for 15 values esp when you search for activity type...We send the parameters as string so that multiple values can also be retrieved from url in reports
for e.g url like - "memtype_in=in&memtype_value=1,2,3
However, there is a restriction for 15 values esp when you search for activity types which can be numerous. In this this case, if the values exceeds 15 the criteria are not respected.
Proposal to remove the limit of 15 max values.
https://github.com/civicrm/civicrm-core/blob/master/CRM/Report/Utils/Get.php#L1735.46.0yashodhayashodhahttps://lab.civicrm.org/dev/core/-/issues/2976Feature Request: Form Builder and Permissions - Provide ability to restrict a...2023-11-02T12:39:11Zjustinfreeman (Agileware)Feature Request: Form Builder and Permissions - Provide ability to restrict access to selected Roles rather than just specific Permission GrantsWould be great if the Form Builder, Permissions had the ability to restrict access to selected Roles rather than just specific Permission Grants.
Roles being far more flexible, can be automatically granted and removed for user accounts ...Would be great if the Form Builder, Permissions had the ability to restrict access to selected Roles rather than just specific Permission Grants.
Roles being far more flexible, can be automatically granted and removed for user accounts and this is a common pattern on Member websites. Whereas Permission Grants are not used in this way.
For example: When a user with the Member role views a Form Builder page, they are granted access to use the Form and view the Search Results because they have the Member role.
![Screenshot_20211202_130808](/uploads/25c2fbadece5eea59beb05d7b269e937/Screenshot_20211202_130808.png)
Agileware Ref: CIVICRM-1899https://lab.civicrm.org/dev/core/-/issues/2965APIv4 Explorer, displays the Field Name as the selectable option. However, th...2023-10-23T05:03:22Zjustinfreeman (Agileware)APIv4 Explorer, displays the Field Name as the selectable option. However, the Field Label only is visible in the Custom Fields UI and may be different to the Field NameAPIv4 Explorer, displays the Field Name as the selectable option. However, the Field Label only is visible in the Custom Fields UI and may be different to the Field Name.
SearchKit on the other hand does use the Field Label to list the ...APIv4 Explorer, displays the Field Name as the selectable option. However, the Field Label only is visible in the Custom Fields UI and may be different to the Field Name.
SearchKit on the other hand does use the Field Label to list the available fields.
This can make it very difficult to use the APIv4 Explorer to select the correct fields if:
1. There are a large number of fields, and
2. The Field Name differs from the Field Label
This is a very common occurrence on older CiviCRM sites which tend to have fields that have been renamed over time and have a lot of old fields still active on the site.
My preference would be to always list the Field Label when displaying fields in the APIv4 Explorer.
![Screenshot_20211123_084139](/uploads/b71225e09f100a8e78d2c321a05704d1/Screenshot_20211123_084139.png)
Agileware Ref: CIVICRM-1890https://lab.civicrm.org/dev/core/-/issues/2940Remove requirement to have a bounce email account setup2022-09-30T20:45:40ZseamusleeRemove requirement to have a bounce email account setupWith some more modern SMTP providers bounce email accounts may not be necessary for CiviMail Bounces to work because e.g. with Amazon SES / Sendgrid you can get the bounces processed via webhooks instead.
However within the code base we...With some more modern SMTP providers bounce email accounts may not be necessary for CiviMail Bounces to work because e.g. with Amazon SES / Sendgrid you can get the bounces processed via webhooks instead.
However within the code base we assume we need a bounce account to generate stuff like [no-reply email address](https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/BAO/Domain.php#L364) or when we are getting [message ids](https://github.com/civicrm/civicrm-core/blob/513f6a469927c5dc5a41905feb7c3cff2cf15f0b/CRM/Campaign/BAO/Petition.php#L548)
It would be nice to get rid of this reliance on it being the default mail account which is the bounce processing one https://github.com/civicrm/civicrm-core/blob/513f6a469927c5dc5a41905feb7c3cff2cf15f0b/CRM/Core/BAO/MailSettings.php#L59
cc @JoeMurrayMonish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/2938Proposal - add created_date, modified_date to civicrm_relationship2022-11-25T02:00:54ZeileenProposal - add created_date, modified_date to civicrm_relationshipIt can be useful to know when a relationship was created in CiviCRM (as opposed to when it started)It can be useful to know when a relationship was created in CiviCRM (as opposed to when it started)https://lab.civicrm.org/dev/core/-/issues/2934Searchkit feedback2023-10-10T05:03:21ZeileenSearchkit feedbackRequest for COUNTIF() in the transforms - ie count if field value > xRequest for COUNTIF() in the transforms - ie count if field value > xhttps://lab.civicrm.org/dev/core/-/issues/2931Deadlocks on acl_contact_cache table2022-07-16T20:22:41ZandyburnsDeadlocks on acl_contact_cache tableWe are seeing the acl_contact_cache table locking up our entire system. The query has been seen running for over 6000 seconds on multiple occasions. Slowly, queries stack up behind this and other requests by other users to use acl_contac...We are seeing the acl_contact_cache table locking up our entire system. The query has been seen running for over 6000 seconds on multiple occasions. Slowly, queries stack up behind this and other requests by other users to use acl_contact_cache stack up as well.
We had not previously set the max_execution_time on the database and that has now been set to 450 on the database, which prevents the long-running query from tangling up the entire system. However, the conflict appears to happen when two users are doing an operation that tries to populate the acl_contact_cache table at the same time.
Our database has almost 900,000 contacts, and almost 500,000 activities logged. So a user goes to search > find activities and leaves conditions blank, and this query occurs:
```
SELECT count(DISTINCT contact_a.id) as rowCount FROM civicrm_contact contact_a LEFT JOIN civicrm_activity_contact
ON ( civicrm_activity_contact.contact_id = contact_a.id ) LEFT JOIN civicrm_activity
ON ( civicrm_activity.id = civicrm_activity_contact.activity_id
AND civicrm_activity.is_deleted = 0 AND civicrm_activity.is_current_revision = 1 ) INNER JOIN civicrm_contact
ON ( civicrm_activity_contact.contact_id = civicrm_contact.id and civicrm_contact.is_deleted != 1 ) INNER JOIN civicrm_acl_contact_cache aclContactCache ON contact_a.id = aclContactCache.contact_id WHERE ( ( civicrm_activity.activity_type_id = 101 AND civicrm_activity.status_id = 9 ) OR ( civicrm_activity.activity_type_id IN (2,3,1,104,109,78,76,85,93,4,5,6,7,8,12,17,19,22,34,40,35,36,37,38,39,44,45,46,47,48,51,58,54,52,80,81,84,94,96,97,99,100,101,103,105,106) ) ) AND aclContactCache.user_id = 26114 AND contact_a.is_deleted = 0 AND (civicrm_activity.activity_type_id IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 17, 19, 22, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 54, 58, 76, 77, 78, 80, 81, 82, 84, 85, 86, 87, 93, 94, 96, 97, 99, 100, 101, 102, 103, 104, 105, 106, 107, 109) AND civicrm_activity.`id` IN (SELECT `id` FROM `civicrm_activity` WHERE id IN (SELECT activity_id FROM civicrm_activity_contact WHERE contact_id IN (SELECT contact_id FROM civicrm_acl_contact_cache WHERE user_id = 26114) AND contact_id IN (SELECT `id` FROM `civicrm_contact` WHERE is_deleted != 1))));
```
6000 seconds later, the database is hung, with about 100 queries stuck behind it. Manually killing all processes that are trying to use the acl_contact_cache table will clear out the backlog of other queries.
The table is transitory, as after getting the results there is a truncate almost immediately after.
One proposed solution could be to individualize the acl_contact_cache table to create for each user, eg. acl_contact_cache_{contact_id} … This would allow the acl_contact_cache_XXX table to get locked and not adversely affect every other user on the system.
The system should do a check upon my first search and see if my acl_contact_cache_XXX table is built, if it is not, build it upon all the contacts I have access to (regardless of the particular search).
Additionally the acl_contact_cache_XXX table could persist for a pre-defined period of time (e.g. 1 hour) so it doesn't have to search everyone in the db each time. Same concept for civicrm_group_contact_cache. Could have major performance benefits.
Also finding a way to keep this table from being locked would likely require a reimagination of how the table is actually used. Another consideration would be the use of the MEMORY engine for this table, to permit faster population since its contents are transitory anyway
Thoughts?
@eileen @kcristiano
Similar issue on acl_cache table: https://lab.civicrm.org/dev/core/-/issues/1486.https://lab.civicrm.org/dev/core/-/issues/2929Geocoding failures kill contributions2023-10-23T16:08:42ZandrewcormickdockeryGeocoding failures kill contributionsOverview
----------------------------------------
Last night between about 6pm and midnight AEDT, it appears that Nominatim (the Open Street Map API provider) had an intermittent outage. Some requests were receiving a 502 error. We hav...Overview
----------------------------------------
Last night between about 6pm and midnight AEDT, it appears that Nominatim (the Open Street Map API provider) had an intermittent outage. Some requests were receiving a 502 error. We have configured this as our Geocoding provider in CiviCRM under /civicrm/admin/setting/mapping?reset=1
People who were using CiviCRM in ways which caused the Geomapping provider to be called were receiving fatal errors as a result. This included people who were making donations (i.e. entering their address on the form which in turn was handed to the Geomapping provider for geocoding).
Current behaviour
----------------------------------------
When a request to the Geomapping provider fails (for example with a 502 error), the generating transaction within CiviCRM returns a fatal error and does not complete. This includes financial transactions, and indeed any administrative function that involves adding or updating a contact's address.
Expected behaviour
----------------------------------------
A Geomapping provider failure should never cause financial transactions to fail. I can't think of any reason why adding or updating an address should fail either. Fatal errors should only occur when an issue occurs which genuinely prevent a transaction from succeeding. A payment can still be processed regardless of whether that person's address can be geocoded or not. I expect a 502 Bad Gateway error from Nominatim would be logged, but never cause an address to fail to be added/updated. Obviously that would occur without the latitude/longitude being recorded. I can't even see the utility in informing the user. It's a system admin problem.
Environment information
----------------------------------------
* __CiviCRM:__ _5.35.2_
* __PHP:__ _7.3__
* __CMS:__ _Drupal 7.82_
* __Database:__ _MariaDB 10.4.21_
* __Web Server:__ _Nginx 1.10.3_
Comments
----------------------------------------
_Anything else you would like the reviewer to note._https://lab.civicrm.org/dev/core/-/issues/2928Expand Gender options?2024-01-16T16:07:07ZJoeMurrayExpand Gender options?Overview
----------------------------------------
Currently core has a Gender field with options in tarball of: Male, Female, Other. There has been a great deal of social change in this area with widespread use of additional terms. In ad...Overview
----------------------------------------
Currently core has a Gender field with options in tarball of: Male, Female, Other. There has been a great deal of social change in this area with widespread use of additional terms. In addition, it might be useful to have default questions and options for whether a contact is transgendered and what pronouns they would like.
Example use-case
----------------------------------------
1. Click on **Contacts -> New Individual**.
1. Enter **First Name** and **Last Name** and click **Save**.
Current behaviour
----------------------------------------
![2021-10-26_11-46-51](/uploads/f5985494ccec6ce0c0c143cc688014b2/2021-10-26_11-46-51.png)
Proposed behaviour
----------------------------------------
I'm going to break up the proposal into several parts that can be discussed and possibly approved separately.
1. Add to the list of gender options after Other: Prefer not to specify.
2. After Male and before Other, insert new gender options: Non-binary, Genderfluid.
2. Change from Female to Female/woman and from Male to Male/man.
3. Change the Gender field from single select to multiselect.
4. Add a new core field under Gender: Are you transgender? Yes, No, Prefer not to specify.
5. Add a new core field under the above: What pronouns do you use? He/him/his, She/her/hers, They/them/theirs, Zie/hir/hirs, Other.
6. Make the field for Pronouns multiselect.
An alternative that might make sense is to do some of this in an extension. Another alternative would be to assist with best practices by providing questions or options but defaulting to disabled.
Comments
----------------------------------------
Here is a note from an organization that works in this area in response to my query on behalf of a client for a longer list of genders:
> Coming up with a universal list of terminology to describe trans people can be difficult, since the exact terms used to describe the concepts involved varies dramatically from language to language and culture to culture. While trans communities in Canada and the US tend to use the same terminology, it can vary even with other majority Anglophone countries, to say nothing of the rest of the world.
>
> There is a lot of discussion out there in the data world about how best to record sexual orientation and gender identity (or “SOGI”) data, and best practices depend on the field in question. That said, a good starting point is to divide things into four questions framed like so:
>
> 1. What is your gender? (select all that apply)
> a. Female/woman
> b. Male/man
> c. Non-binary
> d. Genderfluid
> e. Other
> f. Prefer not to specify
>
> 2. Are you transgender?
> a. Yes
> b. No
> c. Prefer not to specify
>
> 3. What pronouns do you use? (select all that apply)
> a. He/him/his
> b. She/her/hers
> c. They/them/theirs
> d. Zie/hir/hirs
> e. Other
>
> Breaking things out like this has several advantages. First, it recognizes that “transgender” is not itself a gender identity, but is an umbrella category that includes certain men, women, and non-binary and genderfluid people. It simplifies the first question, keeping a shorter list for people to have to scroll through to find the proper term to describe themselves. It also allows organizations to easily modify the list to fit their specific use cases – an organization working in Samoa or with a significant Samoan population could add “fa’afafine” to the list while one dealing with a significant Native/First Nations population could add “two-spirit.”
>
> It also permits trans individuals to interact with the organization without either lying or outing themselves, by not forcing them to select either (for example) “cisgender woman” or “transgender woman.” (If you’re not familiar with the term, ‘cisgender’ is a word indicating a person who is not ‘transgender’.) In fact, it also allows the organization to sort their data in such a way that they don’t unnecessarily out trans people to staff and volunteers, by allowing them to pull a list of all women or all men that doesn’t automatically flag whether or not they are transgender.
>
> In addition, it reflects the fact that not all non-binary and genderfluid people consider themselves to fit under the “transgender” umbrella – this is a complicated issue that’s largely unknown outside of the larger LGBTQ community, and is often disregarded by organizations working with these communities.
>
> Finally, it flags the most useful information – what pronouns to use when referring to the person – into a separate easy to reference field that can be included on a display screen or other record without being specifically tied to someone’s transgender status.
>
> Of course, this is only scratching the surface of the issues that might be relevant to a specific organization’s needs. For instance, they should also consider having a separate “legal name” and “preferred name” field – the former is something that would only be referenced when writing checks or otherwise transacting business in the person’s legal name, but have the “preferred name” be what actually displays for all other interactions with the person. This would also allow cisgender people who primarily go by a short form, initials, or nickname to have that be displayed instead of their full name.
>
> A medical organization, meanwhile, might need to collect information on whether or not someone is intersex, or what their “gender assigned at birth” is, and there can be a lot of ways to capture this information based on how the organization interacts with its staff and patients.
>
> For additional discussion on collecting SOGI data, with multiple examples, please take a look at these resources:
>
> https://www.thetrevorproject.org/wp-content/uploads/2021/07/Measuring-Youth-Sexual-Orientation-and-Gender-Identity.pdf
>
> https://williamsinstitute.law.ucla.edu/quick-facts/survey-measures/JoeMurrayJoeMurrayhttps://lab.civicrm.org/dev/core/-/issues/2926Proposal: "Open in new tab" on .crm-summary-link icons should open contact pa...2022-03-19T14:07:30ZBradley TaylorProposal: "Open in new tab" on .crm-summary-link icons should open contact page, not summary content.When searching for contacts (as well as in some other contexts) each contact has a contact-type icon which shows some extra details on hover:
![Screenshot_2021-10-24_at_15.28.16](/uploads/064fefcdbc311b69c021809af11d6534/Screenshot_2021...When searching for contacts (as well as in some other contexts) each contact has a contact-type icon which shows some extra details on hover:
![Screenshot_2021-10-24_at_15.28.16](/uploads/064fefcdbc311b69c021809af11d6534/Screenshot_2021-10-24_at_15.28.16.png)
Often when searching it is ideal to open each result in a new-tab, whilst keeping the original search page in the original tab. However, when right-clicking on the contact-icon and doing "open in new tab" you get a page something like this (i.e. the contents of the summary popup):
![Screenshot_2021-10-24_at_15.30.27](/uploads/44e0a6aa0c3cdf7486d9c2d2f8f4de1c/Screenshot_2021-10-24_at_15.30.27.png)
I think that this should actually open the contact record instead. (note that click is preventDefault'd, but this doesn't stop the ability to open in a new tab).
Currently the JS uses the `a` element's `href` attribute to populate the summary popup. I suggest that a data attribute is used instead, allowing the `href` to point to the contact record. If we do this, I think there may be an argument for removing the preventDefault on click.
The `crm-summary-link` class may be used by third-party extensions. We can retain backwards compatiability for these extensions by keeping the current logic in cases where the new data attribute is not set.https://lab.civicrm.org/dev/core/-/issues/2864Meta - token usage 5.43 standardisation effort2022-09-02T01:38:26ZeileenMeta - token usage 5.43 standardisation effortWe are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can ...We are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can be around the one release - documentation is [here](https://lab.civicrm.org/-/ide/project/documentation/docs/sysadmin/tree/case/-/docs/upgrade/version-specific.md/)
The goal is to consolidate such that
1. All token rendering is done through the token processor, token hooks are redundant and deprecated
2. All functions on CRM_Utils_Token are unused in core and deprecated except for those in the legacy BAO mailing classes.
3. We freeze, deprecate and phase out in legacy BAO classes in favour of flexmailer
4. All core classes that render tokens do so using consistent token naming & output conventions (rather than dates being formatted in various ways and fields sometimes outputting labels and other times values or name fields https://lab.civicrm.org/dev/core/-/issues/2650)
5. As much as possible we render via tokens rather than smarty in workflow message templates (that couples them with the entity rather than any form flow)
6. pdf & email & scheduled reminder tasks consistently offer the same entity-specific, contact & domain tokens
7. We can start to define obvious missing tokens such as `{domain.now}` (done), `{contribution.balance}` and formatted addresses
Task list / streams - this gives some idea of where things are progressing or blocked
- [x] standardise contact tokens to support/ advertise label style (e.g `{contact.prefix_id:label}`
- [x] make Contact tokens work with partial contact inputs - currently they only work with no contact array passed in, or an array with all possible values - currently blocked on a caching bug https://github.com/civicrm/civicrm-core/pull/21790
- [x] split contact tokens into own class & extend entity tokens like the other classes(blocked on stdise contact tokens)
- [x] switch contact tokens to use apiv4
- [x] work through standardising location token loading (this could go further - ie I had thought about adding loading for billing
- [x] date formatting standardisation [documentation](https://docs.civicrm.org/user/en/latest/common-workflows/tokens-and-mail-merge/#date) https://lab.civicrm.org/dev/core/-/issues/2746
- [x] locale sensitive standardised money formatting https://lab.civicrm.org/dev/core/-/issues/2638 & [pr](https://github.com/civicrm/civicrm-core/pull/21783)
- [x] switch badge tokens to use token processor
- [x] event tokens cleanup & loading fixes + remove unindexed query, move the `fee_amount` and `balance` tokens to the participant tokens class, fix event class to listen to either 'participantId' or 'eventId'. Participant class should run first & load eventId if required. Events should be cached in a static cache as it would be very rare to message thousands of events in one run.
- [x] Recurring contribution tokens work (currently availble in only the recurring_edit template with a pr to extend to cancelled recurring https://test.civicrm.org/job/CiviCRM-Core-PR/44632/
- [x] pdf letter is sane (https://lab.civicrm.org/dev/core/-/issues/2790) enough to extend to deliver tokens for other entities (participant https://lab.civicrm.org/dev/core/-/issues/780)
- [x] email letter extend to deliver tokens for other entities (membership & participant) (https://lab.civicrm.org/dev/core/-/issues/2862)
- [x] scheduled reminders supports participant tokens https://lab.civicrm.org/dev/core/-/issues/779
- [x] domain tokens are consistently available & render https://lab.civicrm.org/dev/core/-/issues/2838 - done but a gap on case tasks as they prioritise being able to filter tokens by case type
- [x] install flexmailer on new installs https://lab.civicrm.org/dev/core/-/issues/2836
- [ ] fix remaining places that process tokens other than via the payment processor
- [x] eliminate replaceGreeting - currently [blocked on apiv4 caching bug](https://github.com/civicrm/civicrm-core/pull/21790)
- [ ] fix CRM_Contact_BAO_Contact_Utils::updateGreeting to not call `getTokenDetails`
- [ ] confirm civicrm_api3_mailing_preview is replaced by flexmailer
- [x] remove getTokenDetails call from CRM_Contribute_Form_Task_PDFLetter - note this appears to be an expensive way to call the contact api & tokens are unused https://github.com/civicrm/civicrm-core/pull/21805
- [x] remove extra rendering in BAO_Pledge https://github.com/civicrm/civicrm-core/pull/21789
- [x] removed getTokenDetails from updatePledgeStatus https://github.com/civicrm/civicrm-core/pull/21847
- [x] remove getTokenDetails from transitionParticipants
- [ ] remove getTokenDetails from sendCancellation & cancelParticipant in selfsvc flow
- [ ] deliverGroup - replaced by flexmailer?
- [ ] MailingPage::preview - replaced by flexmailer?
- [ ] remove hook:;tokenValues from label task
- [ ] getAnonymousTokenDetails ???
- [ ] confirm BAO_Mailing::compose is only via flexmailer
- [x] communicate rc status via dev list
- [x] communicate changes via blog
- [ ] upgrade message https://lab.civicrm.org/dev/core/-/issues/2871
- [ ] docs update to these instructions / check it can still all be done https://docs.civicrm.org/dev/en/latest/step-by-step/create-custom-case-token/
**Currenly out of scope**
- [ ] Token.getlist api - until we have this we have an issue with audience-filtering and with switching tokens depending on the schema. Also case tasks still have to render tokens weirdly without this because they currently filter by case type with partial success https://lab.civicrm.org/dev/core/-/issues/2788
- [ ] Clarify how extensions should render tokens https://lab.civicrm.org/dev/core/-/issues/2863
- [ ] encourage flexmailer with status checks on existing installs https://lab.civicrm.org/dev/core/-/issues/2836
Analysis of current token rendering it core:
Method | Used in | Status
-- | -- | --
Use the token processor | scheduled reminders, civimail with flexmailer in use, activity pdf task | This is the method we are consolidating all paths to use.
Use the functions in CRM_Utils_Token | PDF tasks other than activity | Migrated (PRs on [membership](https://github.com/civicrm/civicrm-core/pull/21521) and [contribution](https://github.com/civicrm/civicrm-core/pull/21524). Also work on fixing up the [pdf task class](https://lab.civicrm.org/dev/core/-/issues/2790) supporting[add participant tokens](https://lab.civicrm.org/dev/core/-/issues/780) (once we’ve finished [the standardisation of those](https://github.com/civicrm/civicrm-core/pull/21587))
| Email tasks | Migrated to token processor. Added tokens for other entities (participant, membership https://lab.civicrm.org/dev/core/-/issues/1396) [also see](https://lab.civicrm.org/dev/core/-/issues/2862)
| Export tasks (when merging contacts in export) | Migrated to token processor
| Profile link rendering | Migrated to token processor
| Storing contact greetings (eg. email_greeting_display) | In progress. Currently resolving inconsistent token syntax and load issues as this needs to be able to use partial pre-loads
Use random ad hoc code | Event badges | [migrated](https://github.com/civicrm/civicrm-core/pull/21587) - this is the only place outside of scheduled reminders where event or participant tokens were used in core
| Send SMS task (BAO_Activity::sendSMS) | migrated
| Labels task CRM_Contact_Form_Task_Label | argh
| CRM_Event_Form_SelfSvcTransfer | Still pretty argh - but this is a workflow message template so the fix is a bit different
| Transition participants | Also a workflow - possibly redundant as seems to be mostly about contact tokens which are resolved in the send function anyway
| Pledge acknowledgements | Also appear to be redundant
| CiviMail with flexmailer not used (many functions & classes) | Push the switch to flexmailer.
| Various unsubscribe / resubscribe type actions | Probably out of scope for this roundhttps://lab.civicrm.org/dev/core/-/issues/2863Agree / document how tokens should be rendered from outside core2023-09-24T22:52:19ZeileenAgree / document how tokens should be rendered from outside coreIdeally we would have one recommended api way to do this. This would be a way that is usable via api explorer and all the ways we use the api and core code would be converted to do things using that same api method
There is [a documenta...Ideally we would have one recommended api way to do this. This would be a way that is usable via api explorer and all the ways we use the api and core code would be converted to do things using that same api method
There is [a documentation pr here](https://lab.civicrm.org/documentation/docs/dev/-/merge_requests/962) - at the moment I don't think it meets this need.
**Permissions & non-workflow rendering**
There is a question as to whether the api would support non-workflow messaging (in which case the api would likely be more like `Message::render()`. The argument against is that the permissions are more complex. However some notes on that
1) the most common use case is probably when checkPermissions = FALSE in php code.
2) there is an appropriate permission already in core
```
render templates' => [
$prefix . ts('render templates'),
ts('Render open-ended template content. (Additional constraints may apply to autoloaded records and specific notations.)'),
],
```
3) currently the v3 MessageTemplate.send api allows access to the core function - it required a messageTemplateID to be set - but this can probably be faked to allow 'any text'https://lab.civicrm.org/dev/core/-/issues/2858Importing soft credit payments by email address fails with confusing message ...2022-06-11T23:42:51ZjhungerfordImporting soft credit payments by email address fails with confusing message if dedupe rule requires extra fieldsOverview
----------------------------------------
Importing soft credit payments (matched to existing contacts by email address) fails if the relevant Unsupervised dedupe rule requires any extra fields. The error message in this case is:...Overview
----------------------------------------
Importing soft credit payments (matched to existing contacts by email address) fails if the relevant Unsupervised dedupe rule requires any extra fields. The error message in this case is:
`Invalid email address(doesn't exist) email@example.com. Row was skipped`
The proposed change passes any required fields to the dedupe check if they are available in the import, but does not amend the error message if it fails.
Example use-case
----------------------------------------
- Add to the Unsupervised rule so it requires First and Last names as well as email address
- Import soft credit payments where the Fundraiser ID is set by External Identifier (or Contact ID) and the Donor is matched by email, First Name, and Last Name
Current behaviour
----------------------------------------
Currently each row in such an import will fail with this message:
`Invalid email address(doesn't exist) email@example.com. Row was skipped`
Proposed behaviour
----------------------------------------
If the required fields are available in the import data, they could be passed to the dedupe function. This is what I've done here (on version 5.39.2):
https://github.com/AsylumSeekersCentre/civicrm-core/commit/c924fb95a2e311a442656bf65c23523091d351da
If the required fields are not available, it would be better if the error message pointed the user in the right direction. Currently it says that the email address does not exist, but in the case of our import, all of the email addresses did exist for contacts of the appropriate type. I haven't addressed this part of the issue with the code above.
Comments
----------------------------------------
I think the patch above is not the whole solution. That code pathway is used when there is no External Identifier or Contact ID but there is an email address. It then calls the Unsupervised dedupe rule, which may not refer to the email address at all (though the default does). The change I've proposed accommodates extra fields added to the rule, but doesn't consider the possible removal of the email address from the rule.5.51.0https://lab.civicrm.org/dev/core/-/issues/2761Proposal: Change Pay Later Billing Address to look up a profile rather than a...2022-10-04T21:07:14ZBarijohnProposal: Change Pay Later Billing Address to look up a profile rather than a coded block.Overview
----------------------------------------
Currently on the pay later option when you use the option of Billing Address the fields are hard coded. This would be better to use a profile then it could be edited by the end user witho...Overview
----------------------------------------
Currently on the pay later option when you use the option of Billing Address the fields are hard coded. This would be better to use a profile then it could be edited by the end user without having to mess with the code.
Example use-case
----------------------------------------
Click on Pay Later Option and enable Billing Fields. This shows multiple fields that might not be required depending on the part of the world. So in that case there could be a reserved profile just for billing fields. This could have a help field to show where this profile is saved, so it would easily amended.
Current behaviour
----------------------------------------
Currently this is set in Core/Payment.php and /civicrm/templates/CRM/Core/BillingBlock.tpl (based on communications with our devs).
Proposed behaviour
----------------------------------------
Make this block look up a billing address profile instead.
Comments
----------------------------------------
Currently this field asks for information that isn't required in the UK and most of our clients would like to be able to edit this directly.https://lab.civicrm.org/dev/core/-/issues/2760Proposal: Stop storing file ids in two tables2023-04-17T15:39:24ZcolemanwProposal: Stop storing file ids in two tablesFile fields in CiviCRM are strange.
When you upload a file to a custom field, it stores a record in the `civicrm_file` table. Then it stores that id in **two** other tables. Once in the custom value column for that field, as you'd expec...File fields in CiviCRM are strange.
When you upload a file to a custom field, it stores a record in the `civicrm_file` table. Then it stores that id in **two** other tables. Once in the custom value column for that field, as you'd expect, and then it also adds a record in the `civicrm_entity_file` table.
https://github.com/civicrm/civicrm-core/blob/5.40/CRM/Core/BAO/CustomValueTable.php#L161-L171
From a data archetecture POV, there's no upside to storing the same value twice in two places, it just creates problems.