CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2022-04-22T18:06:46Zhttps://lab.civicrm.org/dev/core/-/issues/3190How to package SearchKit Displays in an extension2022-04-22T18:06:46ZcolemanwHow to package SearchKit Displays in an extension## Use Case
Ship an extension with SearchKit-based tabs for the Contact Summary screen.
## Requirements
Using a Search Display as a contact summary tab requires 3 interconnected entities:
1. A **SavedSearch** containing the api param...## Use Case
Ship an extension with SearchKit-based tabs for the Contact Summary screen.
## Requirements
Using a Search Display as a contact summary tab requires 3 interconnected entities:
1. A **SavedSearch** containing the api params for the search.
2. A **SearchDisplay** linked to the `SavedSearch` by id (`saved_search_id = $savedSearch.id`).
3. An **Afform** linked to both the `SavedSearch` and the `SearchDisplay` by name (via embedded tag).
## Solutions
### 1. The Status Quo
Afforms can already be packaged as `.aff.html/.aff.json` files, and the Afform API & GUI have excellent handling for packaged Afforms to be overridden and reverted by site admins. If an extension upgrade provides updates to a packaged Afform, overrides will not be affected but the default version will be upgraded automatically.
DAO-based entities like `SavedSearch` and `SearchDisplay` can be packaged as `.mgd.php` files which will be added to the database via `hook_civicrm_managed`. However, the `CRM_Core_ManagedEntities` system doesn't provide any way for admins to override and then revert back to the packaged version, nor to reconcile potentially overridden entities with new packaged versions during extension upgrades. There are also an issue where the system doesn't guerantee write order, and in this case we need it to write the `SavedSearch` prior to the `SearchDisplay`, which contains a foreign key to the former.
Currently there is a workaround using chaining and `'update' = 'never'`, for example as used by the [Deduper Extension](https://github.com/eileenmcnaughton/deduper/blob/master/entities/SavedSearch.mgd.php).
`CRM_Core_ManagedEntities` offers us 2 modes for packaging `SavedSearch` and `SearchDisplay` entities, neither is ideal:
1. **`'update' = 'always'`** will essentially lock the entities. Any changes to the search made by the site admin will be overwritten on a regular basis. This solves the extension upgrade problem at the cost of configurability.
2. **`'update' = 'never'`** will drop the entities into the database and then never touch them again. This is the lesser of two evils because at least this way they are configurable.
**The bottom line:** The status quo allows us to package searches and displays. They will be user-configurable but not revertable (if the admin makes a mess of them, he'd have to delete them all and then uninstall/reinstall the extension). During extension upgrades, the packaged Afforms will update, but changes to `.mgd.php` files will do nothing unless the extension is uninstalled, searches are deleted, and then it's reinstalled.
### 2. Create APIv4 `PackagedEntity` class
This would do something like the `Afform` APIv4 entity: read from the database and *also* from json files. This would need to be opt-in on a per-entity basis (starting with `SavedSearch` and `SearchDisplay`) and would be mostly for configuration entities, as a file/DB hybrid would not be able to perform joins in the GET action.
**Tasks**
1. Add an APIv4 trait `PackagedEntity`
2. Opt-in `SavedSearch` and `SearchDisplay` to use that trait
3. Add a specialized `get` action which would read from json files and the database.
4. Add calculated fields comparable to Afform's `has_base` & `has_local` to `get` output.
**Pros**
- Would give excellent feature parity with Afform's packaged systems.
- Reverting and upgrades would be simple.
**Cons**
- Involves extra file i/o and directory scanning with each `get` - feasible with hundreds of records but not many thousands.
- Restricted to APIv4 as the packaged entities wouldn't exist as far as the DAO is concerned.
- Entities have to opt-in (whereas `mgd` works on any entity).
- Retrofitting an existing API over would involve a slight BC breakage as the new `get` method would be missing `JOIN`, and `HAVING` clauses.
### 3. Improve `CRM_Core_ManagedEntities`
Make the manager aware of whether a record has been manually updated. Add a new `'update' = 'auto'` mode which would intelligently push updates from `.mgd.php` only to untouched records. Add a way to revert a record.
**Tasks**
1. Add an `is_overridden` column to the `civicrm_managed` table.
2. Add some type of trigger to set that column when manual changes are made to a managed entity.
3. Add some type of `Revert` api action (currently the `Managed` class doesn't even have an API so this will take some thought).
4. Add a way to know if a database record has a corresponding managed entry (maybe a calculated field or maybe a DFK join - both would be slightly tricky).
**Pros**
- Fully backward compatible with existing entities.
- Improves the existing "Managed" system instead of inventing a new one.
- Wouldn't be a drag on performance.
**Cons**
- Might be a lot of work.https://lab.civicrm.org/dev/core/-/issues/3189Search-kit offer actions when grouped by an entity2023-12-15T05:03:27ZeileenSearch-kit offer actions when grouped by an entityWhen we have a search with an entity as group by we should offer actions for that entity
eg.
```
SELECT contact_id, SUM(total_amount) as totes FROM civicrm_contribution GROUP BY contact_id HAVING totes > 2000;
```
Should have contact...When we have a search with an entity as group by we should offer actions for that entity
eg.
```
SELECT contact_id, SUM(total_amount) as totes FROM civicrm_contribution GROUP BY contact_id HAVING totes > 2000;
```
Should have contact actions & the ability to create a smart groups
See https://github.com/civicrm/civicrm-core/pull/19936
@colemanwhttps://lab.civicrm.org/dev/core/-/issues/3156Proposal: include translations in installation tar/zip2023-12-10T05:03:30ZalainbProposal: include translations in installation tar/zipOverview
----------------------------------------
The default language of CiviCRM is en_US. Other languages are not included in the standard installation package.
Looking at the countries and languages on stats.civicrm.org, we can see t...Overview
----------------------------------------
The default language of CiviCRM is en_US. Other languages are not included in the standard installation package.
Looking at the countries and languages on stats.civicrm.org, we can see that a big part of the community is using another language than en_US.
It would be more inclusive and respectful to include all languages in the standard installation package.
We should practice what we preach in our [code of conduct](https://civicrm.org/code-of-conduct): "_Communities mirror the societies in which they exist and positive action is essential to counteract the many forms of inequality and abuses of power that exist in society._"
Current behaviour
----------------------------------------
If you want to install CiviCRM in another language than en_US, you need to manually download the l10n file, and unpack it in the appropriate folder. This is an extra (technical) step for non en_US speakers.
Proposed behaviour
----------------------------------------
Include all languages in the standard installation package, or dynamically load the requested language during installation and upgrades.https://lab.civicrm.org/dev/core/-/issues/3080Duplicate Billing location checkbox for a contact is allowed during inline ad...2022-04-07T20:59:16ZVangelisPDuplicate Billing location checkbox for a contact is allowed during inline address editingOverview
----------------------------------------
In the case that someone has 2 addresses registered, they can flag both of them as billing location via the inline address editing checkbox which I believe is wrong. This behaviour does n...Overview
----------------------------------------
In the case that someone has 2 addresses registered, they can flag both of them as billing location via the inline address editing checkbox which I believe is wrong. This behaviour does not replicate when you edit the contact.
I am assuming that this is not an intended behaviour.
Reproduction steps
----------------------------------------
1. On the [CiviCRM demo site](https://dmaster.demo.civicrm.org) go to the default `demo@example.com` contact.
1. Do an inline edit and add an address. Check the checkbox "Billing location for this contact".
1. Add a 2nd address via inline editing and check again the checkbox "Billing location for this contact" on the 2nd address.
1. Review both addresses, both got the "Billing location for this contact".
1. Try to edit the contact using the edit button, check both checkboxes and save the form. Only one address gets the checkbox "Billing location for this contact".
Expected behaviour
----------------------------------------
There should be a validation while saving the inline address form to check if the checkbox is already being used in another address ID and if yes, it should not store the checkbox (or unset it).5.49.0https://lab.civicrm.org/dev/core/-/issues/3067Improving performance of retreiving display names for multivalued custom data2022-11-25T15:39:20ZMichael McAndrewImproving performance of retreiving display names for multivalued custom data.Have been taking a look at https://github.com/civicrm/civicrm-core/pull/20764/files#diff-36ab608da0b5718996afd18c10a8c12653c2a1675e81fd3c4dc3e40c2ebb25d2R1040.
My hunch is that for a query that returns 50,000 rows, the 'group_concat fi....Have been taking a look at https://github.com/civicrm/civicrm-core/pull/20764/files#diff-36ab608da0b5718996afd18c10a8c12653c2a1675e81fd3c4dc3e40c2ebb25d2R1040.
My hunch is that for a query that returns 50,000 rows, the 'group_concat find_in_set' subquery is executed 50,000 times even if pagination is turned on since the limit is applied after it does all the subqueries
If you turn that subquery into a 'superquery', i.e. you do the 'group_concat find_in_set' hydration after the limit, it imrpoves the performance a lot.
I did a proof of concept below. For a query with 500 results it went from 37 seconds to 1.3 seconds.
And for 34,000 it went from never completing to 1.0 seconds.
Not sure how easy it would be to acheive this within the constraints of searchkit @colemanw but it would be great to get your initial thoughts on this.
Proof of concept changes to the query below...
Changing this query:
```sql
SELECT `a`.`id`, (
SELECT GROUP_CONCAT(
`display_name`
ORDER BY FIND_IN_SET(`civicrm_contact`.`id`, REPLACE(`Case_of_abuse_details_1`.`perpetrator_143`, '�', ','))
SEPARATOR '�'
)
FROM `civicrm_contact`
WHERE FIND_IN_SET(`civicrm_contact`.`id`, REPLACE(`Case_of_abuse_details_1`.`perpetrator_143`, '�', ','))
) AS `Case_of_abuse_details.Perpetrator.display_name`
FROM civicrm_activity a
LEFT JOIN `civicrm_value_case_of_abuse_68` `Case_of_abuse_details_1` ON a.id = Case_of_abuse_details_1.entity_id
WHERE ((`a`.`activity_type_id` IS NULL OR (`a`.`activity_type_id` IN (1, 2, 3, 4, 5, 9, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 40, 44, 45, 48, 49, 50, 51, 52, 53, 55, 57, 59, 61, 63, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74)))) AND (`a`.`activity_type_id` = "68") AND (`a`.`is_test` = "0") AND (`a`.`is_deleted` = "0")
GROUP BY `a`.`id`
LIMIT 50
OFFSET 0
```
into this query
```sql
SELECT id,(SELECT GROUP_CONCAT(
`display_name`
ORDER BY FIND_IN_SET(`civicrm_contact`.`id`, REPLACE(`perpetrator_143`, '�', ','))
SEPARATOR '�'
)
FROM `civicrm_contact`
WHERE FIND_IN_SET(`civicrm_contact`.`id`, REPLACE(`perpetrator_143`, '�', ','))
) AS `Case_of_abuse_details.Perpetrator.display_name` FROM (SELECT `a`.`id`, `Case_of_abuse_details_1`.`perpetrator_143`
FROM civicrm_activity a
LEFT JOIN `civicrm_value_case_of_abuse_68` `Case_of_abuse_details_1` ON a.id = Case_of_abuse_details_1.entity_id
WHERE ((`a`.`activity_type_id` IS NULL OR (`a`.`activity_type_id` IN (1, 2, 3, 4, 5, 9, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 40, 44, 45, 48, 49, 50, 51, 52, 53, 55, 57, 59, 61, 63, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74)))) AND (`a`.`activity_type_id` = "68") AND (`a`.`is_test` = "0") AND (`a`.`is_deleted` = "0")
GROUP BY `a`.`id`
LIMIT 50
OFFSET 0) AS alias
```https://lab.civicrm.org/dev/core/-/issues/3053Usability issue reported by user: All Reports page overrides report results s...2023-11-20T05:03:18ZDevAppUsability issue reported by user: All Reports page overrides report results settingWhen clicking on a report name on the All Reports page located at /wp-admin/admin.php?page=CiviCRM&q=civicrm%2Freport%2Flist&reset=1
The resulting link loads the report with output=criteria on the end. This stops the default parameters ...When clicking on a report name on the All Reports page located at /wp-admin/admin.php?page=CiviCRM&q=civicrm%2Freport%2Flist&reset=1
The resulting link loads the report with output=criteria on the end. This stops the default parameters of the results from loading, which was set to view results. The user expected results to appear when loading the report per the saved report setting. The result to the user is the report is broken.
The output=criteria is overriding the saved report behaviour. Whilst there is a view results button the right hand side of the All Reports page, it does create some confusion to the user with consistency. Perhaps if the setting is already set on the report to view results, the output=criteria could be removed from that link to create consistency... Or the GUI clearer about why the report wasn't run.
This isn't a bug, but more a usability issue.https://lab.civicrm.org/dev/core/-/issues/3049Auto-complete option values aren't available to anonymous users2023-09-02T01:04:32ZJonGoldAuto-complete option values aren't available to anonymous usersOverview
----------------------------------------
A custom field of input type "Autocomplete-Select" doesn't return results for anonymous users, even with the "Access AJAX API" permission granted.
Reproduction steps
--------------------...Overview
----------------------------------------
A custom field of input type "Autocomplete-Select" doesn't return results for anonymous users, even with the "Access AJAX API" permission granted.
Reproduction steps
----------------------------------------
1. Change an existing field (e.g. "Soup Selection") from **Dropdown** to **Autocomplete-Select**.
1. Add the field to a profile on a public-facing event page.
1. Grant the anonymous user the **Access AJAX API** permission.
1. View as an anonymous user.
Current behaviour
----------------------------------------
No results are returned.
Expected behaviour
----------------------------------------
Results are returned, as they are with a "Dropdown" input type.
Comments
----------------------------------------
The error is that anonymous users don't have access to the "Optionvalue.getlist" API because it requires "Access CiviCRM". This seems like the wrong permission; I think "Access AJAX API" should be sufficient. From a UX perspective, it's pretty inconsistent that a Select works but not an Autocomplete-Select.
I'm proposing that we allow anonymous users to access OptionValue.get - I'm interested in whether there are use cases where this results in an information disclosure vulnerability. I can't think of any myself.https://lab.civicrm.org/dev/core/-/issues/3037Proposal: update 2-digit year import to be configurable/rolling2022-01-28T14:54:48ZnoahProposal: update 2-digit year import to be configurable/rollingOverview
----------------------------------------
The handling of 2-digit years during import uses a heuristic that no longer makes sense. For example, "1/1/22" is imported as January 1, **19**22 (not 2022). We should implement somethin...Overview
----------------------------------------
The handling of 2-digit years during import uses a heuristic that no longer makes sense. For example, "1/1/22" is imported as January 1, **19**22 (not 2022). We should implement something that makes more sense.
Current behaviour
----------------------------------------
Currently the following is hard-coded into CRM/Utils/Date.php within `convertToDefaultDate()`.
```php
// simple heuristic to determine what century to use
// 00 - 20 is always 2000 - 2020
// 21 - 99 is always 1921 - 1999
if ($year < 21) {
$year = (strlen($year) == 1) ? $cen . '0' . $year : $cen . $year;
}
elseif ($year < 100) {
$year = $prevCen . $year;
}
```
This affects the import of Activity dates and presumably other places where a two-digit year input is selected.
Proposed behaviour
----------------------------------------
- Civi should translate 2-digit years into 4-digit years based on a rolling cutoff. For example, if the cutoff is the current date plus ten years, any two-digit input that would result in a date more than ten years in the future will be placed in the prior century instead.
- The cutoff could be configurable: X years in the future.
Comments
----------------------------------------
I guess configuration would go under civicrm/admin/setting/preferences/date (`CRM_Core_BAO_PreferencesDate`).
Optionally, the setting could be overridable within the import wizard itself.5.47.0https://lab.civicrm.org/dev/core/-/issues/3009authx: Proposal: Make "require" the default user policy instead of "optional"2023-11-04T05:03:19ZDaveDauthx: Proposal: Make "require" the default user policy instead of "optional"If you look at https://docs.civicrm.org/dev/en/latest/framework/authx/#settings the default setting for the `XXX_user` settings is "optional" which means a corresponding cms user doesn't need to exist in order to log in. This seems unint...If you look at https://docs.civicrm.org/dev/en/latest/framework/authx/#settings the default setting for the `XXX_user` settings is "optional" which means a corresponding cms user doesn't need to exist in order to log in. This seems unintuitive as a default. My expectation is that it would work the same way as regular logins by default, so should default to "require".https://lab.civicrm.org/dev/core/-/issues/3003Preserve previous tab when navigating to and from contact page2022-05-05T14:57:50ZBradley TaylorPreserve previous tab when navigating to and from contact pageThe contact page is structured around a series of tabs: Summary, Contributions, Membership etc. There are often times when you click through from a contact to a related entity (say a related contact from the relationships tab). It would ...The contact page is structured around a series of tabs: Summary, Contributions, Membership etc. There are often times when you click through from a contact to a related entity (say a related contact from the relationships tab). It would be nice if using the browser's back button returned you to the tab you were previously looking at rather than returning to the summary tab each time.
This is a bigger issue for users with the "Enable Popup Forms" option unticked, but all users are affected by this.
This shouldn't be a massive technical change, but I do think this is one of those little annoyances, where fixing should improve the overall usability of the CRM for those who need to use it on a daily basis.
**Implementation details**
It is already possible to link to a specific tab with the `selectedChild` query parameter. I propose that whenever the active tab is changed, the [replaceState](https://developer.mozilla.org/en-US/docs/Web/API/History/replaceState) API is used to amend the `selectedChild` parameter in the URL.5.49.0https://lab.civicrm.org/dev/core/-/issues/2994Support oEmbed for external facing pages2024-03-21T18:50:44ZJoeMurraySupport oEmbed for external facing pagesAn important evolution of CiviCRM is to better support remote sites exposing CiviCRM forms and content.
Many platforms including WordPress and Drupal core since 8.6 support oEmbed, which allows a site to pull in content from a different...An important evolution of CiviCRM is to better support remote sites exposing CiviCRM forms and content.
Many platforms including WordPress and Drupal core since 8.6 support oEmbed, which allows a site to pull in content from a different site that exposes its content as oEmbed.
There is a good list of oEmbed providers, and CiviCRM should aspire to join it: https://oembed.com/providers.json
We should expose the following 'pages' as oEmbed content, using appropriate standard markup for use in WordPress and Drupal 8.6+ sites, and support the workflow for confirmation, thank you, and search results as appropriate:
1. Phase 1
1. Contribution page
2. Event info page
3. Event registration page
2. Phase 2
1. Something like a profile create/edit page that does not involve a payment but allows submission of a form to create a contact and activity and perhaps more (eg parent-child relationship)
2. Search Kit page
3. Petition page
3. Phase 3
1. Personal Campaign Page
2. Pages from extensions, eg Grant Application Page
3. CiviSurvey pages
Note: I haven't done the research to understand the details of oEmbed and potential challenges. I think this could be done through extension(s).
I'm posting this in order to generate discussion for a possible Roadmap item.https://lab.civicrm.org/dev/core/-/issues/2993Add option to delete old log files during rotation2023-11-02T05:03:25ZJoeMurrayAdd option to delete old log files during rotationOverview
----------------------------------------
_When creating a new log file, optionally remove old ones like logrotate allows (https://linux.die.net/man/8/logrotate)._
Example use-case
----------------------------------------
CiviCR...Overview
----------------------------------------
_When creating a new log file, optionally remove old ones like logrotate allows (https://linux.die.net/man/8/logrotate)._
Example use-case
----------------------------------------
CiviCRM's simplistic log creation does not actually rotate, it just makes more files for the logs and never deletes any (https://github.com/civicrm/civicrm-core/blob/5433c81f5648bf484ef16debeca3c7e6e98d1a11/CRM/Core/Error.php#L684). This can be dangerous for systems where there is little to no active monitoring of logs, eg small orgs without a provider, or on a shoestring budget. It is also dangerous in cases where a change starts to generate high volumes of error messages, eg a client recently misconfigured a civirule and GBs of log were produced every hour. Running out of disk space causes hard crashes of sites.
It is standard practice to have a process in place to ensure that logs do not end up taking too much disk space. We propose allowing a setting to be added to civicrm.settings.php (eg NUM_LOG_FILES_TO_RETAIN) that would allow the number of logs to be retained to be specified. The default setting would be 0, meaning an infinite number, which would retain current behaviour. We would also be happy to make it another number such as 6, which would be 6 months or more than 1.5GB.
My understanding is that settings that can be added to civicrm.settings.php should be included as comments with explanation that can be uncommented, so we will do this.
Current behaviour
----------------------------------------
_CiviCRM generates log files with random hash names, and creates new ones every month or when the file exceeds 256MB. Log file space grows continuously unless log files are manually deleted._
Proposed behaviour
----------------------------------------
_When a new log file is created, if a maximum number of log files has been specified, then delete those over that number, starting with oldest first_
Comments
----------------------------------------
_To reduce the complexity of the settings pages, we propose that this be implemented as a setting that could be optionally specified in civicrm.settings.php._Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/2991Proposal: rework name fields to make name entity2023-12-13T09:20:59ZAndie HuntProposal: rework name fields to make name entity## Overview
A recent [discussion with Guy](https://chat.civicrm.org/civicrm/pl/wwqowz635fbpifio7on4e8r8dw) and #2883, plus the ongoing mayhem of organization names and personal experience with a variety of great and terrible name handli...## Overview
A recent [discussion with Guy](https://chat.civicrm.org/civicrm/pl/wwqowz635fbpifio7on4e8r8dw) and #2883, plus the ongoing mayhem of organization names and personal experience with a variety of great and terrible name handling systems, have led me to think that the solution to names isn't creating more or better-labeled fields for more types of names but rather to make name its own entity akin to phone, email, address, website, etc.
Once name is an entity, each contact could have multiple names, one primary, but with a bunch of possible types:
- legal name
- trade name
- nickname
- former name
- name in other language/culture
- name in an external system
- common mistakes
There could also be a bunch of name renderings - ways the name is displayed or inserted:
- sort name
- display name
- name for greetings
- name for listings
The latter could generate the existing greetings and potentially others. In particular, I think there's a real gap for how you insert someone's name in emails where you don't want "Dear", "Hi", or any other greeting as such.
Finally, the duplicate matching process could look for any of the possible names, cutting down on duplicates.
## Example use-case
### Organization has multiple names
An organization has their corporate name, colloquial name, and the name of their main product. These all refer to the same contact, but employees might fill the current employer field as one of the various ways. As an admin, you want to avoid having to merge them all the time. As a routine user, you will want to know all the various ways an organization is termed. As a bookkeeper, you will want to send invoices, checks, receipts, etc. with the correct entity name.
In this case, there are three name entities:
| Organization name | Name type | Is primary |
| ------------------------ | --------- | ---------- |
| Widget Productions | General | true |
| Widget Productions, Inc. | Legal | false |
| WidgetPro | General | false |
| Widgetizer | Product | false |
There would be a couple of name renderings:
| Rendering type | Value |
| -------------- | ------------------------ |
| Display Name | Widget Productions |
| Sort Name | Widget Productions |
| Billing name | Widget Productions, Inc. |
### Minister has variety of prefixes
Someone's name is styled differently when it is listed versus when they are addressed. In this case, a minister is styled "The Reverend Sally Jones" or "Rev. Sally Jones" when listed somewhere but her salutation is "Ms. Jones".
I think she'd have only one entry as a name entity but several renderings:
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------------|------------|-------------|-----------|--------|-----------|------------|
| Ms. | The Reverend | Sally | Lynn | Jones | *null* | General | true |
These values are going to depend upon your sitewide preferences and choices for this contact, but a reasonable default might produce the following:
| Rendering type | Value |
|-----------------|--------------------------|
| Display Name | Sally Lynn Jones |
| Sort Name | Jones, Sally Lynn |
| Addressee | The Reverend Sally Jones |
| Email greeting | Dear Sally |
| Postal greeting | Dear Ms. Jones |
| Short name | Sally |
### An organization has a different name in another language
This should be self-explanatory:
| Organization name | Name type | Is primary |
|--------------------------|-----------|------------|
| Médecins Sans Frontières | General | true |
| Doctors Without Borders | Localized | false |
| Rendering type | Value |
|----------------|--------------------------|
| Display Name | Médecins Sans Frontières |
| Sort Name | Médecins Sans Frontières |
| English Name | Doctors Without Borders |
### Someone's family name comes before their given name
I'm on the fence about how this should be handled, but all I know is that we currently handle it terribly (except maybe in translations that assume it?) One option might be to call the fields first, middle, and last names according to position (*Ai* being the family name, but it comes first):
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------|------------|-------------|-----------|--------|-----------|------------|
| Mr. | *null* | Ai | *null* | Weiwei | *null* | General | true |
Alternatively, we could name the fields according to function:
| Prefix | Title | Given name | Additional name | Surname | Suffix | Name type | Is primary |
|--------|--------|------------|-----------------|---------|--------|-----------|------------|
| Mr. | *null* | Weiwei | *null* | Ai | *null* | General | true |
In either case, the renderings should be like
| Rendering type | Value |
|-----------------|---------------|
| Display Name | Ai Weiwei |
| Sort Name | Ai Weiwei |
| Addressee | Mr. Ai Weiwei |
| Email greeting | Dear Weiwei |
| Postal greeting | Dear Mr. Ai |
| Short name | Weiwei |
### Clinic needs to retain legal name for insurance billing
Someone's real name doesn't match their legal name, let alone their nickname. While most organizations wouldn't want or need to record a contact's dead name, a clinic might need it for billing correspondence. In this case, a patient is named Stephen, or Steve for short, but his legal name is Barbara.
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------|------------|-------------|-----------|--------|-----------|------------|
| Mr. | *null* | Stephen | Wayne | Smith | *null* | General | true |
| Mr. | *null* | Steve | *null* | Smith | *null* | Nickname | false |
| *null* | *null* | Barbara | Wayne | Smith | *null* | Legal | false |
All of the name renderings would use the primary name or nickname, but the clinic could add an additional name rendering for how they refer to patients when billing to the insurance company.
| Rendering type | Value |
|-----------------|----------------------|
| Display Name | Stephen Wayne Smith |
| Sort Name | Smith, Stephen Wayne |
| Addressee | Mr. Stephen Smith |
| Email greeting | Dear Steve |
| Postal greeting | Dear Mr. Smith |
| Short name | Steve |
| Insurance name | Barbara Smith |
### Household name possibilities
The current household name field befuddles lots of people and, in my opinion, makes households less attractive to use. If a household can have multiple names, it might be more useful.
You might use a Household name field that has everyone all listed out, or you could use the Last name field.
| Household name | Last name | Name type | Is primary |
|-------------------------------------|----------------|-----------|------------|
| Katherine Williams and Kendra Green | *null* | General | true |
| *null* | Williams-Green | General | false |
| Rendering type | Value |
|-----------------|------------------------------------------|
| Display Name | Katherine Williams and Kendra Green |
| Sort Name | Katherine Williams and Kendra Green |
| Addressee | The Williams-Green Household |
| Email greeting | Dear Katherine Williams and Kendra Green |
| Postal greeting | Dear Katherine Williams and Kendra Green |
Or if everyone shares a last name, a single name entity could do it all:
| Given name | Surname | Name type | Is primary |
|---------------|---------|-----------|------------|
| Jack and Jill | Hill | General | true |
(If we don't feel the need to stick with the legacy name field names, there's really no reason that "Given name" couldn't replace Organization name and Household name, too, so this illustrates that.)
| Rendering type | Value |
|-----------------|-------------------------|
| Display Name | Jack and Jill Hill |
| Sort Name | Hill, Jack and Jill |
| Addressee | The Hill Household |
| Email greeting | Dear Jack and Jill |
| Postal greeting | Dear Jack and Jill Hill |
## Current behavior
Right now, each of these situations is pretty annoying. It might require a separate custom field, located away from the name fields. It might make it difficult to automatically and accurately generate greetings for many people.
Currently, there are fields in the `civicrm_contact` table for each name part, plus nickname and legal name.
The only name renderings are display name and sort name (set sitewide), and addressee and email and postal greetings (site-wide presets and defaults, with custom options).
There's no provision for additional renderings or any way to do fallbacks like pick a nickname and fall back to a first name.
There's a limited number of alternative names for a contact.
Alternative names are not compared against primary names when deduping.
## Proposed behavior
### Schema
The name fields would be gone from the contact entity, and a new `civicrm_contact_name` entity is added with one of the following sets of fields:
| Traditional | Radical |
|-------------------|-------------------|
| id | id |
| contact_id | contact_id |
| contact_name_type | contact_name_type |
| is_primary | is_primary |
| first_name | given_name |
| middle_name | additional_name |
| last_name | surname |
| prefix_id | prefix_id |
| suffix_id | suffix_id |
| formal_title | formal_title |
| household_name | |
| organization_name | |
The `contact_name_type` would have reserved options General, Nickname, Legal, and Localized, but more could be added.
A new `civicrm_contact_rendering` entity is added with the following set of fields:
| Fields |
|-------------------------------|
| id |
| contact_id |
| contact_rendering_type |
| contact_rendering_template_id |
| value |
The `contact_rendering_type` would have reserved options Display name, Sort name, Addressee, Email greeting, Postal greeting, and Short name, but more could be added.
The `contact_rendering_template_id` would be akin to the existing `addressee_id`, `email_greeting_id`, and `postal_greeting_id`, while `value` would be the actual generated or custom value.
### Upgrade
On upgrade, the `civicrm_contact_name` would be populated with the main name fields, with nickname and legal name as additional rows. On upgrade, the `civicrm_contact_rendering` would be populated with rows for each contact's display name, sort name, addressee, and greetings.
### Contact rendering templates
The contact rendering templates would have the option to specify a contact name type to provide the default value, with a fallback to the primary contact name. For example, email greeting might default to "Dear {nickname.first_name}", where it picks the value of `first_name` from a contact's name entry with the name type of Nickname, and if that isn't available, it picks the primary name entry's value for `first_name`.
### Dedupe
When matching contacts on any of the name fields, the dedupe process would compare against all rows in the `civicrm_contact_name` field, regardless of `contact_name_type` or if it's the primary name.
## Comments
This is all very early in the process: I think it's important to do something like this, but I don't have my heart on the specific mechanisms here. I think it would need to be evaluated for performance in a variety of ways, but I don't expect it would have too huge of an impact.https://lab.civicrm.org/dev/core/-/issues/2979Proposal to remove the limit of 15 max values for multiple values can also be...2021-12-09T16:28:04ZyashodhaProposal to remove the limit of 15 max values for multiple values can also be retrieved from url in reportsWe send the parameters as string so that multiple values can also be retrieved from url in reports
for e.g url like - "memtype_in=in&memtype_value=1,2,3
However, there is a restriction for 15 values esp when you search for activity type...We send the parameters as string so that multiple values can also be retrieved from url in reports
for e.g url like - "memtype_in=in&memtype_value=1,2,3
However, there is a restriction for 15 values esp when you search for activity types which can be numerous. In this this case, if the values exceeds 15 the criteria are not respected.
Proposal to remove the limit of 15 max values.
https://github.com/civicrm/civicrm-core/blob/master/CRM/Report/Utils/Get.php#L1735.46.0yashodhayashodhahttps://lab.civicrm.org/dev/core/-/issues/2976Feature Request: Form Builder and Permissions - Provide ability to restrict a...2023-11-02T12:39:11Zjustinfreeman (Agileware)Feature Request: Form Builder and Permissions - Provide ability to restrict access to selected Roles rather than just specific Permission GrantsWould be great if the Form Builder, Permissions had the ability to restrict access to selected Roles rather than just specific Permission Grants.
Roles being far more flexible, can be automatically granted and removed for user accounts ...Would be great if the Form Builder, Permissions had the ability to restrict access to selected Roles rather than just specific Permission Grants.
Roles being far more flexible, can be automatically granted and removed for user accounts and this is a common pattern on Member websites. Whereas Permission Grants are not used in this way.
For example: When a user with the Member role views a Form Builder page, they are granted access to use the Form and view the Search Results because they have the Member role.
![Screenshot_20211202_130808](/uploads/25c2fbadece5eea59beb05d7b269e937/Screenshot_20211202_130808.png)
Agileware Ref: CIVICRM-1899https://lab.civicrm.org/dev/core/-/issues/2965APIv4 Explorer, displays the Field Name as the selectable option. However, th...2023-10-23T05:03:22Zjustinfreeman (Agileware)APIv4 Explorer, displays the Field Name as the selectable option. However, the Field Label only is visible in the Custom Fields UI and may be different to the Field NameAPIv4 Explorer, displays the Field Name as the selectable option. However, the Field Label only is visible in the Custom Fields UI and may be different to the Field Name.
SearchKit on the other hand does use the Field Label to list the ...APIv4 Explorer, displays the Field Name as the selectable option. However, the Field Label only is visible in the Custom Fields UI and may be different to the Field Name.
SearchKit on the other hand does use the Field Label to list the available fields.
This can make it very difficult to use the APIv4 Explorer to select the correct fields if:
1. There are a large number of fields, and
2. The Field Name differs from the Field Label
This is a very common occurrence on older CiviCRM sites which tend to have fields that have been renamed over time and have a lot of old fields still active on the site.
My preference would be to always list the Field Label when displaying fields in the APIv4 Explorer.
![Screenshot_20211123_084139](/uploads/b71225e09f100a8e78d2c321a05704d1/Screenshot_20211123_084139.png)
Agileware Ref: CIVICRM-1890https://lab.civicrm.org/dev/core/-/issues/2940Remove requirement to have a bounce email account setup2022-09-30T20:45:40ZseamusleeRemove requirement to have a bounce email account setupWith some more modern SMTP providers bounce email accounts may not be necessary for CiviMail Bounces to work because e.g. with Amazon SES / Sendgrid you can get the bounces processed via webhooks instead.
However within the code base we...With some more modern SMTP providers bounce email accounts may not be necessary for CiviMail Bounces to work because e.g. with Amazon SES / Sendgrid you can get the bounces processed via webhooks instead.
However within the code base we assume we need a bounce account to generate stuff like [no-reply email address](https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/BAO/Domain.php#L364) or when we are getting [message ids](https://github.com/civicrm/civicrm-core/blob/513f6a469927c5dc5a41905feb7c3cff2cf15f0b/CRM/Campaign/BAO/Petition.php#L548)
It would be nice to get rid of this reliance on it being the default mail account which is the bounce processing one https://github.com/civicrm/civicrm-core/blob/513f6a469927c5dc5a41905feb7c3cff2cf15f0b/CRM/Core/BAO/MailSettings.php#L59
cc @JoeMurrayMonish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/2938Proposal - add created_date, modified_date to civicrm_relationship2022-11-25T02:00:54ZeileenProposal - add created_date, modified_date to civicrm_relationshipIt can be useful to know when a relationship was created in CiviCRM (as opposed to when it started)It can be useful to know when a relationship was created in CiviCRM (as opposed to when it started)https://lab.civicrm.org/dev/core/-/issues/2934Searchkit feedback2023-10-10T05:03:21ZeileenSearchkit feedbackRequest for COUNTIF() in the transforms - ie count if field value > xRequest for COUNTIF() in the transforms - ie count if field value > xhttps://lab.civicrm.org/dev/core/-/issues/2931Deadlocks on acl_contact_cache table2022-07-16T20:22:41ZandyburnsDeadlocks on acl_contact_cache tableWe are seeing the acl_contact_cache table locking up our entire system. The query has been seen running for over 6000 seconds on multiple occasions. Slowly, queries stack up behind this and other requests by other users to use acl_contac...We are seeing the acl_contact_cache table locking up our entire system. The query has been seen running for over 6000 seconds on multiple occasions. Slowly, queries stack up behind this and other requests by other users to use acl_contact_cache stack up as well.
We had not previously set the max_execution_time on the database and that has now been set to 450 on the database, which prevents the long-running query from tangling up the entire system. However, the conflict appears to happen when two users are doing an operation that tries to populate the acl_contact_cache table at the same time.
Our database has almost 900,000 contacts, and almost 500,000 activities logged. So a user goes to search > find activities and leaves conditions blank, and this query occurs:
```
SELECT count(DISTINCT contact_a.id) as rowCount FROM civicrm_contact contact_a LEFT JOIN civicrm_activity_contact
ON ( civicrm_activity_contact.contact_id = contact_a.id ) LEFT JOIN civicrm_activity
ON ( civicrm_activity.id = civicrm_activity_contact.activity_id
AND civicrm_activity.is_deleted = 0 AND civicrm_activity.is_current_revision = 1 ) INNER JOIN civicrm_contact
ON ( civicrm_activity_contact.contact_id = civicrm_contact.id and civicrm_contact.is_deleted != 1 ) INNER JOIN civicrm_acl_contact_cache aclContactCache ON contact_a.id = aclContactCache.contact_id WHERE ( ( civicrm_activity.activity_type_id = 101 AND civicrm_activity.status_id = 9 ) OR ( civicrm_activity.activity_type_id IN (2,3,1,104,109,78,76,85,93,4,5,6,7,8,12,17,19,22,34,40,35,36,37,38,39,44,45,46,47,48,51,58,54,52,80,81,84,94,96,97,99,100,101,103,105,106) ) ) AND aclContactCache.user_id = 26114 AND contact_a.is_deleted = 0 AND (civicrm_activity.activity_type_id IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 17, 19, 22, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 54, 58, 76, 77, 78, 80, 81, 82, 84, 85, 86, 87, 93, 94, 96, 97, 99, 100, 101, 102, 103, 104, 105, 106, 107, 109) AND civicrm_activity.`id` IN (SELECT `id` FROM `civicrm_activity` WHERE id IN (SELECT activity_id FROM civicrm_activity_contact WHERE contact_id IN (SELECT contact_id FROM civicrm_acl_contact_cache WHERE user_id = 26114) AND contact_id IN (SELECT `id` FROM `civicrm_contact` WHERE is_deleted != 1))));
```
6000 seconds later, the database is hung, with about 100 queries stuck behind it. Manually killing all processes that are trying to use the acl_contact_cache table will clear out the backlog of other queries.
The table is transitory, as after getting the results there is a truncate almost immediately after.
One proposed solution could be to individualize the acl_contact_cache table to create for each user, eg. acl_contact_cache_{contact_id} … This would allow the acl_contact_cache_XXX table to get locked and not adversely affect every other user on the system.
The system should do a check upon my first search and see if my acl_contact_cache_XXX table is built, if it is not, build it upon all the contacts I have access to (regardless of the particular search).
Additionally the acl_contact_cache_XXX table could persist for a pre-defined period of time (e.g. 1 hour) so it doesn't have to search everyone in the db each time. Same concept for civicrm_group_contact_cache. Could have major performance benefits.
Also finding a way to keep this table from being locked would likely require a reimagination of how the table is actually used. Another consideration would be the use of the MEMORY engine for this table, to permit faster population since its contents are transitory anyway
Thoughts?
@eileen @kcristiano
Similar issue on acl_cache table: https://lab.civicrm.org/dev/core/-/issues/1486.