CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-04-18T06:50:39Zhttps://lab.civicrm.org/dev/core/-/issues/4241Refresh custom fields names in "Reuse an existing set"2023-04-18T06:50:39ZkristinecRefresh custom fields names in "Reuse an existing set"Overview
----------------------------------------
When adding a new custom field and selecting "reuse an existing set" of custom fields, the dropdown list for some of the multiple choice option sets do not pick up the custom field set na...Overview
----------------------------------------
When adding a new custom field and selecting "reuse an existing set" of custom fields, the dropdown list for some of the multiple choice option sets do not pick up the custom field set names. For example, you have a multiple choice question called "Educational Status" in Custom Field Set A and the same name "Educational Status" in Custom Field B.
Current behaviour
----------------------------------------
In CiviCRM 5.59, the custom field set labeling are missing for some old custom fields (made before a certain Civi update?).
![Screenshot_2023-04-15_111421](/uploads/dc83d16e1c6fcf8a2e9cce1961260909/Screenshot_2023-04-15_111421.png)
Proposed behaviour
----------------------------------------
The word "refresh" or "reset" the dropdown list probably isn't the right word. But the expected behavior should appear as:
Custom Field Set A:: Educational Status
Custom Field Set B:: Educational Status
P.S. And yes, technically if the options in "Educational Status" are exactly the same, this field set should have been shared rather than exist in two separate custom field sets.https://lab.civicrm.org/dev/core/-/issues/4203add end date to petition2023-03-27T06:52:21Zjamieadd end date to petitionOverview
----------------------------------------
This proposed feature would add a new field to petitions called: End date. If someone tries to sign a petition that has an end date in the past, they would get a friendly error warning t...Overview
----------------------------------------
This proposed feature would add a new field to petitions called: End date. If someone tries to sign a petition that has an end date in the past, they would get a friendly error warning them that the petition is no longer active.
Current behaviour
----------------------------------------
Now, we can create petitions and manually disable them. However there are two problems:
1. If you disable a petition it no longer shows up in advanced search so you can no longer find signatories.
2. It's easy to forget to disable a petition. That means these potentially spam-collecting forms can live on for too long
3.
Proposed behaviour
----------------------------------------
By adding an end date when you create the petition, you can set it up and forget about it.
Comments
----------------------------------------
I'm open to taking a crack at implementing, but wanted to first get feedback to see if this is the best way to go about it or if there are any potential pitfalls.https://lab.civicrm.org/dev/core/-/issues/4194Proposal - remove some fields from dedupe field options2023-03-27T02:42:05ZeileenProposal - remove some fields from dedupe field optionsWhen creating dedupe rules there are some fields in the drop down that don't make sense to me
![image](/uploads/87e8fe511a5e795ce362dd7027705505/image.png)
Specifically
**Irrational (technical term)**
(if these are present the dedupe...When creating dedupe rules there are some fields in the drop down that don't make sense to me
![image](/uploads/87e8fe511a5e795ce362dd7027705505/image.png)
Specifically
**Irrational (technical term)**
(if these are present the dedupe rule functionality won't apply)
- 'id' => 'Contact ID',
- 'external_identifier' => 'External Identifier',
**Dumb (technical term)**
- Contact.is_deceased
- 'addressee_id' => 'Addressee',
- 'addressee_custom' => 'Addressee Custom',
- 'do_not_email' => 'Do Not Email',
- 'do_not_mail' => 'Do Not Mail',
- 'do_not_phone' => 'Do Not Phone',
- 'do_not_sms' => 'Do Not Sms',
- 'do_not_trade' => 'Do Not Trade',
- 'email_greeting_id' => 'Email Greeting',
- 'email_greeting_custom' => 'Email Greeting Custom',
- 'external_identifier' => 'External Identifier',
- 'image_URL' => 'Image Url',
- 'is_opt_out' => 'No Bulk Emails (User Opt Out)',
- 'postal_greeting_id' => 'Postal Greeting',
- 'postal_greeting_custom' => 'Postal Greeting Custom',
- 'preferred_communication_method' => 'Preferred Communication Method',
- 'communication_style_id' => 'Communication Style',
- 'signature_html' => 'Signature Html',
- 'signature_text' => 'Signature Text',
**Marginal (laymans' term) **
- 'geo_code_1' => 'Latitude',
- 'geo_code_2' => 'Longitude',
- 'master_id' => 'Master Address ID',
<details>
<summary>Full list - excluding custom fields</summary>
```
/**
* Get the list of supportedFields to test against.
*
* This is a statically maintained (in this test list).
*
*/
public function getSupportedFields() {
return [
'civicrm_address' =>
[
'name' => 'Address Name',
'city' => 'City',
'country_id' => 'Country',
'county_id' => 'County',
'geo_code_1' => 'Latitude',
'geo_code_2' => 'Longitude',
'master_id' => 'Master Address ID',
'postal_code' => 'Postal Code',
'postal_code_suffix' => 'Postal Code Suffix',
'state_province_id' => 'State',
'street_address' => 'Street Address',
'supplemental_address_1' => 'Supplemental Address 1',
'supplemental_address_2' => 'Supplemental Address 2',
'supplemental_address_3' => 'Supplemental Address 3',
],
'civicrm_contact' =>
[
'addressee_id' => 'Addressee',
'addressee_custom' => 'Addressee Custom',
'id' => 'Contact ID',
'source' => 'Contact Source',
'contact_sub_type' => 'Contact Subtype',
'do_not_email' => 'Do Not Email',
'do_not_mail' => 'Do Not Mail',
'do_not_phone' => 'Do Not Phone',
'do_not_sms' => 'Do Not Sms',
'do_not_trade' => 'Do Not Trade',
'email_greeting_id' => 'Email Greeting',
'email_greeting_custom' => 'Email Greeting Custom',
'external_identifier' => 'External Identifier',
'image_URL' => 'Image Url',
'legal_identifier' => 'Legal Identifier',
'legal_name' => 'Legal Name',
'nick_name' => 'Nickname',
'is_opt_out' => 'No Bulk Emails (User Opt Out)',
'organization_name' => 'Organization Name',
'postal_greeting_id' => 'Postal Greeting',
'postal_greeting_custom' => 'Postal Greeting Custom',
'preferred_communication_method' => 'Preferred Communication Method',
'preferred_language' => 'Preferred Language',
'sic_code' => 'Sic Code',
'user_unique_id' => 'Unique ID (OpenID)',
'sort_name' => 'Sort Name',
'communication_style_id' => 'Communication Style',
],
'civicrm_email' =>
[
'email' => 'Email',
'signature_html' => 'Signature Html',
'signature_text' => 'Signature Text',
],
'civicrm_im' =>
[
'name' => 'IM Screen Name',
],
'civicrm_note' =>
[
'note' => 'Note',
],
'civicrm_openid' =>
[
'openid' => 'OpenID',
],
'civicrm_phone' =>
[
'phone_numeric' => 'Phone',
'phone_ext' => 'Phone Extension',
],
'civicrm_website' =>
[
'url' => 'Website',
],
];
}
```
</details>
'sic_code' => 'Sic Code',
'user_unique_id' => 'Unique ID (OpenID)',https://lab.civicrm.org/dev/core/-/issues/4192Improvement: Required? when creating Custom Field2023-03-23T08:14:19ZStoobImprovement: Required? when creating Custom FieldAfter many years in my position working with clients, and even explicit training on this issue, clients continue to flummox themselves by not reading the 'help text' and clicking **Required? [ ]** in a Custom Data field ... when that in ...After many years in my position working with clients, and even explicit training on this issue, clients continue to flummox themselves by not reading the 'help text' and clicking **Required? [ ]** in a Custom Data field ... when that in fact isn't their intent.
Proposal Pt 1: Expanding the caution to the label itself like so: **[ ] Always require a value?**
Proposal Pt 2: Clarifying the help text like so. **This forces a value in this field to create or edit this type of record. To make a field required on a specific form, use a Profile.**
In a comparison other software:
Salesforce:
`[ ] Always require a value in this field in order to save a record`
Network for Good:
`[ ] Required?` same as Civi but I like their help text a bit better than Civi's `If "Yes" this custom field will be required when manually adding/editing contacts.`https://lab.civicrm.org/dev/core/-/issues/4169Let "Number" and "Money"-type custom fields be nullable2023-03-13T07:39:11ZnoahLet "Number" and "Money"-type custom fields be nullableOverview
----------------------------------------
There are many times when NULL would be a meaningful/useful value in a custom field of type "Number" (float) or "Money" (decimal). However, it is not currently possible, via the form laye...Overview
----------------------------------------
There are many times when NULL would be a meaningful/useful value in a custom field of type "Number" (float) or "Money" (decimal). However, it is not currently possible, via the form layer, to set these fields to NULL. Blank values are currently turned into zero.
Example use-case
----------------------------------------
Let's say we need to track the elevation (meters above or below sea level) of the addresses in our database.
1. Create a custom field "Geographic Elevation" of type "Number", extending the Address entity. Leave the "Default value" blank.
1. Go to a contact and edit one of their addresses, or create a new one.
1. On the create/edit form, leave the "Geographic Elevation" field blank -- because let's say we don't know the elevation at this address.
1. Submit the form and view the saved value.
Current behaviour
----------------------------------------
"Geographic Elevation" is set to zero. That means sea level. But that's not correct -- we actually don't know the elevation.
Proposed behaviour
----------------------------------------
When the field submitted with a blank value (empty string), the database field should be set to NULL, and displayed as blank.
Comments
----------------------------------------
NULL is also a meaningful value for Money fields. Say we have a field on individuals called "Net Worth". Zero (the person is destitute) and NULL (we don't know how much money they have) are quite different.
It _is_ possible to set the field to NULL through the API, but only by passing the string 'null' (due to a PEAR DB limitation).
<details><summary>Show API example</summary>
````php
// NULL will be saved as 0
Civi\Api4\Address::update()
->addValue('Geography.Elevation', NULL)
->addWhere('id', '=', 27)
->execute();
$address = Civi\Api4\Address::get()
->addSelect('Geography.Elevation')
->addWhere('id', '=', 27)
->execute()->single();
// [
// 'id' => 27,
// 'Geography.Elevation' => 0,
// ]
// but 'null' will be saved as NULL
Civi\Api4\Address::update()
->addValue('Geography.Elevation', 'null')
->addWhere('id', '=', 27)
->execute();
$address = Civi\Api4\Address::get()
->addSelect('Geography.Elevation')
->addWhere('id', '=', 27)
->execute()->single();
// [
// 'id' => 27,
// 'Geography.Elevation' => null,
// ]
````
</details>https://lab.civicrm.org/dev/core/-/issues/2669Target contact missing for automatically generated case activities2023-02-23T16:00:45ZAndreasandreas.howiller@civiservice.deTarget contact missing for automatically generated case activitiesOverview
----------------------------------------
Some actions on cases leave automatically generated activities in the case activity log, but have no target contacts. Examples: Change case title, change case tags. As a result, they cann...Overview
----------------------------------------
Some actions on cases leave automatically generated activities in the case activity log, but have no target contacts. Examples: Change case title, change case tags. As a result, they cannot be edited through the user interface. While this is a minor issue, some users may want to leave notes about the activity (for example, why they changed a case title) and therefore need to edit it.
Reproduction steps
----------------------------------------
1. Open any case in CiviCRM and change the case title.
2. Tap edit for the corresponding activity in the activity timeline.
3. Tap save.
Current behaviour
----------------------------------------
The dialog will freeze while the CiviCRM logo keeps turning. A fatal error "DB Error: constraint violation" will be filed in the logs.
If you add any contact as a target contact in the database or in the edit screen the error won't appear anymore.
What is confusing, too: When editing the activity the client contact already wrongly shows up as target contact in this dialog and if you'd like to change it there you need to search the contact (the link to choose the client contact below the search field won't do the change here).
Expected behaviour
----------------------------------------
The Client contact is filed as target contact by default and connected problems disappear. E.g. editable activities can be saved after edit through UI now.
Environment information
----------------------------------------
* __CiviCRM:__ _5.35.2_ and _5.38.0_
* __CMS:__ _WordPress 5.7.2_https://lab.civicrm.org/dev/core/-/issues/4136Saving Repeating Events causes Registration date to be copied and/or overwritten2023-02-20T06:29:43ZshaneonabikeSaving Repeating Events causes Registration date to be copied and/or overwrittenOverview
----------------------------------------
Repeating Events are amazing for structuring recurring events. For a client, we had about 12 events scheduled and I wanted to add a new payment processor to those events. Upon adding tha...Overview
----------------------------------------
Repeating Events are amazing for structuring recurring events. For a client, we had about 12 events scheduled and I wanted to add a new payment processor to those events. Upon adding that to the Fees , and applying it to all the Events it changed all the ```Registration dates``` to match the first event :face_palm:
Reproduction steps
----------------------------------------
1. Create an Event
1. Set some fees and one payment processor
1. Set a Start date and Registration date a week before the start date
1. Set the repeat tab to ```Repeats every```: 1 month on the Third Thursday
1. Set an End after 11 occurrences
1. Save the Event choosing ```Every Event``` in the popup
1. Edit one of the Events
1. Check off another payment processor
1. Save the Event
1. The Registration date has changed on all the events
Expected behaviour
----------------------------------------
Basically, I was thinking that either A) we detect this is a modification and don't propagate OR B) we just don't propagate that field since it doesn't seem to make sense (at least to me) to do that on repeating events.https://lab.civicrm.org/dev/core/-/issues/2410Image custom field type2023-01-30T20:19:36ZMichael McAndrewImage custom field typeOverview
----------------------------------------
We want to add a custom field type of image that can be used to store images as part of a CiviCRM entity.
In terms of how this might be different to the file field type, I think the main...Overview
----------------------------------------
We want to add a custom field type of image that can be used to store images as part of a CiviCRM entity.
In terms of how this might be different to the file field type, I think the main consideration is how these fields would be displayed (resizing, thumbnails, etc.)
Example use-case
----------------------------------------
1. Store (one or more) images related to a contact as part of the contact record.
2. *** feel free to add more***
Current behaviour
----------------------------------------
At the moment, there is a single core field for a contact image (e.g. a headshot of an indiviual or a company logo). No other images can be added. Images can be added as files but are not displayed as one would expect an image to be displayed.
Proposed behaviour
----------------------------------------
People should be able to define image fields and upload images to them. The original should be retained and resized images should also be created (e.g. for thumbnails).
(Wireframes, mockups and more thinking required.)
Comments
----------------------------------------
Some relevant content (previous workarounds, etc.):
- https://drupal.stackexchange.com/questions/220599/how-to-add-image-field-in-civicrm
- https://civicrm.stackexchange.com/questions/15900/how-to-add-image-field-in-civicrmhttps://lab.civicrm.org/dev/core/-/issues/4098Feature Request: Ability to hide FormBuilder Search Forms when there are no r...2023-01-30T08:30:41ZbrienneFeature Request: Ability to hide FormBuilder Search Forms when there are no results foundOverview
----------------------------------------
By the combined powers of SearchKit and FormBuilder, you can create Search Forms and display them on the Contact Summary Page. While you can limit the type of contact on which to display ...Overview
----------------------------------------
By the combined powers of SearchKit and FormBuilder, you can create Search Forms and display them on the Contact Summary Page. While you can limit the type of contact on which to display it, i.e. Organization or Individual, it is currently not possible to hide the display when there are no results returned from the SearchKit search.
Example use-case
----------------------------------------
Creating a display block on Organizations to display the "Primary Contact". If the Organization has a primary contact, then the user would want to see that displayed in the block. If they do not have a primary contact, then a user may want to hide that display altogether.
1. Make a SearchKit search with the following criteria:
* Search for *Contacts*
* With (required) *Contact Related Contacts* if Relationship to contact = *Primary Contact of*
* Contact Type = *Organization*
1. Make a Table display for that SearchKit and configure as desired.
2. Make a Form from that display, check the *Add to Contact Summary Page* option, and select **Organization** next to the *For* label.
Current behaviour
----------------------------------------
Currently, it is not possible to hide the display block when there are no results found. Instead, you get a bright blue box on the Contact Summary Page that states "None found".
![Selection_202](/uploads/d03094d0e269f5e8529084f9e3e91ca6/Selection_202.png)
Proposed behaviour
----------------------------------------
There should a variety of options for users to configure how these blocks are displayed/not displayed when there are no options (such as the "no results options with Drupal Views).
For example, there could be a checkbox on the FormBuilder section that adds the display to the Contact Summary Page that gives the option to hide the display when there are no results. This type of checkbox could also be on the SearchKit display options.
![Selection_204](/uploads/cf167a5469addcff5093897b850b790e/Selection_204.png)
![Selection_205](/uploads/004f81cdb74c390c5568a25cfec1ec5c/Selection_205.png)
Comments
----------------------------------------
This is an unfunded feature request of suggestions for UX improvements.https://lab.civicrm.org/dev/core/-/issues/4073Problem with navigation keys stored in civicrm_settings table / proposal to m...2023-01-12T09:58:57ZJKingsnorthProblem with navigation keys stored in civicrm_settings table / proposal to move itWe've had a huge bloat in our log_civicrm_settings logging table - millions of rows. The culprit is the navigation key, which is stored in civicrm_settings with a name of 'navigation' for many contact IDs.
Every time caches are flushed,...We've had a huge bloat in our log_civicrm_settings logging table - millions of rows. The culprit is the navigation key, which is stored in civicrm_settings with a name of 'navigation' for many contact IDs.
Every time caches are flushed, the navigation cache strings are 'NULLed' by `\CRM_Core_Config::clearDBCache();` and then rewritten by
`CRM_Core_BAO_Navigation::resetNavigation();`
In practice, if I truncate log_civicrm_settings, and then run I get:
- 2785 distinct contact IDs in log_civicrm_settings
- 39270 new entries for 'navigation' in log_civicrm_settings!
My contact ID's navigation value is first NULLed and updated 13 times to different values with the same log_conn_id ! Multiply that by our 4 domains, and that's 56 entries per contact per cache flush...
---
- [ ] 1 - Why is it being updated so many times? That's a performance issue even without logging.
- [ ] 2 - Can we move the 'navigation' key out of civicrm_settings, and into a cache table? Probably its own cache table, like civicrm_navigation_cachehttps://lab.civicrm.org/dev/core/-/issues/4033Proposal: Add optional separate mailing label format for organizations2023-01-11T23:31:32ZlarsssandergreenProposal: Add optional separate mailing label format for organizationsOften, I find myself editing the mailing label format when making mailing labels for organizations. In this case, we want Addressee and Name (the name of the person at the org plus the name of the org on the next line), while for individ...Often, I find myself editing the mailing label format when making mailing labels for organizations. In this case, we want Addressee and Name (the name of the person at the org plus the name of the org on the next line), while for individuals we just want the Addressee (otherwise, it would say their name twice). I imagine we aren't alone in this.
In order to avoid this hassle, it would make sense to have an option to add a second mailing label format for organizations. If there is a format specified in this field, it would be used for organizations. If this field is left blank, the default label format would be used for organizations. I think that would keep it simple and wouldn't change anything unless someone intentionally added an organizational label format. We could also add a format for households if desired - would that be valuable to anyone?https://lab.civicrm.org/dev/core/-/issues/2559Cannot get Auth Code in Oauth2 from Microsoft Azure Application2023-01-10T14:28:11ZspalmstromCannot get Auth Code in Oauth2 from Microsoft Azure ApplicationOverview
----------------------------------------
You cannot get an Auth Code in Oath2 from a single-tenant Microsoft Azure Application because the access token string is
```
https://login.microsoftonline.com/common/oauth2/v2.0/token
``...Overview
----------------------------------------
You cannot get an Auth Code in Oath2 from a single-tenant Microsoft Azure Application because the access token string is
```
https://login.microsoftonline.com/common/oauth2/v2.0/token
```
when it should be:
```
https://login.microsoftonline.com/<tenant ID>/oauth2/v2.0/token
```
Reproduction steps
----------------------------------------
1. Click on Admin -> Oauth2 Administration
1. Select Microsoft Exchange Online
1. Click on Add token and enter an MS account
Current behaviour
----------------------------------------
```
AADSTS50194: Application '226037fb-d13a-4f81-ba32-561601248bea'(MissionAssist Mail) is not configured as a multi-tenant application. Usage of the /common endpoint is not supported for such applications created after '10/15/2018'. Use a tenant-specific endpoint or configure the application to be multi-tenant.
```
Expected behaviour
----------------------------------------
A token should be added.
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ _Edge_ but probably not relevant
* __CiviCRM:__ _5.36.1_ <!-- If this problem relates to an upgrade, then specify both old and new versions -->
* __PHP:__ _7.4.16__ but probably not relevant
* __CMS:__ _Drupal 9.1.7_ but probably not relevant.
* __Database:__ _MySQL 8.0.24_ but probably not relevant
* __Web Server:__ _IIS_ but probably not relevant.
Comments
----------------------------------------
It would be good if the setup could prompt for the tenant ID>5.59.0https://lab.civicrm.org/dev/core/-/issues/2481Discussion drop support for php 7.2 and mysql 5.62023-01-04T18:01:36ZeileenDiscussion drop support for php 7.2 and mysql 5.6We have been dropping support for EOL php & mysql on ESR releases because that gives sites the option of delaying for another 6 months for just $120. So far we are not aware of any sites opting for the ESR for that reason but we are crea...We have been dropping support for EOL php & mysql on ESR releases because that gives sites the option of delaying for another 6 months for just $120. So far we are not aware of any sites opting for the ESR for that reason but we are creating the option.
With 5.39 coming up we should consider whether we want to drop support for
php 7.2 (EOL Dec 2020)
mysql 5.6 (EOL Feb 2021)
Note that historically supporting older versions of mysql has been low effort although some features are not supported. However, it might be holding us back from using new features
Supporting older versions of php has historically been more problematic as it has been at the expense of supporting newer versions. In particular we often find packages don't support latest and EOL versions.
So dropping support for older php versions is more helpful than older mysql versions.
Last ESR we considered dropping support for php 7.1 and mysql 5.6. We did the former and not the latter. At the time the stats were:
mysql 5.6 : 805
php 7.1 : 658.
Maria DB 10.1 : 943
Currently they are
mysql 5.6 : 603
php 7.2 : 2846
Maria DB 10.1 : 526
Based on those numbers I think it's clearly too early for php 7.2 and I would be OK targetting both for the NEXT ESR (5.45)https://lab.civicrm.org/dev/core/-/issues/4034CRM Builder - Profiles allow multiple Formatting Free HTML fields in forms2023-01-03T14:46:14ZJonny ToomeyCRM Builder - Profiles allow multiple Formatting Free HTML fields in formsOverview
----------------------------------------
When building a conribution form, or altering the profile for a form, the Javascript rightly detects duplciate fields and prevents saving. Would like to make an exception for Free HTML fi...Overview
----------------------------------------
When building a conribution form, or altering the profile for a form, the Javascript rightly detects duplciate fields and prevents saving. Would like to make an exception for Free HTML fields (could be expanded to anything from the Formatting section if that section should expand)
Current workaround
----------------------------------------
Saving after each change gets around this for now
Proposed behaviour
----------------------------------------
alter the markDuplicates() method so that is_duplicate class is only applied when the ufFieldModel does not have the label 'Free HTML'. This will allow multiple Free HTML fields for formatting forms
Proposed solution
----------------------------------------
civicrm/js/model/crm.uf.js
![markDuplicates](/uploads/fe6b7f9c7da464418d2bd6a27f75639a/markDuplicates.PNG)https://lab.civicrm.org/dev/core/-/issues/4032civicrm_group_contact has a email_id field, but this does not appear to be used2022-12-20T10:04:29ZBradley Taylorcivicrm_group_contact has a email_id field, but this does not appear to be usedOverview
----------------------------------------
I'm currently working with an organisation who are moving to CiviCRM. They have contacts, who frequently have multiple employers, typically with a work email address for each employment. ...Overview
----------------------------------------
I'm currently working with an organisation who are moving to CiviCRM. They have contacts, who frequently have multiple employers, typically with a work email address for each employment. When signing up to email groups, it's often useful for these contacts to be able to control which group subscription goes to which email address.
For example:
We have a contact, Bob.
Bob is employed by CompanyA and CompanyB, and has the email addresses bob@company-a.com and bob@company-b.com. bob@company-a.com is the primary email address.
Bob signs up for two three mailing lists, but wants two to go to his primary email address and one to go to his secondary email address.
Current behaviour
----------------------------------------
This isn't something which is possible through the current CiviCRM UI, which is fine (I can't imagine this use-case is that common).
However, looking at the database table `civicrm_group_contact` it has a field `email_id`, with the comment `Optional email to associate with this membership`. This field is exposed in the CiviCRM API and SearchKit.
As far as I can tell, it appears that this `email_id` field is not actually used by any logic within CiviCRM core. It seems to be at least 10 years old (dating back to the pre-Git days), and so it's possible the `email_id` was introduced as part of a feature that was never finished.
Expected behaviour
----------------------------------------
I had assumed that when `email_id` was set, this email address would be used as the email address to actually be emailed, when sending a bulk mailing to a group via the CiviMail component. This unfortunately does not seem to be the case.
I think if the field exists, it should be used in this way. If not it should probably be deprecated and removed from core.
Currently it is possible to set the "Location Type" and "Selection Method" when selecting a group as part of the new mailing form. If a location type is set I would expect it to take precedence over the configured `email_id` (although this detail could be debated either way).
Whilst it might be nice if there was an associated UI to select the email address, I don't think any UI changes are required to make the `email_id` field useful - if CiviCRM simply uses the value (as set via plugin code or the API) that would be enough to enable the use case detailed above.
Comments
----------------------------------------
It's possible that I'm completely misunderstanding what the `email_id` field is for, and it is actually used - if so I'd love to know the details from someone closer to this area of the core code!https://lab.civicrm.org/dev/core/-/issues/4041Should we hard-code messages that event registration cancellations are not re...2022-12-20T10:01:32ZlarsssandergreenShould we hard-code messages that event registration cancellations are not refundable?Self-service cancellation of event registrations does not support refunds, so there are at least two places where the registrants are told that there are no refunds (in the registration confirmation email if self-service cancellation is ...Self-service cancellation of event registrations does not support refunds, so there are at least two places where the registrants are told that there are no refunds (in the registration confirmation email if self-service cancellation is enabled and when they actually try to cancel, though this has been broken since 5.43).
I wonder if it makes sense for CiviCRM to put this expectation that there are no refunds into code. Even though refunds are not supported via the self-service form, organizations may have policies that are different and may manually process refunds in some cases. I think it would make more sense to simply not say anything about refunds and allow organizations to define their own policies in the event registration.
So my proposal: Remove the warning about no refunds from self-service cancellation (which doesn't currently work anyways), remove the text about no refunds from event registration confirmation templates and remove the help text that mentions no refunds from the self-service cancellation option on the event registration set up.
It's far from ideal either way, but I think leaving it open is better than specifying a policy that may not work for all.https://lab.civicrm.org/dev/core/-/issues/3934Settings - Deprecate "serialize" flag2022-12-16T01:05:12ZtottenSettings - Deprecate "serialize" flag# Background: Basic settings
Settings are "key-value" pairs. The "value" part allows type `mixed` (*ie `string` or `int` or `float` or `array` or whatever*).
Support for `mixed` is consistent between `Civi::settings()` and `$civicrm_se...# Background: Basic settings
Settings are "key-value" pairs. The "value" part allows type `mixed` (*ie `string` or `int` or `float` or `array` or whatever*).
Support for `mixed` is consistent between `Civi::settings()` and `$civicrm_setting`.
```php
// Consistent access to a scalar
Civi::setting()->set('my_string', 'abc');
$civicrm_setting['domain']['my_string'] = 'abc';
assert Civi::setting()->get('my_string') === 'abc';
```
```php
// Consistent access to an array
Civi::setting()->set('my_array', [1,2,3]);
$civicrm_setting['domain']['my_array'] = [1,2,3];
assert Civi::setting()->get('my_string') === [1,2,3];
```
Under the hood, values are stored in MySQL using `serialize()`. You can see this in both recent and ancient code:
* 5.54's `setDb()`: https://github.com/civicrm/civicrm-core/blob/5.54/Civi/Core/SettingsBag.php#L413
* 5.54's `loadValues()`: https://github.com/civicrm/civicrm-core/blob/5.54/Civi/Core/SettingsBag.php#L137
* 4.3's `setItem()`: https://github.com/civicrm/civicrm-core/blob/4.3/CRM/Core/BAO/Setting.php#L325
* 4.3's `getItems()`: https://github.com/civicrm/civicrm-core/blob/4.3/CRM/Core/BAO/Setting.php#L216
# Background: `serialize` metadata
There is an additional option in `*.setting.php` metadata called `serialize`. Superficially, this resembles `serialize` metadata for DAO fields.
This option is defined for 8 settings: `contact_view_options`, `contact_edit_options`, `advanced_search_options`, `user_dashboard_options`, `address_options`, `contact_autocomplete_options`, `contact_reference_options`, and `quicksearch_options`.
In all cases, the setting has a serialization format of `SERIALIZE_SEPARATOR_BOOKEND`.
APIv4's `Setting` is a wrapper for `Civi::settings()`. Notably, it interprets the `serialize` metadata and applies an (additional/secondary) serialization.
# What's good about this?
It makes it easier for REST/AJAX agents to read and write those 8 historical settings (*the pre-existing settings with the annoying BOOKEND format*).
At the time of introduction, it probably had no effects on compatibility and required no upgrade steps.
# What's bad about this?
It encourages people to add unnecessary serialization layers. If you are writing a _new_ setting with structured data, then you'll see the `serialize` metadata and think: "I should add that!" Here are some docs that suggest `JSON` for a setting:
https://docs.civicrm.org/dev/en/latest/framework/setting/#settings-definition
But you shouldn't do that on a new setting! Here's why:
* __It adds complexity -- double-encoding.__ Every read and write of the data goes through both `serialize()` and some additional serialization method.
* __It adds complexity -- conflicted APIs.__ The setting can be viewed through various programmatic interfaces (e.g. local PHP array `$civicrm_setting`; local PHP method `Civi::setting()->get()`; e.g. remote REST method `Setting.get`). Some APIs present an `array` -- but others present an encoded string (like `^1^2^3^`). There is no decent reason why `$civicrm_setting`, `Civi::setting()`, and `Setting.get` should present it differently.
* __It distracts from the real data-modeling problem of serialized fields.__ We generally do simple settings (strings/ints) because they're easier. Complex settings (like `mailing_backend`) are possible, but we don't do them as much - because the tooling/validation/etc is not as good. Intuitively, complex settings could work better if we added more metadata. _But this is the wrong metadata._ Switching `mailing_backend` between `serialize`/JSON/CSV/YAML/etc doesn't make this any better. What you need is a sharper description of the data inside of that serialized blob.
* __It becomes overall harder to change.__ Perhaps settings should be stored with `json_encode()` instead of `serialize()`? `serialize()` has additional kinds of vulnerabilities. JSON has better support in MySQL. Plot out the path for how to migrate the entirety of `civicrm_setting.value` -- and compare that to switching serialization on a setting-by-setting basis. It's easier to change if individual settings are _not_ committed to specific serializers.
# What to do
Simplest thing:
* Keep the mechanism
* Present it as deprecatedhttps://lab.civicrm.org/dev/core/-/issues/3645Eventually google will require OAUTH for bounce processing and may require it...2022-12-08T20:55:07ZDaveDEventually google will require OAUTH for bounce processing and may require it for outbound SMTP through gmail serversAs noted on [stackexchange](https://civicrm.stackexchange.com/questions/34142/sending-mail-via-gmail-google-announce-username-password-authentication-will-b), Google sent out an email outlining plans to require OAUTH and discontinue supp...As noted on [stackexchange](https://civicrm.stackexchange.com/questions/34142/sending-mail-via-gmail-google-announce-username-password-authentication-will-b), Google sent out an email outlining plans to require OAUTH and discontinue support for just username/password.
The way I read it, this is mostly applying to POP/IMAP clients for reading mail, and they are *hinting* that someday they might require it for outbound SMTP, i.e. at Administer - System Settings - Outbound Mail you choose SMTP and have it set to use gmail servers. The relevant quote is
> No change is required for scanners or other devices using simple mail transfer protocol (SMTP) or LSAs to send emails. If you replace your device, look for one that sends email using OAuth.
(LSA means less secure apps, which is google-speak for username/password)
They are targeting June 2020 - Feb 2021 for IMAP etc, but as noted above they have not specifically set a date for changes regarding sending outbound emails.
It occurs to me though that bounce processing and the email processor use POP/IMAP. For the email processor you can work around it by just forwarding it to a non-gmail address and polling that, but bounce processing might be harder depending on setup.https://lab.civicrm.org/dev/core/-/issues/4007Proposal: SearchKit Templates2022-12-05T23:15:19ZcolemanwProposal: SearchKit TemplatesBackground
-------------
Olly is working on a new Extension called [SearchKit Reports](https://lab.civicrm.org/extensions/search_kit_reports) which is a collection of SavedSearches. The idea is to reproduce most of CiviReport using Searc...Background
-------------
Olly is working on a new Extension called [SearchKit Reports](https://lab.civicrm.org/extensions/search_kit_reports) which is a collection of SavedSearches. The idea is to reproduce most of CiviReport using SearchKit, and people can then use those packaged searches to get a jump-start on building their own searches.
Since these searches are intended to be used as templates, why not actually make them templates.
Rationale
-------------
I think this would benefit the SK UI because because so far, the "Packaged Searches" tab contains stuff the average user shouldn't mess with unless they know what they're doing; searches like "Administer Custom Fields" provide critical functionality to various CiviCRM screens that could potentially be broken if messed with.
Now the [SearchKit Reports](https://lab.civicrm.org/extensions/search_kit_reports) extension is introducing a bunch of packaged searches with the opposite intention. They provide *no* functionality out of the box and we *want* the user to mess with them & experiment as much as they want. We also want to encourage a workflow where they don't directly edit the packaged version but save a copy.
Design
-----------
I'm thinking that SavedSearchTemplates would:
1. Appear on their own tab in SearchKit
2. Not be runnable outside the SearchKit Admin UI - clicking on one would pull up a new pre-configured search (the same as clicking the "Clone" button on a SavedSearch)
3. Stored in their own table `civicrm_saved_search_template`
4. Can be created, updated & packaged like regular saved searches
To point 3, I thought about adding an `is_template` column like we do in the `civicrm_event` table, and then thought about what a PITA that column is, and is it really so hard to create a new table? No, not hard at all.https://lab.civicrm.org/dev/core/-/issues/1634Evaluate if any indexed fields are unused2022-12-04T23:07:09ZeileenEvaluate if any indexed fields are unused
**Proposal** (note this is being updated based on discussion & comments may refer to an earlier version)
1. Remove the following columns from the xml from civicrm_activity
* phone_id
* phone_number
* relationship_id
1. Remove th...
**Proposal** (note this is being updated based on discussion & comments may refer to an earlier version)
1. Remove the following columns from the xml from civicrm_activity
* phone_id
* phone_number
* relationship_id
1. Remove the index from the xml on
* medium_id
* is_deleted
1. During upgrade we drop the above columns, if empty.
**Follow ups to consider**
2. There are other columns in the civicrm_activity table that are case specific - we might consider indexing is_current_revision & original_id only when CiviCase is enabled
3. I've been doing some tests on searches and found that searching is faster if I DROP the contribution_status_id index - it might be interesting to test the activity_type_id index although I suspect it has a much greater cardinality & is more useful
**Impact of the above**
1. Data would not be lost but api fields would no longer access those fields
2. Developers who might be using them outside of core could be impacted - we can mitigate by communicating on the dev list & perhaps putting checks & deprecation notices onto sites with data in the fields for a few months before making any changes.
3. DB size would be reduced. Note that empty fields contribute notably to table size IF they are indexed
4. Dev confusion & efficiency is improved by not having unused stuff in core.
**Background**
Obviously that's not something we should rush into so I'll have to ping the dev list etc
Looking at our civicrm_activity table it appears that each index has a base size - of around a half a gig. From there, the index size increases based on how much data is in the table. So an index on an empty field is around 57% of the size of our largest index.
There are 5 fields that are indexed + empty in our database for the civicrm_activity table (
```
Original id used for CiviCase
Medium id
Phone id
relationship_id
is_deleted
```
Plus - is_current_revision is effectively null
So my first question is are these all used in other databases - e.g when civicase is in use.
I couldn't spot references to phone_id and it feels 'wrong' to me anyway as I think you would want to either link to the contact or have a hard reference. I wonder if some of these fields are quietly obsolete?
It's very unsafe to drop core fields. However, I'm pondering dropping the indexes on these fields
@DaveD I'd appreciate your thoughts....