CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2024-03-11T05:03:24Zhttps://lab.civicrm.org/dev/core/-/issues/3692Replace all calls to deprecated method CRM_Core_PseudoConstant::activityType()2024-03-11T05:03:24ZherbdoolReplace all calls to deprecated method CRM_Core_PseudoConstant::activityType()There are calls to `CRM_Core_PseudoConstant::activityType()` (deprecated) in many places such as `$activityTypes = CRM_Core_PseudoConstant::activityType(TRUE, FALSE, FALSE, 'name');` ~~but the actual method accepts no parameters. And cha...There are calls to `CRM_Core_PseudoConstant::activityType()` (deprecated) in many places such as `$activityTypes = CRM_Core_PseudoConstant::activityType(TRUE, FALSE, FALSE, 'name');` ~~but the actual method accepts no parameters. And changes to that method predate the migration from SVN... so some guesswork needs to be done~~.
Update: oops I just noticed it has `func_get_args()` so it is accepting parameters after all.
In some cases it seems that it can be replaced with `CRM_Core_PseudoConstant::getKey('CRM_Activity_BAO_Activity', 'activity_type_id', 'ACTIVITYNAME')` but in other cases it actually does need a list of activity types. And in some cases needs to filter out disabled components, which is what the original method is doing.https://lab.civicrm.org/dev/core/-/issues/3603Undefined index: error-codes in recaptcha_check_answer()2024-02-16T05:03:24Zchris_bluejacUndefined index: error-codes in recaptcha_check_answer()## Overview
When non-authenticated user subscribes to mailing list at /civicrm/mailing/subscribe, checks the "I am not a robot" reCAPTCHA, and clicks subscribe. The subscription request is processed properly, but an error is thrown:
Not...## Overview
When non-authenticated user subscribes to mailing list at /civicrm/mailing/subscribe, checks the "I am not a robot" reCAPTCHA, and clicks subscribe. The subscription request is processed properly, but an error is thrown:
Notice: Undefined index: error-codes in recaptcha_check_answer() (line 159 of /bitnami/drupal/modules/contrib/civicrm/ext/recaptcha/lib/recaptcha/recaptchalib.php).
## Reproduction steps
1. /civicrm/mailing/subscribe
2. complete email address, check relevant mailing list (in this case, "General Newsletter")
3. Check "I'm not a robot" in reCAPTCHA
4. Click "Subscribe"
5. Subscription is processed properly.
6. Error reported on confirmation page.
## Current behaviour
Error message presented.
Functionality of reCAPTCHA and subscription is okay.
~~~
Notice: Undefined index: error-codes in recaptcha_check_answer() (line 159 of /bitnami/drupal/modules/contrib/civicrm/ext/recaptcha/lib/recaptcha/recaptchalib.php).
~~~
## Expected behaviour
No error message
## Environment Information
- CiviCRM: 5.40.2
- PHP: 7.3.27
- CMS: Drupal 7.82
- Database: MySQL 5.7.33
- Web Server: Apache 2.4.46
- reCAPTCHA: Version 5.40.2
![Screenshot_2021-10-31_084608](/uploads/286470768de6dd8bc6a632827856eafd/Screenshot_2021-10-31_084608.png)https://lab.civicrm.org/dev/core/-/issues/3559CiviCRM bounce processing, processed emails are never deleted from the mailbo...2023-09-08T22:34:57Zjustinfreeman (Agileware)CiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little valueCiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little value.
Possible solutions:
1. Once a bounce email has been processed it should be deleted from the mailbox,...CiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little value.
Possible solutions:
1. Once a bounce email has been processed it should be deleted from the mailbox, or
1. A new Scheduled Job process should be implemented which deletes all processed email which is older than 30 days
Relates to https://lab.civicrm.org/dev/mail/-/issues/108
Agileware Ref: CIVICRM-1977https://lab.civicrm.org/dev/core/-/issues/3489Increase field size for mailing bounce type2024-01-29T10:07:13ZKurund JalmiIncrease field size for mailing bounce typeThe database fields in civicrm_mailing_bounce_type are too restrictive. The name field is limited to 24 characters and the description is limited to 255 characters. This isn’t long enough for the name and description provided by some ema...The database fields in civicrm_mailing_bounce_type are too restrictive. The name field is limited to 24 characters and the description is limited to 255 characters. This isn’t long enough for the name and description provided by some email service providers - e.g. AWS SES.
I propose that the name field should be increased to 256 characters (`VARCHAR(256)`) and the description field should be increased to 2048 characters (`VARCHAR(2048)`).https://lab.civicrm.org/dev/core/-/issues/3479make the hook usable by passing the sql2024-02-04T05:03:25Zyashodhamake the hook usable by passing the sqlCurrently, we are not passing the sql in the _alterReportVar_ hook.
`CRM_Utils_Hook::alterReportVar('sql', $this, $this);`
Objective is to make the hook usable by passing the sql instead of redundant params.Currently, we are not passing the sql in the _alterReportVar_ hook.
`CRM_Utils_Hook::alterReportVar('sql', $this, $this);`
Objective is to make the hook usable by passing the sql instead of redundant params.yashodhayashodhahttps://lab.civicrm.org/dev/core/-/issues/3477SearchKit - dashboard tables2023-12-03T16:32:00Zaydunsaidan.saunders@squiffle.ukSearchKit - dashboard tablesOverview
----------------------------------------
This is about how we can produce dashboard-like summary data with SearchKit to replace existing dashboards and make them more customisable.
For example, consider the Cases dashboard:
...Overview
----------------------------------------
This is about how we can produce dashboard-like summary data with SearchKit to replace existing dashboards and make them more customisable.
For example, consider the Cases dashboard:
- the rows are Case Types
- the columns are the values of Case Status
- the table cells are counts of matching cases, linked to another search
Option 1
--------
SearchKit can produce the counts with eg:
![image](/uploads/16d551f0332f28f4f2a5ef8a6e4a1ff4/image.png)
Maybe a new Display type could restructure the data to produce the desired format.
(Note that is should be based off of CaseType rather than Cases so that types with no cases are shown. But CaseType is not currently available.)
Option 2
--------
See https://gist.github.com/aydun/d49197c81205b463ef5cdef38c97b317
The recently added data segmentation functionality adds 'virtual fields' for group-by purposes. Maybe we could add virtual fields based around the idea of the gist to create virtual fields from an option group so an option group of case_status becomes a set of fields `sum(status_id=1), sum(status_id=2)` etc.
However, I don't think SearchKit/Api4 can produce sql like that - but maybe it can!
Both of these would need to accommodate linking the cells to another parameterized search.
Option 3
----------------------------------------
Something else ...https://lab.civicrm.org/dev/core/-/issues/3474Afform - display privacy flag icons2024-03-07T23:35:11Zaydunsaidan.saunders@squiffle.ukAfform - display privacy flag iconsOverview
----------------------------------------
Form builder allows an icon to be selected conditionally when displaying a field. However, the icons that we use for the privacy flags 'do not email', 'do not phone' etc are actually mul...Overview
----------------------------------------
Form builder allows an icon to be selected conditionally when displaying a field. However, the icons that we use for the privacy flags 'do not email', 'do not phone' etc are actually multiple stacked icons and cannot be added to the display.
Example use-case
----------------------------------------
It would be useful (probably recommended) that when displaying an email address (or phone etc) for a contact with the 'do not email' privacy flag that the icon is shown next to the email address.
Current behaviour
----------------------------------------
A single icon can be displayed, but not stacked icons.
Proposed behaviour
----------------------------------------
Be able to display the privacy flag icons.
Comments
----------------------------------------
The privacy icons are added by the smarty function `CRM/Core/Smarty/plugins/function.privacyFlag.php`
Form builder allows using smarty in the rewrite field so I tried a rewrite of:
`{privacyFlag field=do_not_email condition=[do_not_email]} `
but 2 problems:
- the generated html is displayed rather than the icons
- conditional does not work because the `[do_not_email]` is passed as 'Yes' or 'No' instead of 1 or 0.
One approach would be to refactor the stacked icons out of the smarty function and make privacy flag icons a specific option in form builder.
Alternatively the form builder support for icons could be expanded to allow arbitrary stacked icons.https://lab.civicrm.org/dev/core/-/issues/3466On_hold field for phone record2024-02-28T05:03:22Zmagnolia61On_hold field for phone recordOverview
----------------------------------------
I think by design the civicrm_phone table should also have an on_hold field in order to 'block' individual phone numbers from receiving calls and texts.
I think this would help to better...Overview
----------------------------------------
I think by design the civicrm_phone table should also have an on_hold field in order to 'block' individual phone numbers from receiving calls and texts.
I think this would help to better refine the means to comply to privacy regulations.
Current behaviour
----------------------------------------
Only on a contact level do_not_call and do_not_sms are available as options
Proposed behaviour
----------------------------------------
Just like the civicrm_email table individual phone numbers can be 'disabled' from receiving (automatic) texts and calls.
Comments
----------------------------------------
I am not really able to code this but would be able to help think about it and help test.https://lab.civicrm.org/dev/core/-/issues/3455Contact import - do we need a separate 'no match' type of error?2023-01-31T04:27:52ZeileenContact import - do we need a separate 'no match' type of error?Can we fold tracking of 'no_match' contacts into general 'invalid/error' contacts during import - ie in terms of how they are reported on the contact summary screen.
During contact validation (ie when MapField is submitted & Preview for...Can we fold tracking of 'no_match' contacts into general 'invalid/error' contacts during import - ie in terms of how they are reported on the contact summary screen.
During contact validation (ie when MapField is submitted & Preview form is presented the following outcomes are possible:
- Valid = imported / will attempt to import
- Invalid = failed to import
During the contact import the following outcomes are possible...
- Valid = imported / will attempt to import
- Invalid = failed to import
- No match = failed to import in a way that relates to contact types or something
- Unparsed warning = imported but didn't succeed with USPS parsing (does that even work?)
- Duplicates - updated/skipped depending on the option
No Match is returned in a handful of cases which mostly seem to be around contact subtype - although it could be the type of an existing contact doesn't match the relationship or something.
These are in a separate csv which can be downloaded - the text on the summary screen is
"CiviCRM has detected mismatched contact IDs. These records have not been updated"
"You may then correct them, and import the new file with the corrected data"
- which is clear as mud to me.
Is there any advantage to tracking these separately to the main 'invalid/error' rows outputhttps://lab.civicrm.org/dev/core/-/issues/3454Migrate hook_civicrm_caseTypes to hook_civicrm_managed2024-01-26T05:03:29ZtottenMigrate hook_civicrm_caseTypes to hook_civicrm_managedOverview
---------
Since v4.x, `hook_civicrm_caseTypes` has provided a way to register a case-type using a stored file from an extension. `hook_civicrm_managed` was recently expanded (#3429 for 5.50) in a way that appears to provide a s...Overview
---------
Since v4.x, `hook_civicrm_caseTypes` has provided a way to register a case-type using a stored file from an extension. `hook_civicrm_managed` was recently expanded (#3429 for 5.50) in a way that appears to provide a superset of the functionality. During review of @herbdool's https://github.com/civicrm/civicrm-core/pull/23313, @colemanw suggested deprecating `hook_caseTypes` in favor of `hook_managed`.
This issue is to identify the steps/phases of a migration.
Pro/Con
---------
* Reasons to transition `hook_civicrm_caseTypes` => `hook_civicrm_managed`
* They support the same use-case. Having more than one way to do the same thing can be confusing.
* `hook_managed` is more generic. You get more utility/leverage from understanding it.
* `hook_managed` has more options vis-a-vis lifecycle, cleanup, etc.
* Reasons not to transition `hook_civicrm_caseTypes` => `hook_civicrm_managed`
* If it ain't broke, don't fix it.
* `hook_caseTypes` has multiple mappings which appear to work together (ie `CRM_Case_ManagedEntities`: `CaseTypes`,`ActivityTypes`,`RelationshipTypes`).
* This specific application of `hook_managed` is new. Needs more usage to see if it's fit-to-purpose.
Transition Path
---------------
IMHO, the transition would look something like this:
* Near term / Phase 1
* Do some testing around extension lifecycle/upgrades - eg if you upgrade an extension (*getting a new CaseType `definition` that adds, edits, removes some ActivityType*), then... do the interdependent things update properly? What happens if you've edited the `ActivityType` locally or added another `ActivityType`? Do policies like `cleanup=>unmodified` or `cleanup=>unused` actually work? (*I expect these to be somewhat sensible, since each part seems to have satisfactory precedent; but I encourage playing with it during QA, since this is a bit unusual and it's important to the story of "managed CaseType".*) Perhaps publish an example extension that shows that the technique is fit-for-purpose.
* Mid-term / Phase 2
* Add some docs which tell the story of exporting via `civicrm/api4` -- eg an alternative to https://docs.civicrm.org/dev/en/latest/extensions/civix/#generate-case-type
* Try doing a conversion from `xml/case/*.xml` to `*.mgd.php`. Make sure that IDs/FKs (`case_type_id`, `activity_type_id`, etc) are stable.
* Remove https://docs.civicrm.org/dev/en/latest/extensions/civix/#generate-case-type and maybe `civix generate:case-type` (Leave a pointer to the new docs.)
* Update `CRM_Case_ManagedEntities` and/or `mixins/case-xml` to emit a warning. Encourage folks to convert to `mgd`.
* (*Stretch goal*) Add a step to `civix upgrade` to auto-convert `*.xml` to `*.mgd.php` (or at least show a warning about the XML files). (See also: https://github.com/totten/civix/tree/master/upgrades)
* Long-term / Phase 3
* Remove `hook_caseTypes` aka `mixins/case-xml` from core
* Remove `hook_caseTypes` aka `mixins/case-xml` from civixhttps://lab.civicrm.org/dev/core/-/issues/3449Afform: checkboxes should have default false value2022-08-04T17:46:30ZRichAfform: checkboxes should have default false valueOverview
----------------------------------------
If you have a checkbox used, for example, for "is_test" on a contribution search table but without explictly configuring a default value, the UI shows an un-checked checkbox, which *shou...Overview
----------------------------------------
If you have a checkbox used, for example, for "is_test" on a contribution search table but without explictly configuring a default value, the UI shows an un-checked checkbox, which *should* be the visual representation of "is_test = 0" but is actually representing *no filter on is_test*.
After operating the checkbox the UI and the meaning are in-sync.
While this can be solved by the form builder designer setting an explicit default value, there is no logical sense to *not* having a default value when the form element is binary.
Current behaviour
----------------------------------------
If person who built the form forgets / does not know to include a default value on a checkbox, the UI is initially broken, results are misleading.
Proposed behaviour
----------------------------------------
A checkbox should insist on a default value, defaulting to false, so that the UI and the meaning of the filter are always in sync.
Comments
----------------------------------------
https://chat.civicrm.org/civicrm/pl/9613mtpi6j8ypxf879p7eyyz5ahttps://lab.civicrm.org/dev/core/-/issues/3442Allow other services to standardize addresses and spin off CRM_Utils_Address_...2024-01-21T05:03:28ZherbdoolAllow other services to standardize addresses and spin off CRM_Utils_Address_USPS into an extensionOverview
----------------------------------------
Currently `CRM_Core_BAO_Address::fixAddress` is hardcoded to use `CRM_Utils_Address_USPS` alone. It allows an "address_standardization_provider" to be set but it will only recognize USPS...Overview
----------------------------------------
Currently `CRM_Core_BAO_Address::fixAddress` is hardcoded to use `CRM_Utils_Address_USPS` alone. It allows an "address_standardization_provider" to be set but it will only recognize USPS.
Proposed behaviour
----------------------------------------
Allow for other providers and restructure so that USPS is not hardcoded here. USPS could extend an `AddressStandardization` class or something like that. Could use `CRM_Utils_GeocodeProvider` for inspiration.https://lab.civicrm.org/dev/core/-/issues/3437Provide for a custom entity to add a supported field and query to the DedupeR...2022-06-18T16:31:29ZdarrickProvide for a custom entity to add a supported field and query to the DedupeRuleGroup classOverview
----------------------------------------
Using hook_civicrm_dupeQuery a programmer may add to the list of supported fields available in the select list when adding or editing a Rule Group via "civicrm/contact/deduperules".
H...Overview
----------------------------------------
Using hook_civicrm_dupeQuery a programmer may add to the list of supported fields available in the select list when adding or editing a Rule Group via "civicrm/contact/deduperules".
However, when the rule is run and exception is thrown because the function which creates the query [CRM_Dedupe_BAO_DedupeRule::sql()](https://github.com/civicrm/civicrm-core/blob/1062500743cfe49f419dde49a2b06b8e684e7b3c/CRM/Dedupe/BAO/DedupeRule.php#L46) throws an error, as it only supports hard-coded tables.
Therefore, hook_civicrm_dupeQuery with the option 'table' is never called.
If, the above function were to return NULL instead of throwing an error. hook_civicrm_dupeQuery would be called allowing the programmer to return the query required for their supported field.
However, the above produces two further issues. Via [CRM_Dedupe_BAO_DedupeRuleGroup::fillTable()](https://github.com/civicrm/civicrm-core/blob/1062500743cfe49f419dde49a2b06b8e684e7b3c/CRM/Dedupe/BAO/DedupeRuleGroup.php#L187):
noRules will be set to TRUE in the tableQuery() function if the custom supported field is the only field used in the rule.
The noRules affects the type of temp table created and also the threshold query returned:
```
if ($this->params && !$this->noRules) {
$this->temporaryTables['dedupe'] = CRM_Utils_SQL_TempTable::build() ...
```
And the programmer needs to be aware that the table key can only be found via the following search pattern:
```
$patternColumn = '/t1.(\w+)/';
...
preg_match($patternColumn, $query, $matches);
```
Proposed behaviour
----------------------------------------
I propose modifying [CRM_Dedupe_BAO_DedupeRule::sql()](https://github.com/civicrm/civicrm-core/blob/1062500743cfe49f419dde49a2b06b8e684e7b3c/CRM/Dedupe/BAO/DedupeRule.php#L46) so it either magically determines the name of the field in the custom entities table which links to the contact record (i.e. contact_id) and also optionally determines if the entity_type field also needs to be added in the where clause.
The above would allow a custom entity to easily provide their fields for use in User created Dedupe Rule Groups. The sql does a lot of heavy lifting and tableQuery requires a rather specific query.
Comments
----------------------------------------
I don't know of a magical way to find the right contact_id column.
Another option is to split the sql function up so a programmer can call it from the dupeQuery with their own id and entity_table field name.
dedupeIndexes is awfully lonesome out in CRM_Contact_Form_DedupeRules. Maybe that functionality can be brought back into the CRM_Dedupe_DAO_DedupeRule class:
[CRM-6245](https://issues.civicrm.org/jira/browse/CRM-6245)
[CRM-3837](https://issues.civicrm.org/jira/browse/CRM-3837)https://lab.civicrm.org/dev/core/-/issues/3429API4: Make CaseType a managed entity2024-01-24T14:08:18ZherbdoolAPI4: Make CaseType a managed entityOverview
----------------------------------------
Allow CaseType to be a managed entity so it can be exported via api4.
Example use-case
----------------------------------------
1. Go to `/civicrm/api4`
1. Select CaseType and Export
C...Overview
----------------------------------------
Allow CaseType to be a managed entity so it can be exported via api4.
Example use-case
----------------------------------------
1. Go to `/civicrm/api4`
1. Select CaseType and Export
Current behaviour
----------------------------------------
Cannot export and use as a managed entity.https://lab.civicrm.org/dev/core/-/issues/3414Import code cleanup2023-01-31T04:41:08ZeileenImport code cleanupI've been working to rationalise the import code prior to fixing a couple of larger things. This ticket is intended to explain what is happening in the current code as I make sense of it....
## Basic structure
The import has 4 screens...I've been working to rationalise the import code prior to fixing a couple of larger things. This ticket is intended to explain what is happening in the current code as I make sense of it....
## Basic structure
The import has 4 screens which are (in order)
DataSource
MapField
Preview
Summary
The first 3 screens each call the relevant Parser class in the submit function - in each case in a different 'mode' with the goal of achieving different things.
The code then 'sets' submitted values and values from the parser on the form to pass it through to the next form - this setting appears to be for 2 reasons
1) because the early civi devs didn't know how else to pass values submitted on one screen to the others
2) because they had calculated certain values and perceived it to be more performant not to do so again.
_**Passing form values - the new way**_
The import forms can now all get the submitted values using the `getSubmittedValue` function - this can fetch a field submitted on the `DataSource` form - even if the contact is currently on the MapField form based on [a coded array of which values are on which form](https://github.com/civicrm/civicrm-core/blob/dfa2f16c836659a5122ee2524246083e465bca87/CRM/Import/Forms.php#L99) - so in the new code we use `getSubmittedValue` rather than the endless set+get routine to get the submitted values (this is in progress - [see](https://github.com/civicrm/civicrm-core/pull/23288/files#diff-4f0a1c0d1f7a02c448ebe273bd5b951a1811dd14949b11b9c8914b2c198fcebaR114)
## Running the parser
Here is the goal of the parser at each step
**DataSource**
The DataSource screen doesn 'really' use the parser although it calls it in MAPFIELD mode- it actually only uses the csv parsing functionality of it it to get
1) rows - it actually requests the first 100 rows & then uses 2 of them to pass to the MapField screen as sammple data - these are assigned to the template as `dataValues`
2) column headers - if there are some are assigned to the template
3)various rowCount, columnCount variables to be assigned to the template to use with the smarty `{section}` tag
_**DataSource parsing- the new way**_
The DataSource form doesn't really need to call the Parser class at all. The MapField can get it's own values once
1) these functions are available on all imports (currently only Contact)
```
$this->assign('columnNames', $this->getColumnHeaders());
$this->assign('columnCount', $this->getNumberOfColumns());
$this->assign('dataValues', array_values($this->getDataRows(2)));
```
2) The rewrite of MapTable.tpl done for Contact import is done for the others - allowing us to stop needing all those count variables.
The datasource class is also doing way too much handling of the temp tables - once https://github.com/civicrm/civicrm-core/pull/23273 is merged the datasource classes (CSV and SQL which are currently used by the contact import and are a todo for the other imports) will handle creating the tables and dropping them and adding the status columns
**MapField**
The mapfield class calls the parser in PREVIEW mode - the goal here is to validate the rows in the datasource and to provide access to download data about rows that have not imported
_**MapField parsing- the new way**_
The goal here is to separate out the validate function and call only that not 'run' - the validate function would update the output directly rather than doing this weird array_shifting pushin on errors.
**Preview**
The preview class calls the parser in import mode - this is where the importing happend
**Summary**
This class does not call the parser - however there is a postProcess there to drop the temp table - the intent in fact is that we RETAIN the temp table to export output on the fly rather than in CSV files - see https://github.com/civicrm/civicrm-core/pull/23291
## Managing the mappings
The way it works is that the selection is passed to the postProcess on `MapField` looking something like
`[['first_name'], ['phone_type_id', '1', 2], ['2_a_b', 'phone_type_id', '1', 2]]`
This is interpreted in 2 ways
1) to be saved in civicrm_mapping_field - for this purpose the above is translated to
```
[
[
'name' => 'First Name'
],
[
'name' => 'Phone',
'location_type_id' => 1,
'phone_type_id' => 2,
],
[
'name' => 'Phone',
'relationship_type_id' => 2,
'relationship_direction' => 'a_b',
'location_type_id' => 1,
'phone_type_id' => 2,
],
]
```
This conversion becomes more readable with [this change](https://github.com/civicrm/civicrm-core/pull/23288/files#diff-6381f637d7d41cb4ea44e8337383a8be3fb2f0c4ce894e8513981113fc2ea672R3571-R3581)
Note that the label rather than name is saved - this is a todo to fix - see https://github.com/civicrm/civicrm-core/pull/23288
2) to be set on the parser - (trigger warning) - this is passed as a series of arrays like
`$phoneTypeIDs = [NULL, 2, 2,]`
`$relationshipDirections = [NULL, NULL, 'a_b']`
etc etc - the parser then (in a hugely unreadable way) matches all these arrays up by index to get the sort of info visible in the above mapping array - it does that in `setActiveFields` which converts the row to a `$params` keyed in a semi-meaningful way
- [x] Stop dropping Temp Tables on completion (contact import only) https://github.com/civicrm/civicrm-core/pull/23291
- [x] Save Mappings by name not label - Contact - https://github.com/civicrm/civicrm-core/pull/23288
## Handy files for testing
[contact-more-columns.csv](/uploads/c4b70665db7a5e45e1a4b7c6c5237815/contact-more-columns.csv)
[contact-more-than-100-rows.csv](/uploads/2627012d2b7d5278ba5d7abe94c59ab2/contact-more-than-100-rows.csv)
[contact-limited.csv](/uploads/740549ad5aab60cda19efca36c384a28/contact-limited.csv)
[memberships.csv](/uploads/d20113bd1bff48ca3b1dbf793c63c74f/memberships.csv)
[a-b-c.csv](/uploads/90e8106f0fb98b2b3d49629a0b0b37db/a-b-c.csv)https://lab.civicrm.org/dev/core/-/issues/3287SearchKit: ability to export and import saved searches so they can be shared ...2022-04-22T15:53:53ZherbdoolSearchKit: ability to export and import saved searches so they can be shared between sitesThe clone functionality works well enough, but I was looking around to see if there was the ability to export and import the saved searches so I could, say, develop a search on a dev site and then deploy it once ready. Or share it betwee...The clone functionality works well enough, but I was looking around to see if there was the ability to export and import the saved searches so I could, say, develop a search on a dev site and then deploy it once ready. Or share it between clients.https://lab.civicrm.org/dev/core/-/issues/3240Searchkit: ability to have simple "exposed filters" on the Display so staff c...2022-04-22T15:51:54ZherbdoolSearchkit: ability to have simple "exposed filters" on the Display so staff can do simple filtering (for example, set a date range but nothing else)I quite like the power of SearchKit but I was thinking that we'd share the display of a search with a client. But it would be great if they had the ability to do a simple filter, such as restricting the date range. Rather than give them ...I quite like the power of SearchKit but I was thinking that we'd share the display of a search with a client. But it would be great if they had the ability to do a simple filter, such as restricting the date range. Rather than give them all the power and complexity of the full search. (I guess I'm just used to this in Views).https://lab.civicrm.org/dev/core/-/issues/3181PHP 8.12023-07-21T14:03:03ZJoeMurrayPHP 8.1We should support PHP 8.1.
The deprecations and breaking changes are listed in the release notes at https://www.php.net/releases/8.1/en.php#deprecations_and_bc_breaks
Here are the backward compatibility issues:
1. [ ] Tentative return...We should support PHP 8.1.
The deprecations and breaking changes are listed in the release notes at https://www.php.net/releases/8.1/en.php#deprecations_and_bc_breaks
Here are the backward compatibility issues:
1. [ ] Tentative return types in PHP built-in class methods
1. [ ] HTML entity en/decode functions process single quotes and substitute by default
1. [ ] $GLOBALS variable restrictions.
1. [ ] MySQLi: Default error mode set to exceptions
I am most worried about HTML encode/decode.
[ ] create edge test setup for PHP 8.1seamusleeseamusleehttps://lab.civicrm.org/dev/core/-/issues/3155Save button on API4 explorer is misleading2022-05-03T14:09:45ZBradley TaylorSave button on API4 explorer is misleadingWhen doing a `get` search with the API4 explorer a "Save..." button is shown. This is designed to "save" the result(s) as a smart group.
When I first started using the API4 explorer it took me several months to realise that this was the...When doing a `get` search with the API4 explorer a "Save..." button is shown. This is designed to "save" the result(s) as a smart group.
When I first started using the API4 explorer it took me several months to realise that this was the use of this button. I assumed that the use case was to search the form configuration in order to quickly get back to an API call in future. Exlanation of the button is shown in the sidebar on hover, but I didn't read this, assuming that it wasn't relavant.
I would like to suggest that the button text is changed to "Save smart group" or "Create smart group" to make the use case much clearer for new users.
This could have layout implications, however the buttons already wrap poorly on a default d7 instalation, so re-thinking the button positions might not be such a bad thing.5.50.0https://lab.civicrm.org/dev/core/-/issues/3152Data stored in universal time does not handle DST consistently2023-11-30T05:00:20ZtottenData stored in universal time does not handle DST consistently[[_TOC_]]
Overview
----------------------------------------
CiviCRM has a number of `TIMESTAMP` columns -- these are stored in universal time (UTC) and displayed in the user's timezone. However, there is a subtle error in handling Dayl...[[_TOC_]]
Overview
----------------------------------------
CiviCRM has a number of `TIMESTAMP` columns -- these are stored in universal time (UTC) and displayed in the user's timezone. However, there is a subtle error in handling Daylight Savings Time (DST): if the current-date and the target-date sit on different sides of the DST-switch, then the time may present as +/- 1 hour.
This bug was one of the major subissues identified in https://lab.civicrm.org/dev/core/-/issues/2122. Although that particular feature was rolled-back/deferred from 5.47, the DST bug still exists -- it's just less obvious.
Example use-case
----------------------------------------
1. Create a contact record.
2. View the contact record. Note the creation time (`civicrm_contact.created_date`).
3. Change the system clock - set to a date where DST differs (eg if today is March 30, then go to December 5).
4. View the contact record. Note the creation time (`civicrm_contact.created_date`).
Current behavior
----------------------------------------
The displayed value of `civicrm_contact.created_date` _appears_ to change by +/- 1 hour, depending on when you view it.
> (Viewed on Mar 31, 2022)
>
> ![Screen_Shot_2022-03-31_at_12.40.53_AM](/uploads/8a2f5a0e4fb6ec884aede78e30d31b1a/Screen_Shot_2022-03-31_at_12.40.53_AM.png)
> (Viewed on Dec 5, 2022)
>
> ![Screen_Shot_2022-12-05_at_12.44.57_AM](/uploads/2df387488ee029a949010da163bf2ea8/Screen_Shot_2022-12-05_at_12.44.57_AM.png)
Why? CiviCRM sends a note to MySQL about the current user's timezone (`SET time_zone = '...'`). However, it doesn't identify the timezone effectively. It gives [the current numeric offset (at the moment of viewing)](https://github.com/civicrm/civicrm-core/blob/5.47.3/CRM/Utils/System/Base.php#L758-L762) - but (in locales with DST) the offsets fluctuate over time.
(_Ex: On Mar 31, the offset in California is `-0700`. Under current/long-standing law, the offset will be `-0800` on Dec 5. Of course, the US Congress is reconsidering this law... so we don't really know what the offset will be!_)
Proposed behavior 1: Fix MySQL timezones
----------------------------------------
CiviCRM should send the timezone as a symbolic name, such as `Europe/Helsinki`, `America/Los_Angeles`, or `Australia/Sydney`. These symbolic-names have an underlying database which allows them adjust automatically based on DST-rules/target-dates/current-law. On the surface, the fix is extremely simple:
```diff
diff --git a/CRM/Utils/System/Base.php b/CRM/Utils/System/Base.php
index a4660834c5..8e40f6da35 100644
--- a/CRM/Utils/System/Base.php
+++ b/CRM/Utils/System/Base.php
@@ -755,10 +755,9 @@ abstract class CRM_Utils_System_Base {
* Set timezone in mysql so that timestamp fields show the correct time.
*/
public function setMySQLTimeZone() {
- $timeZoneOffset = $this->getTimeZoneOffset();
- if ($timeZoneOffset) {
- $sql = "SET time_zone = '$timeZoneOffset'";
- CRM_Core_DAO::executequery($sql);
+ $timeZone = $this->getTimeZoneString();
+ if ($timeZone) {
+ CRM_Core_DAO::executequery('SET time_zone = %1', [1 => [$timeZone, 'String']]);
}
}
```
There are a couple of catches.
* __Timezone rules change__ (occasionally). Any software that supports timezones ultimately needs a _data feed_ with current rules. The good news: IANA publishes a free/open feed (https://www.iana.org/time-zones; aka `tzdata`; aka `zoneinfo`), most Linux/Unix distros have this feed, and MySQL can read it (`mysql_tzinfo_to_sql`). It usually requires one command (which could run during system-config, system-startup, and/or cron):
```bash
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql mysql
```
The problem: we have no measures for (a) how many CiviCRM deployments actually subscribe to this feed and (b) how many could subscribe, if they chose to.
* __Timezone names may be inconsistent__ (occasionally). For example, in different contexts, it's been fashionable to refer to California's timezone as `America/Los_Angeles`, `US/Pacific`, and `PST8PDT`. (The current+official fashion is `America/Los_Angeles` - the others are deprecated.) However, since Civi integrates with various layers (different CMSs; PHP APIs; MySQL APIs), there are edge-case where the layers may choose different names. (*I'm not super-concerned, but we should raise sensible warnings when names are invalid or mismatched.*)
The central issue is - how to cope when data isn't available? This comes to mind:
* (Status check) If the active TZ (`getTimeZoneString()`) has a deprecated name (eg `PST8PDT`) or an offset (eg `-0700`), show a warning.
* (Status check) If the active TZ (`getTimeZoneString()`) isn't supported by MySQL, show a warning.
* (Runtime) If the active TZ (`getTimeZoneString()`) isn't supported by MySQL, fallback to sending offset.
Proposed behavior 2: Change format. Use only PHP TZs.
----------------------------------------
(_This expands on one of @haystack's suggestions in dev/core#2122._)
If you assume that MySQL time services aren't available - what else would you do? You could use PHP time services.
The astute observer will note the status-quo (using both PHP and MySQL time-services) creates two points-of-failure. If either PHP _or_ MySQL has bad/incomplete/old timezone data, then you'll get mis-calculations _somewhere_. Consolidating on PHP time-services would reduce the #dependencies.
Both Drupal and WordPress take this approach. (I suspect this is extremely useful for maximizing compatibility with heterogeneous web-hosts.) They each do it a bit differently, but some central concepts are the same:
* In Drupal, PHP processes read+write temporal data in universal time -- as an `INT` (Unix-style, seconds-since-epoch).
* In WordPress, PHP processes read+write temporal data in universal time -- as a `DATETIME` with a `_gmt` suffix (eg `post_modified_gmt`).
* Hypothetically, you could hardcode MySQL to `SET time_zone='+0:00'`. PHP processes would read+write temporal data in universal time -- as a `TIMESTAMP`.
In all those cases, the onus is on the PHP devs to convert to/from universal-time when implementing functionality (eg "find records from March 1 - March 15" or "find records from this afternoon" or "extract the hour:minute component").
But there is a catch here: Civi already relies on several MySQL time-services. The schema works a certain way; the reports/searches/UIs/APIs expect the schema to work a certain way; etc.
The central issue is - how do you manage/QA all the changes (in schema+logic) required to change time-service?
Comments
----------------------------------------
* I haven't tested, but I'm fairly certain there will be another manifestation in CiviMail scheduling. Ex:
* You live in a timezone where DST changes on March 16.
* On March 10, you schedule a mail-blast for 2:00pm on March 20. It stores the schedule with the wrong offset.
* When March 20 comes, the mailing actually goes out at 3:00pm (or maybe 1:00pm).