CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-11-06T05:03:22Zhttps://lab.civicrm.org/dev/core/-/issues/3020Environments as collections of settings2023-11-06T05:03:22ZtottenEnvironments as collections of settingsOverview
----------------------------------------
Revise the conception of "environment" so that it can more usefully model these distinctions:
* As a developer working on patches with abstract sample data, you want to toggle some sett...Overview
----------------------------------------
Revise the conception of "environment" so that it can more usefully model these distinctions:
* As a developer working on patches with abstract sample data, you want to toggle some settings/behaviors (eg `backtrace=1`).
* As a developer/trainer/etc working on with a local copy of real data, you want to toggle different settings/behaviors (eg restricting cron jobs that might communicate to real people).
This issue proposes that an `environment` is *merely and strictly* a collection of settings - changing the environment will change the active settings.
Off-shoot of discussion circa https://github.com/civicrm/civicrm-core/pull/22377#issuecomment-1006058134
Current behaviour
----------------------------------------
* Sysadmins can design bespoke meta-settings by editing `civicrm.settings.php`
* `civibuild` users can design bespoke meta-settings by placing files [`/etc/civicrm.settings.d/*.php`](https://docs.civicrm.org/dev/en/latest/tools/civibuild/#settings-civicrm)
* The setting `environment` has the primary effect of toggling some guards for `JobManager` / `mailing_backend` / `runInNonProductionEnvironment`.
Proposed behaviour
----------------------------------------
1. Use `environment` to load files which list standard/default values for various settings. eg
* `civicrm-core.git:env/Development.json` has a list of defaults to use in "Development" envs
* `/etc/civicrm.settings.d/env/Development.json` has a list of local overrides to use in "Development" envs
2. Make it easier to set `environment` on new builds, eg
* Add an install option (eg `cv core:install --env=Demo`)
* Add an env-var (eg `CIVICRM_ENVIRONMENT=Staging`) - respected by both installer+runtime
* Add a self-documenting subcommand to configure `civibuild`, eg
```bash
civibuild config --env=Development
```
3. Don't hard-code `environment` behaviors in PHP. Always use settings. eg
* For the JobManager/runInNonProductionEnvironment stuff, there should be a setting `restrictScheduledJobs`. For "Development" envs, the effective-default is `restrictScheduledJobs=true`.
For example, the standard definition of a "Development" environment might be:
```javascript
// FILE: civicrm-core.git:env/Development.json
{
"debug_enabled": true,
"backtrace": true,
"restrictScheduledJobs": true,
"assetCache": 0
}
```
But I could still tune it for my system:
```javascript
// FILE: /etc/civicrm.settings.d/env/Development.json
{
"restrictScheduledJobs": false,
"mailing_backend": {
"outBound_option" => 0,
"smtpServer": "localhost",
"smtpPort": 1025,
"smtpAuth": FALSE
}
}
```
In a similar vein, it may also help to allow `global $civicrm_setting` to target by environment:
```php
## Normal usage
$civicrm_setting['domain']['backtrace'] = 1;
## Target settings per environment.
$civicrm_setting['#Development']['backtrace'] = 1;
$civicrm_setting['#Production']['backtrace'] = 0;
```https://lab.civicrm.org/dev/core/-/issues/2855Preserve pristine ids' for further manipulation via hooks for reports2023-11-06T05:03:21ZyashodhaPreserve pristine ids' for further manipulation via hooks for reportsPreserve pristine ids' for further manipulation via hooks for reports. Ids like state_province_id, country_id, etc are all over-written by the respective look up.
There could be a use-case to display abbreviated forms for some of these ...Preserve pristine ids' for further manipulation via hooks for reports. Ids like state_province_id, country_id, etc are all over-written by the respective look up.
There could be a use-case to display abbreviated forms for some of these fields for which we need the id. I suggest that we push the pristine ids to the bunch (we already have link, hover, etc).yashodhayashodhahttps://lab.civicrm.org/dev/core/-/issues/4465Afform - Default value for boolean filters not displayed correctly in all fie...2023-11-06T00:00:55Zxavi-xalocAfform - Default value for boolean filters not displayed correctly in all field types## Overview
A PR (https://github.com/civicrm/civicrm-core/pull/25605) has been merged to solve issues ( https://lab.civicrm.org/dev/core/-/issues/4113, for instance) with default values for booleans filters in Search Kit. Although that ...## Overview
A PR (https://github.com/civicrm/civicrm-core/pull/25605) has been merged to solve issues ( https://lab.civicrm.org/dev/core/-/issues/4113, for instance) with default values for booleans filters in Search Kit. Although that PR solves many of the issues identified, some problems still remain.
## Current behaviour
Although filter works fine when the field is of type "radio button", that's not the case when the type is "select" or "checkboxes":
- Select: Works fine when "yes" is the default value, but doesn't show default value when it is set to "no"
- Checkboxes: Doesn't get checked when default value is set to "yes".
## Environment information
* **CiviCRM:** _5\.61.2_Kurund JalmiKurund Jalmihttps://lab.civicrm.org/dev/core/-/issues/4746Standalone - Syntax error2023-11-04T21:40:17ZtottenStandalone - Syntax errorThis appears to be a consistent test-regression circa 5.68.alpha:
* List: https://test.civicrm.org/job/CiviCRM-E2E-Matrix/BKPROF=min,BLDTYPE=standalone-clean,CIVIVER=master,label=bknix-tmp/
* Example: https://test.civicrm.org/job/CiviCR...This appears to be a consistent test-regression circa 5.68.alpha:
* List: https://test.civicrm.org/job/CiviCRM-E2E-Matrix/BKPROF=min,BLDTYPE=standalone-clean,CIVIVER=master,label=bknix-tmp/
* Example: https://test.civicrm.org/job/CiviCRM-E2E-Matrix/BKPROF=min,BLDTYPE=standalone-clean,CIVIVER=master,label=bknix-tmp/6857/console
The test-log shows a PHP fatal:
```
In PasswordReset.php line 23:
[ParseError]
syntax error, unexpected 'string' (T_STRING), expecting function (T_FUNCTIO
N) or const (T_CONST)
```RichRichhttps://lab.civicrm.org/dev/core/-/issues/4747testRegexpOperators fails on `max`2023-11-04T21:39:46ZtottentestRegexpOperators fails on `max`Starting in 5.68, the test `api\v4\Action\ContactGetTest::testRegexpOperators` appears to be failing consistently on `max` environment (php81 + mysql80).
* Example: https://test.civicrm.org/job/CiviCRM-Core-Matrix/BKPROF=max,CIVIVER=mas...Starting in 5.68, the test `api\v4\Action\ContactGetTest::testRegexpOperators` appears to be failing consistently on `max` environment (php81 + mysql80).
* Example: https://test.civicrm.org/job/CiviCRM-Core-Matrix/BKPROF=max,CIVIVER=master,SUITES=phpunit-api4,label=bknix-tmp/13173/testReport/(root)/api_v4_Action_ContactGetTest/testRegexpOperators/
* List: https://test.civicrm.org/job/CiviCRM-Core-Matrix/BKPROF=max,CIVIVER=master,SUITES=phpunit-api4,label=bknix-tmp/
(*Since this would've gotten through PR review, it probably works on `php81`+`mysql57`. This smells like a mysqld compatibility issue.*)
cc @colemanwhttps://lab.civicrm.org/dev/core/-/issues/4750clear cache does not reliably remove durable temp tables2023-11-04T16:45:03Zlcdwebclear cache does not reliably remove durable temp tablesDurable (civicrm_tmp_d%) temp tables are intended for use when the table must be used in multiple transactions and therefore cannot be a true MySQL temp table. These should be cleared out when the clean cache function is triggered from t...Durable (civicrm_tmp_d%) temp tables are intended for use when the table must be used in multiple transactions and therefore cannot be a true MySQL temp table. These should be cleared out when the clean cache function is triggered from the UI, scheduled jobs, or CLI -- with a few exceptions for those created within the last two days or that are tied to the civicrm_user_job table. Currently there is faulty logic that causes them to persist much longer than necessary, thus cluttering the DB.lcdweblcdwebhttps://lab.civicrm.org/dev/core/-/issues/3005DB Error on smart group with form values stored as html entities2023-11-04T05:03:20ZjitendraDB Error on smart group with form values stored as html entitiesWe're encountering DB errors on some sites for those who have smart group form values stored as `<` or `>`.
It seems these smart groups were created using search builder with >, < as the operator. Eg payment with amount < 30, etc....We're encountering DB errors on some sites for those who have smart group form values stored as `<` or `>`.
It seems these smart groups were created using search builder with >, < as the operator. Eg payment with amount < 30, etc.
To replicate, probably just try hitting `Update Smart Group Counts` button on Manage group page. It errors if form_values has any saved_search entry with `<` or `>` in it.
````
SELECT * FROM `civicrm_saved_search` WHERE `form_values` LIKE '%<%' OR `form_values` LIKE '%>%';
````
If i try creating a new smart group using the operator, it correctly decodes the value before storing it in the database, so probably it just affects the existing groups.
Approach to fix:
- Replace these characters while retrieving the SQL for group contact cache - https://github.com/civicrm/civicrm-core/blob/master/CRM/Contact/BAO/GroupContactCache.php#L633 -
return str_replace(['<', '>'], ['<', '>'], "$select {$sqlParts['from']} {$sqlParts['where']} {$sqlParts['group_by']} {$sqlParts['having']}");
Might be Related - https://lab.civicrm.org/dev/core/-/issues/1328 & https://github.com/civicrm/civicrm-core/pull/21556https://lab.civicrm.org/dev/core/-/issues/3009authx: Proposal: Make "require" the default user policy instead of "optional"2023-11-04T05:03:19ZDaveDauthx: Proposal: Make "require" the default user policy instead of "optional"If you look at https://docs.civicrm.org/dev/en/latest/framework/authx/#settings the default setting for the `XXX_user` settings is "optional" which means a corresponding cms user doesn't need to exist in order to log in. This seems unint...If you look at https://docs.civicrm.org/dev/en/latest/framework/authx/#settings the default setting for the `XXX_user` settings is "optional" which means a corresponding cms user doesn't need to exist in order to log in. This seems unintuitive as a default. My expectation is that it would work the same way as regular logins by default, so should default to "require".https://lab.civicrm.org/dev/core/-/issues/4734ADMIN_UI: default checkbox2023-11-03T15:31:20ZGuillaumeSorelADMIN_UI: default checkboxCould it be possible to have checkboxes per default for each new admin screen using SK, so it becomes possible to select multiple lines (for mailings, messages templates, custom fields...) and proceed bulk actions on them, especially del...Could it be possible to have checkboxes per default for each new admin screen using SK, so it becomes possible to select multiple lines (for mailings, messages templates, custom fields...) and proceed bulk actions on them, especially delete?
New SK screens are sweet but actions still need to be proceeded one-by-one.https://lab.civicrm.org/dev/core/-/issues/1771Saved import mapping doesn't work for multi-record custom sets2023-11-03T05:03:20ZAndy ClarkSaved import mapping doesn't work for multi-record custom setsWhen importing a multi-record custom field set, the mapping saves correctly. Tables civicrm_mapping and civicrm_mapping_field appear to be correct. However, when you go to use the saved mapping, only the first (top) field appears to hav...When importing a multi-record custom field set, the mapping saves correctly. Tables civicrm_mapping and civicrm_mapping_field appear to be correct. However, when you go to use the saved mapping, only the first (top) field appears to have been saved - in my case the first field was the external identifier. The import then fails with 'unknown error'. Even so, the mapping tables are OK but the import fails. Using release 5.24.4https://lab.civicrm.org/dev/core/-/issues/3015Undefined Index2023-11-03T05:03:19ZJoeMurrayUndefined IndexOverview
----------------------------------------
_Update multiple participants caused undefined index error on dmaster._
Reproduction steps
----------------------------------------
1. Create custom alphanumeric field for participants,...Overview
----------------------------------------
_Update multiple participants caused undefined index error on dmaster._
Reproduction steps
----------------------------------------
1. Create custom alphanumeric field for participants, eg add Idea to Food Preference custom field set on dmaster.
1. Create profile with just this new field, eg test.
1. Find all participants, select a couple, then choose Update multiple participants search results action.
1. Select profile created above. Click continue.
1. See Notice: Undefined index: size in CRM_Event_Form_Task_Batch->buildQuickForm() (line 108 of /srv/buildkit/build/dmaster/web/sites/all/modules/civicrm/CRM/Event/Form/Task/Batch.php).
1. Note that the Idea field has a column but there is no field provided to enter data.
Current behaviour
----------------------------------------
![2022-01-03_11-51-42](/uploads/3b007e3aa61eb2b3cc00bd3d10e36360/2022-01-03_11-51-42.png)
Expected behaviour
----------------------------------------
No error. Ability to enter / edit data in Idea field.
Environment information
----------------------------------------
dmaster on mac chrome.EdselopezEdselopez2022-01-05https://lab.civicrm.org/dev/core/-/issues/4718[PHP 8.1] Find and Merge Duplicate Contacts: Undefined array key "weight" in...2023-11-03T02:04:37Zjofranzfranz@systopia.de[PHP 8.1] Find and Merge Duplicate Contacts: Undefined array key "weight" in CRM_Core_Action## Having
- Drupal 9.5.11
- 5.66.0
## Visiting
.../civicrm/contact/deduperules?reset=1
## Seeing
22x:
```Warning: Undefined array key "weight" in CRM_Core_Action::{closure}() (line 318 of /var/www/civicrm_env/autophpdrupalextupdate...## Having
- Drupal 9.5.11
- 5.66.0
## Visiting
.../civicrm/contact/deduperules?reset=1
## Seeing
22x:
```Warning: Undefined array key "weight" in CRM_Core_Action::{closure}() (line 318 of /var/www/civicrm_env/autophpdrupalextupdate/drupal/vendor/civicrm/civicrm-core/CRM/Core/Action.php).```5.67.0https://lab.civicrm.org/dev/core/-/issues/3742Upgrade to 5.51.1 from 5.50.4 on Drupal through drush command2023-11-03T02:03:10ZmasettoUpgrade to 5.51.1 from 5.50.4 on Drupal through drush commandOverview
----------------------------------------
I tried updating a site from 5.50.3 to 5.51.1 and I received this error:
```
error] $Fatal Error Details = Array
(
[callback] => Array
(
[0] => CRM_Core_Error
...Overview
----------------------------------------
I tried updating a site from 5.50.3 to 5.51.1 and I received this error:
```
error] $Fatal Error Details = Array
(
[callback] => Array
(
[0] => CRM_Core_Error
[1] => exceptionHandler
)
[code] => -19
[message] => DB Error: no such field
[mode] => 16
[debug_info] => UPDATE civicrm_queue SET status = NULL WHERE name = 'CRM_Upgrade' [nativecode=1054 ** Unknown column 'status' in 'field list']
[type] => DB_Error
[user_info] => UPDATE civicrm_queue SET status = NULL WHERE name = 'CRM_Upgrade' [nativecode=1054 ** Unknown column 'status' in 'field list']
[to_string] => [db_error: message="DB Error: no such field" code=-19 mode=callback callback=CRM_Core_Error::exceptionHandler prefix="" info="UPDATE civicrm_queue SET status = NULL WHERE name = 'CRM_Upgrade' [nativecode=1054 ** Unknown column 'status' in 'field list']"]
)
```
Then I rebuild `civicrm_queue` manually:
```
SET FOREIGN_KEY_CHECKS = 0;
DROP TABLE civicrm_queue;
CREATE TABLE `civicrm_queue` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(64) COLLATE utf8mb4_unicode_ci NOT NULL COMMENT 'Name of the queue',
`type` varchar(64) COLLATE utf8mb4_unicode_ci NOT NULL COMMENT 'Type of the queue',
`runner` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'Name of the task runner',
`batch_limit` int(10) unsigned NOT NULL DEFAULT 1 COMMENT 'Maximum number of items in a batch.',
`lease_time` int(10) unsigned NOT NULL DEFAULT 3600 COMMENT 'When claiming an item (or batch of items) for work, how long should the item(s) be reserved. (Seconds)',
`retry_limit` int(11) NOT NULL DEFAULT 0 COMMENT 'Number of permitted retries. Set to zero (0) to disable.',
`retry_interval` int(11) DEFAULT NULL COMMENT 'Number of seconds to wait before retrying a failed execution.',
`status` varchar(16) COLLATE utf8mb4_unicode_ci DEFAULT 'active' COMMENT 'Execution status',
`error` varchar(16) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'Fallback behavior for unhandled errors',
`is_template` tinyint(4) NOT NULL DEFAULT 0 COMMENT 'Is this a template configuration (for use by other/future queues)?',
PRIMARY KEY (`id`),
UNIQUE KEY `UI_name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci ROW_FORMAT=DYNAMIC
```
and the command `drush cvupdb` worked fine.
But the CiviCRM menu is no longer visible.
Reproduction steps
----------------------------------------
1. go to the Drupal site root dir
1. `composer update`
1. `drush cvupdb`
1. Got an error "**DB Error: no such field**".
2. CiviCRM menu is no longer visibile
Environment information
----------------------------------------
* __Browser:__ _Firefox 59.0.1/Chrome 78.0.3904/Safari 13_
* __CiviCRM:__ _CiviCRM 5.51.1_
* __PHP:__ _7.4_
* __CMS:__ Drupal 9_
* __Database:__ _10.5.12-MariaDB_https://lab.civicrm.org/dev/core/-/issues/3101CiviCRM 5.47.0 Upgrade Failure with Extended Reports 5.112023-11-03T02:02:24ZkcristianoCiviCRM 5.47.0 Upgrade Failure with Extended Reports 5.11Logging here as extended reports is widely used.
CiviCRM 5.46 to 5.47.0
ExtendedReport v 5.11.0
WP 5.8.3
php 7.4
Upgrade via the UI or CLI fails:
```
Exception: "API error: DB Error: constraint violation on ReportTemplate.create"
#0...Logging here as extended reports is widely used.
CiviCRM 5.46 to 5.47.0
ExtendedReport v 5.11.0
WP 5.8.3
php 7.4
Upgrade via the UI or CLI fails:
```
Exception: "API error: DB Error: constraint violation on ReportTemplate.create"
#0 /home/cvdemo/public_html/wp-content/plugins/civicrm/civicrm/CRM/Core/ManagedEntities.php(375): CRM_Core_ManagedEntities->onApiError("ReportTemplate", "create", (Array:8), (Array:4))
#1 /home/cvdemo/public_html/wp-content/plugins/civicrm/civicrm/CRM/Core/ManagedEntities.php(187): CRM_Core_ManagedEntities->updateExistingEntity(Object(CRM_Core_DAO_Managed), (Array:10))
#2 /home/cvdemo/public_html/wp-content/plugins/civicrm/civicrm/CRM/Core/ManagedEntities.php(167): CRM_Core_ManagedEntities->reconcileEnabledModule("nz.co.fuzion.extendedreport")
#3 /home/cvdemo/public_html/wp-content/plugins/civicrm/civicrm/CRM/Core/ManagedEntities.php(128): CRM_Core_ManagedEntities->reconcileEnabledModules()
#4 /home/cvdemo/public_html/wp-content/plugins/civicrm/civicrm/CRM/Core/Invoke.php(413): CRM_Core_ManagedEntities->reconcile()
#5 /home/cvdemo/public_html/wp-content/plugins/civicrm/civicrm/CRM/Upgrade/Form.php(818): CRM_Core_Invoke::rebuildMenuAndCaches(FALSE, TRUE)
#6 /home/cvdemo/public_html/wp-content/plugins/civicrm/civicrm/CRM/Queue/Task.php(73): CRM_Upgrade_Form::doCoreFinish(Object(CRM_Queue_TaskContext), "5.47.alpha1", "5.47.0", "5.47.0", "/tmp/civicrm-post-upgrade7axXs1")
#7 /home/cvdemo/public_html/wp-content/plugins/civicrm/civicrm/CRM/Queue/Runner.php(215): CRM_Queue_Task->run(Object(CRM_Queue_TaskContext))
#8 /home/cvdemo/public_html/wp-content/plugins/civicrm/civicrm/CRM/Queue/Runner.php(169): CRM_Queue_Runner->runNext()
#9 /home/cvdemo/public_html/wp-content/plugins/civicrm/civicrm/CRM/Upgrade/Headless.php(49): CRM_Queue_Runner->runAll()
#10 /home/cvdemo/public_html/wp-content/plugins/civicrm/wp-cli/civicrm.php(1083): CRM_Upgrade_Headless->run()
#11 /home/cvdemo/public_html/wp-content/plugins/civicrm/wp-cli/civicrm.php(171): CiviCRM_Command->upgradeDB()
#12 [internal function](): CiviCRM_Command->__invoke((Array:0), (Array:0))
#13 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/WP_CLI/Dispatcher/CommandFactory.php(100): call_user_func((Array:2), (Array:1), (Array:0))
#14 [internal function](): WP_CLI\Dispatcher\CommandFactory::WP_CLI\Dispatcher\{closure}((Array:1), (Array:0))
#15 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/WP_CLI/Dispatcher/Subcommand.php(491): call_user_func(Object(Closure), (Array:1), (Array:0))
#16 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(399): WP_CLI\Dispatcher\Subcommand->invoke((Array:1), (Array:0), (Array:0))
#17 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(422): WP_CLI\Runner->run_command((Array:2), (Array:0))
#18 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1194): WP_CLI\Runner->run_command_and_exit()
#19 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/WP_CLI/Bootstrap/LaunchRunner.php(23): WP_CLI\Runner->start()
#20 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/bootstrap.php(77): WP_CLI\Bootstrap\LaunchRunner->process(Object(WP_CLI\Bootstrap\BootstrapState))
#21 phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/wp-cli.php(27): WP_CLI\bootstrap()
#22 phar:///usr/local/bin/wp/php/boot-phar.php(11): include("phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/wp-cli.php")
#23 /usr/local/bin/wp(4): include("phar:///usr/local/bin/wp/php/boot-phar.php")
#24 {main}
Error: API error: DB Error: constraint violation on ReportTemplate.create
Array
(
[is_error] => 1
[error_message] => API error: DB Error: constraint violation on ReportTemplate.create
)
```
You can either disable extended reports or upgrade extended reports to get the upgrade to succeed.
This seems to be a new issue as my testing to 5.47-RC did not have this issue. I had not tested upgrading in about a week. The two sites I tested this on are 'demo' sites that had not been on the RC.
We should be recommending extension updates before Core upgrade, but I was surprised I had this error.
cc @eileen Let me know if this issue belongs over at the extended report repo.https://lab.civicrm.org/dev/core/-/issues/4723SearchKit joins to custom fields appear to be broken2023-11-02T20:48:31ZScottMasonSearchKit joins to custom fields appear to be brokenThe latest version of SearchKit allows you to join to custom fields. However, if you try to do this and then add data to a display from the joined entity the data is not shown.
Steps to recreate.
1. Create a contact reference custom fi...The latest version of SearchKit allows you to join to custom fields. However, if you try to do this and then add data to a display from the joined entity the data is not shown.
Steps to recreate.
1. Create a contact reference custom field for activities (for example Activities details > products)
2. Create a new SK search for activities and join to the custom field
![image](/uploads/ebbeb025d909e2df4b8c87d97d7a9ada/image.png)
3. Add info from the linked entity to the display
4. The info is not shown.
![image](/uploads/870046a2cf17c723bfff75917401beb0/image.png)
I have tested this on the lastest demo site and it appears to be an issue there.
fyi @kurundhttps://lab.civicrm.org/dev/core/-/issues/4350Proposal: Smart Group Profiler to help identify slow smart groups2023-11-02T14:29:14ZlarsssandergreenProposal: Smart Group Profiler to help identify slow smart groupsSlow smart groups are a problem that I'm sure we've all experienced. It's difficult for users to guess which of their potentially many smart groups is exceptionally slow or if they just have too many, each taking a little time. @JonGold ...Slow smart groups are a problem that I'm sure we've all experienced. It's difficult for users to guess which of their potentially many smart groups is exceptionally slow or if they just have too many, each taking a little time. @JonGold has a [profiling script](https://gist.github.com/MegaphoneJon/59089c1e155645a438af41051bca7ce7), but that's not accessible to many. I'm sure many orgs would be happy to cut or re-work their slow smart groups if it meant they could use the listing accordion on individual contacts, etc — but they just don't know which ones they need to cut!
What I propose is a Smart Group Profiler in Admin that lets admin users easily check their smart groups in the UI. Here's a quick sketch:
- The user can select groups to profile or leave the select empty to profile all (active, non-hidden) smart groups
- The user can select to run once or average up to three times, to reduce the potential for random variation
- The profiler queues CRM_Contact_BAO_GroupContactCache::add for each group, in random order
- Once the queue finishes, the user gets a table with average time and links to edit the Smart Group criteria for each
I'd be happy to do this as an extension, but it also seems like something that address a core problem, so I think it makes sense to have in core.5.67.0https://lab.civicrm.org/dev/core/-/issues/4741Error: datefmt_create: invalid locale: U_ILLEGAL_ARGUMENT_ERROR2023-11-02T11:56:43Zaydunsaidan.saunders@squiffle.ukError: datefmt_create: invalid locale: U_ILLEGAL_ARGUMENT_ERROROverview
----------------------------------------
Trying to access some civicrm pages produces "datefmt_create: invalid locale: U_ILLEGAL_ARGUMENT_ERROR"
This is being encountered with php8.1, but not php7.4. It is on Joomla 4, but I d...Overview
----------------------------------------
Trying to access some civicrm pages produces "datefmt_create: invalid locale: U_ILLEGAL_ARGUMENT_ERROR"
This is being encountered with php8.1, but not php7.4. It is on Joomla 4, but I don't think that is the problem.
The Joomla stacktrace is:
```
| # | Function | Location |
|-----|-------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
| 1 | () | JROOT/administrator/components/com_civicrm/civicrm/vendor/pear/log/php-8.1-strftime.php:106 |
| 2 | IntlDateFormatter->__construct() | JROOT/administrator/components/com_civicrm/civicrm/vendor/pear/log/php-8.1-strftime.php:106 |
| 3 | PHP81_BC\{closure}() | JROOT/administrator/components/com_civicrm/civicrm/vendor/pear/log/php-8.1-strftime.php:205 |
| 4 | PHP81_BC\{closure}() | |
| 5 | preg_replace_callback() | JROOT/administrator/components/com_civicrm/civicrm/vendor/pear/log/php-8.1-strftime.php:186 |
| 6 | PHP81_BC\strftime() | JROOT/administrator/components/com_civicrm/civicrm/vendor/pear/log/Log.php:887 |
| 7 | Log->formatTime() | JROOT/administrator/components/com_civicrm/civicrm/vendor/pear/log/Log/file.php:294 |
| 8 | Log_file->log() | JROOT/administrator/components/com_civicrm/civicrm/CRM/Core/Error.php:601 |
| 9 | CRM_Core_Error::debug_log_message() | JROOT/administrator/components/com_civicrm/civicrm/CRM/Core/Error.php:573 |
| 10 | CRM_Core_Error::debug_var() | JROOT/administrator/components/com_civicrm/civicrm/CRM/Core/Error.php:442 |
| 11 | CRM_Core_Error::handleUnhandledException() | JROOT/administrator/components/com_civicrm/civicrm/CRM/Core/Invoke.php:39 |
| 12 | CRM_Core_Invoke::invoke() | JROOT/components/com_civicrm/civicrm.php:84 |
| 13 | civicrm_invoke() | JROOT/components/com_civicrm/civicrm.php:11 |
| 14 | require_once() | JROOT/libraries/src/Dispatcher/LegacyComponentDispatcher.php:71 |
| 15 | Joomla\CMS\Dispatcher\LegacyComponentDispatcher::Joomla\CMS\Dispatcher\{closure}() | JROOT/libraries/src/Dispatcher/LegacyComponentDispatcher.php:73 |
| 16 | Joomla\CMS\Dispatcher\LegacyComponentDispatcher->dispatch() | JROOT/libraries/src/Component/ComponentHelper.php:361 |
| 17 | Joomla\CMS\Component\ComponentHelper::renderComponent() | JROOT/libraries/src/Application/SiteApplication.php:208 |
| 18 | Joomla\CMS\Application\SiteApplication->dispatch() | JROOT/libraries/src/Application/SiteApplication.php:249 |
| 19 | Joomla\CMS\Application\SiteApplication->doExecute() | JROOT/libraries/src/Application/CMSApplication.php:293 |
| 20 | Joomla\CMS\Application\CMSApplication->execute() | JROOT/includes/app.php:61 |
| 21 | require_once() | JROOT/index.php:32 |
```
The error occurs when CRM_Core_Error() tries to write to a logfile - so something should be written to the log, but the act of writing creates a new separate error.
In `JROOT/administrator/components/com_civicrm/civicrm/vendor/pear/log/php-8.1-strftime.php:106` right before the call to `IntlDateFormatter()` `$locale` has the value 'C' which is determined from a call to `setlocale(LC_TIME, '0')`
Other uses of `IntlDateFormatter()` call `CRM_Core_I18n::getLocale()` to find the locale but this call is in the pear log module's compatibility layer for the deprecated `strftime()`.
As an ugly hack, setting `$locale = "en_US";` gets things working again.
Environment information
----------------------------------------
* __CiviCRM:__ _5.66.1_
* __PHP:__ _8.1_ - does not occur with 7.4
* __CMS:__ _Joomla 4_5.67.0https://lab.civicrm.org/dev/core/-/issues/2999Get API Error: "invalid string" if param value contains `select` string in it2023-11-02T11:36:40ZjitendraGet API Error: "invalid string" if param value contains `select` string in itTo replicate execute the following apis.
```
civicrm_api3('Contact', 'get', ['display_name' => 'selecton person name']);
```
or
```
civicrm_api3('Contribution', 'get', [
'sequential' => 1,
'source' => "xxxxselect xxxx",
]);
```
...To replicate execute the following apis.
```
civicrm_api3('Contact', 'get', ['display_name' => 'selecton person name']);
```
or
```
civicrm_api3('Contribution', 'get', [
'sequential' => 1,
'source' => "xxxxselect xxxx",
]);
```
or any other entity with `select` string in the param value.
The problematic part is https://github.com/civicrm/civicrm-core/blob/master/api/v3/utils.php#L884-L886, which results into an error if param value contains `select` string in it.
Seems to be present in core from a long time. Why is input params considered as an invalid string for SELECT? Is it to avoid any SQL queries in the params? If yes, probably we can replace it with a more valid check? eg an existence of select, from, etc?https://lab.civicrm.org/dev/core/-/issues/2993Add option to delete old log files during rotation2023-11-02T05:03:25ZJoeMurrayAdd option to delete old log files during rotationOverview
----------------------------------------
_When creating a new log file, optionally remove old ones like logrotate allows (https://linux.die.net/man/8/logrotate)._
Example use-case
----------------------------------------
CiviCR...Overview
----------------------------------------
_When creating a new log file, optionally remove old ones like logrotate allows (https://linux.die.net/man/8/logrotate)._
Example use-case
----------------------------------------
CiviCRM's simplistic log creation does not actually rotate, it just makes more files for the logs and never deletes any (https://github.com/civicrm/civicrm-core/blob/5433c81f5648bf484ef16debeca3c7e6e98d1a11/CRM/Core/Error.php#L684). This can be dangerous for systems where there is little to no active monitoring of logs, eg small orgs without a provider, or on a shoestring budget. It is also dangerous in cases where a change starts to generate high volumes of error messages, eg a client recently misconfigured a civirule and GBs of log were produced every hour. Running out of disk space causes hard crashes of sites.
It is standard practice to have a process in place to ensure that logs do not end up taking too much disk space. We propose allowing a setting to be added to civicrm.settings.php (eg NUM_LOG_FILES_TO_RETAIN) that would allow the number of logs to be retained to be specified. The default setting would be 0, meaning an infinite number, which would retain current behaviour. We would also be happy to make it another number such as 6, which would be 6 months or more than 1.5GB.
My understanding is that settings that can be added to civicrm.settings.php should be included as comments with explanation that can be uncommented, so we will do this.
Current behaviour
----------------------------------------
_CiviCRM generates log files with random hash names, and creates new ones every month or when the file exceeds 256MB. Log file space grows continuously unless log files are manually deleted._
Proposed behaviour
----------------------------------------
_When a new log file is created, if a maximum number of log files has been specified, then delete those over that number, starting with oldest first_
Comments
----------------------------------------
_To reduce the complexity of the settings pages, we propose that this be implemented as a setting that could be optionally specified in civicrm.settings.php._Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/4727Basic/Advanced search and CiviReport filters showing `name` in tag dropdown i...2023-11-01T23:19:57ZDaveDBasic/Advanced search and CiviReport filters showing `name` in tag dropdown instead of `label`![untitled4](/uploads/ea919fe7f1f9257beb0f2c656f3efe2a/untitled4.png)
Also the QILL for the civireport (the QILL for the search is ok).
![untitled3](/uploads/ed24469c7b1110d79340b0f4b3555fb4/untitled3.png)![untitled4](/uploads/ea919fe7f1f9257beb0f2c656f3efe2a/untitled4.png)
Also the QILL for the civireport (the QILL for the search is ok).
![untitled3](/uploads/ed24469c7b1110d79340b0f4b3555fb4/untitled3.png)5.68.0