CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2021-04-07T14:12:26Zhttps://lab.civicrm.org/dev/core/-/issues/2513Dashlet reports not scrollable2021-04-07T14:12:26ZAllenShawDashlet reports not scrollableOverview
----------------------------------------
Scrolling problems prevent full view of expanded dashlet content.
Current behaviour
----------------------------------------
On the CiviCRM home dashboard, a dashlet may be expanded usin...Overview
----------------------------------------
Scrolling problems prevent full view of expanded dashlet content.
Current behaviour
----------------------------------------
On the CiviCRM home dashboard, a dashlet may be expanded using the "View Fullscreen" icon (fa-expand). This does expand the dashlet, but that expanded dashlet is not scrollable. If the dashlet output is many rows, the dashlet simply extends its display below the bottom edge of the viewport.
At the same time, scrollbars are removed from the page itself. Therefore, there's no way to scroll down to see the bottom of the expanded dashlet output. (Mouse-wheel scrolling is also nonresponsive in this situation.)
See this animaged image:
![anim](/uploads/b0e83ea948708aabcdd5cc033d65e23d/anim.gif)
Expected behaviour
----------------------------------------
I should have some way of scrolling down to see the full dashlet content.
Environment information
----------------------------------------
* Browser: Chromium
Version 89.0.4389.90 (Official Build) Built on Ubuntu , running on Ubuntu 16.04 (64-bit)
* Drupal/CiviCRM: dmaster.demo.civicrm.org, as of today ("Powered by CiviCRM 5.37.alpha1.")
Comments
----------------------------------------
A dev site with civicrm 5.28.3 shows me that the expanded dashlet had working vertical scrollbars in that version.https://lab.civicrm.org/dev/core/-/issues/2512Recording the payment is not working if we deleted the "New" Membership Status2023-07-22T05:03:29Zahed_compucorpRecording the payment is not working if we deleted the "New" Membership StatusOverview
----------------------------------------
Some Admins will delete the `New` membership status rule and that is possible because CiviCRM allow adding/editing/removing membership status rules.
Reproduction steps
------------------...Overview
----------------------------------------
Some Admins will delete the `New` membership status rule and that is possible because CiviCRM allow adding/editing/removing membership status rules.
Reproduction steps
----------------------------------------
![Peek_2021-04-01_21-46](/uploads/201ca4c8839b920a896b55ea1de026a0/Peek_2021-04-01_21-46.gif)
Current behaviour
----------------------------------------
Recording a payment for a pending membership will not work and CiviCRM will throw the following error.
```
'New' is not a valid option for field status_id
```
Expected behaviour
----------------------------------------
Recording a payment for a pending membership should work.
Environment information
----------------------------------------
* __CiviCRM:__ _Master/5.35.1/5.28.3_
PR: https://github.com/civicrm/civicrm-core/pull/19975https://lab.civicrm.org/dev/core/-/issues/2511[Unconfirmed Regression] CiviCRM 5.35.1, contact checksum URL is corrupted in...2021-04-27T01:47:46Zjustinfreeman (Agileware)[Unconfirmed Regression] CiviCRM 5.35.1, contact checksum URL is corrupted in Mailings when click thru tracking is enabledCiviCRM 5.35.1, when click thru tracking is enabled, the contact checksum URL is corrupted and will not work. Issue is most likely in the flexmailer extension, as that rewrites the tracker.
Agileware Ref: CIVICRM-1693CiviCRM 5.35.1, when click thru tracking is enabled, the contact checksum URL is corrupted and will not work. Issue is most likely in the flexmailer extension, as that rewrites the tracker.
Agileware Ref: CIVICRM-1693https://lab.civicrm.org/dev/core/-/issues/2510Search kit: Case search with related contact activities includes deleted acti...2023-07-19T13:26:07ZDaveDSearch kit: Case search with related contact activities includes deleted activities by defaultI don't know if it's so much of a bug as unintuitive since typically deleted things are left out by default and e.g. a plain activity search just on activities alone leaves out deleted ones. So it's something about the join maybe.I don't know if it's so much of a bug as unintuitive since typically deleted things are left out by default and e.g. a plain activity search just on activities alone leaves out deleted ones. So it's something about the join maybe.https://lab.civicrm.org/dev/core/-/issues/2509Search kit: Links to case (and other) activities go to the wrong form2023-08-01T01:50:48ZDaveDSearch kit: Links to case (and other) activities go to the wrong formIf you do a search either purely on activities and the results happen to include case activities, or a search on cases and include the related contact activities, the links to the activities in the results list go to the regular activity...If you do a search either purely on activities and the results happen to include case activities, or a search on cases and include the related contact activities, the links to the activities in the results list go to the regular activity form not the case activity form. This matters because:
1. It's not all the right fields on the form.
2. Form hooks won't work properly.
3. You get errors on save.
@colemanwhttps://lab.civicrm.org/dev/core/-/issues/2508Improve unit tests by checking if session::setStatus sets errors2023-07-20T05:03:20ZDaveDImprove unit tests by checking if session::setStatus sets errorsThe alert popups that come up in civi happen two ways: One is via javascript-only, which is difficult to test in the regular phpunit tests, and the other is via CRM_Core_Session::setStatus().
One approach is as in https://github.com/civ...The alert popups that come up in civi happen two ways: One is via javascript-only, which is difficult to test in the regular phpunit tests, and the other is via CRM_Core_Session::setStatus().
One approach is as in https://github.com/civicrm/civicrm-core/pull/19939 but it's not workable in practice because it generates too much extra visual noise on screen. However it pointed out some tests which could be improved. See also https://github.com/civicrm/civicrm-core/pull/19939#issuecomment-809683859 for some additional details. The last one here about financialtype I think I saw a variation of on-screen once and was just that the type of popup was wrong (i.e. using error instead of success).
```
not ok 599 - Error: CRM_Contact_Form_Task_EmailCommonTest::testPostProcessWithSignature
not ok 603 - Error: CRM_Contact_Form_Task_UseraddTest::testUserCreateFail
not ok 653 - Error: CRM_Contact_Page_View_UserDashBoardTest::testDashboardContentEmptyContact
not ok 654 - Error: CRM_Contact_Page_View_UserDashBoardTest::testDashboardContentContributionsWithInvoicingEnabled
not ok 655 - Error: CRM_Contact_Page_View_UserDashBoardTest::testDashboardContentContributions
not ok 698 - Failure: CRM_Contribute_BAO_ContributionRecurTest::testAutoRenewalWhenOneMemberIsDeceased
not ok 944 - Error: CRM_Core_BAO_AddressTest::testSharedAddressChaining3
not ok 1115 - Error: CRM_Core_BAO_SettingTest::testGetItemGroup_Override
not ok 1116 - Error: CRM_Core_BAO_SettingTest::testDefaults
not ok 1117 - Error: CRM_Core_BAO_SettingTest::testOnChange
not ok 1118 - Error: CRM_Core_BAO_SettingTest::testSetCivicrmEnvironment
not ok 1119 - Error: CRM_Core_BAO_SettingTest::testPseudoConstants
not ok 507 - Error: api_v3_ContactTest::testMergedGetWithPermanentlyDeletedContact
not ok 1382 - Failure: api_v3_JobTest::testProcessMembershipDeceased
not ok 1418 - Failure: api_v3_LoggingTest::testEnableDisableLogging
not ok 1522 - Failure: api_v3_MembershipStatusTest::testDeceasedMembershipInline
not ok 280 - Error: Civi\Test\ExampleHookTest::testPageOutput
fatal error - ext/oauth-client/tests/phpunit/CRM/OAuth/MailSetupTest.php:18
not ok 2 - Failure: Civi\Financialacls\FinancialTypeTest::testChangeFinancialTypeName
```https://lab.civicrm.org/dev/core/-/issues/2507API4: "Invalid field 'contact.is_deleted'"2023-10-26T05:03:28ZedvanleeuwenAPI4: "Invalid field 'contact.is_deleted'"I am getting several errors on Api4 and the field is_deleted on a contact. The field is available in the database. At the moment, I do not know how to reproduce this, so I will investigate further. Furthermore, I cannot see any problems....I am getting several errors on Api4 and the field is_deleted on a contact. The field is available in the database. At the moment, I do not know how to reproduce this, so I will investigate further. Furthermore, I cannot see any problems.
```
[error]
$Fatal Error Details = array:3 [
"message" => "Invalid field 'contact.is_deleted'"
"code" => null
"exception" => API_Exception {#2233
-extraParams: array:1 [
"error_code" => 0
]
#message: "Invalid field 'contact.is_deleted'"
#code: 0
#file: "/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/Civi/Api4/Query/Api4SelectQuery.php"
#line: 518
trace: {
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/Civi/Api4/Query/Api4SelectQuery.php:518 {
› if ($strict && !$field) {
› throw new \API_Exception("Invalid field '$fieldName'");
› }
}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/Civi/Api4/Query/Api4SelectQuery.php:371 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/Civi/Api4/Query/Api4SelectQuery.php:349 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/Civi/Api4/Query/Api4SelectQuery.php:245 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/Civi/Api4/Query/Api4SelectQuery.php:114 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/Civi/Api4/Query/Api4SelectQuery.php:129 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/Civi/Api4/Generic/DAOGetAction.php:110 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/Civi/Api4/Generic/DAOGetAction.php:98 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/Civi/Api4/Provider/ActionObjectProvider.php:68 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/Civi/API/Kernel.php:149 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/Civi/Api4/Generic/AbstractAction.php:238 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/CRM/Profile/Page/Dynamic.php:449 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/CRM/Profile/Page/Dynamic.php:332 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/CRM/Profile/Page/View.php:91 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/CRM/Profile/Page/View.php:159 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/CRM/Core/Invoke.php:312 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/CRM/Core/Invoke.php:68 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/CRM/Core/Invoke.php:36 { …}
/home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/drupal/civicrm.module:458 { …}
/home/covsp/domains/covs.nl/public_html/includes/menu.inc:527 { …}
/home/covsp/domains/covs.nl/public_html/index.php:21 { …}
}
}
]
apr 03 11:22:01 [debug] $backTrace = #0 /home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/CRM/Core/Error.php(433): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 /home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/CRM/Core/Invoke.php(39): CRM_Core_Error::handleUnhandledException(Object(API_Exception))
#2 /home/covsp/domains/covs.nl/public_html/sites/all/modules/civicrm/drupal/civicrm.module(458): CRM_Core_Invoke::invoke((Array:3))
#3 /home/covsp/domains/covs.nl/public_html/includes/menu.inc(527): civicrm_invoke("profile", "view")
#4 /home/covsp/domains/covs.nl/public_html/index.php(21): menu_execute_active_handler()
#5 {main}
```https://lab.civicrm.org/dev/core/-/issues/2506Autocomplete-select custom field (Multi-Select=true) values with checked ar...2021-04-26T20:07:08ZKurund JalmiAutocomplete-select custom field (Multi-Select=true) values with checked are reset in case of form rule errorsOverview
----------------------------------------
Custom field of type Autocomplete-select (Multi-Select=true) with does not retain selected values in case of form errors.
Reproduction steps
----------------------------------------
1. C...Overview
----------------------------------------
Custom field of type Autocomplete-select (Multi-Select=true) with does not retain selected values in case of form errors.
Reproduction steps
----------------------------------------
1. Create a custom group for Activities > Meeting
![screenshot-dmaster.demo.civicrm.org-2021.04.01-17_33_45](/uploads/4e7c1510b18ba86477d2ec2ca03aac9e/screenshot-dmaster.demo.civicrm.org-2021.04.01-17_33_45.png)
2. Go to add new activity >> select the custom field values and remove date to trigger formrule
![screenshot-dmaster.demo.civicrm.org-2021.04.01-17_37_39](/uploads/3df4fb6e5c26d53ddcab0855850eec19/screenshot-dmaster.demo.civicrm.org-2021.04.01-17_37_39.png)
3. Formrule is triggered and the selected custom field values are lost
![screenshot-dmaster.demo.civicrm.org-2021.04.01-17_38_04](/uploads/56edc30349618759c176e28b0f4038ea/screenshot-dmaster.demo.civicrm.org-2021.04.01-17_38_04.png)
Current behaviour
----------------------------------------
* See above screenshots from https://dmaster.demo.civicrm.org
Expected behaviour
----------------------------------------
* Form should retain selected values for custom field
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ _Firefox 59.0.1/Chrome 78.0.3904/Safari 13_
* __CiviCRM:__ _Master/5.20.0/5.19.1/5.18.2/..._ <!-- If this problem relates to an upgrade, then specify both old and new versions -->
* __PHP:__ _7.0/7.1/7.2/7.3/...__
* __CMS:__ _Backdrop 1.5/Drupal 7.30/Joomla 3.3/WordPress 4.5/..._
* __Database:__ _MySQL 5.7.7/MariaDB 10.4/..._
* __Web Server:__ _Apache 2.4/Nginx 1.16/..._
Comments
----------------------------------------
_Anything else you would like the reviewer to note._colemanwcolemanwhttps://lab.civicrm.org/dev/core/-/issues/2505CiviReport does not localize custom fields of type Number2023-02-16T13:30:23ZjaapjansmaCiviReport does not localize custom fields of type Number**How to reproduce**
1. Set localization settings: **decimal separator** to `,` and **thousand separator** to `.`
2. Add a custom group for cases
3. Add a custom field of **Type** `Number` and **Field type** `Text`
4. Enter a value such...**How to reproduce**
1. Set localization settings: **decimal separator** to `,` and **thousand separator** to `.`
2. Add a custom group for cases
3. Add a custom field of **Type** `Number` and **Field type** `Text`
4. Enter a value such as `1,234.56` which is localized `1.234,56`
5. Create a new CiviCase detail report. Check the column to display the custom field.
**Expected result**
Value of the custom field displayed as `1.234,56`
**Actual result**
Value of custom field is displayed as `1234.56` (note the decimal separator and the missing thousand separator).
**Comment**
The display value of custom number fields is also broken on the manage case screen.
We have a function `CRM_Utils_Money::formatLocaleNumericRoundedForDefaultCurrency` to format money fields. However we do not have such a function for number fields with a decimal. So basically nowhere in CiviCRM a number field is formatted according to localization settings. Unless the smarty modifier `|crmNumberFormat` is used in the template.
A fix should probably happen in `CRM_Core_BAO_CustomField::formatDisplayValue`, see https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/BAO/CustomField.php#L1187
However I am not exactly sure how to fix this as I could not find a number formatting function such as we have for the money field. Does such a function exists and if so where do I find it?5.38.0https://lab.civicrm.org/dev/core/-/issues/2504CiviReport does not localize custom date fields when exporting to CSV2021-04-09T13:32:59ZjaapjansmaCiviReport does not localize custom date fields when exporting to CSVCiviReport does not localize custom fields of type Date
**How to reproduce**
1. Set localization settings: **Date Format Complete date** to `%d-%m-%Y`
2. Add a custom group for cases
3. Add a custom field of Type Date and no time.
4. E...CiviReport does not localize custom fields of type Date
**How to reproduce**
1. Set localization settings: **Date Format Complete date** to `%d-%m-%Y`
2. Add a custom group for cases
3. Add a custom field of Type Date and no time.
4. Enter a value such as 31-04-2021 (1st of april 2021)
5. Create a new CiviCase detail report. Check the column to display the custom field.
6. Export the report to CSV
**Expected result**
Value of the custom field displayed as `01-04-2021` as it is shown in the report and according to the localization setting.
**Actual result**
Value of custom field is displayed as `2021-04-01` which is ISO format.
**Comment**
The format for iso format is hard coded in: https://github.com/civicrm/civicrm-core/blob/master/CRM/Report/Utils/Report.php#L270
I am not sure if that is the desired behavior. I dont think so as the custom fields of type money are localized into a CSV export. So I would argue that the date should also be formatted according to the localization settings.https://lab.civicrm.org/dev/core/-/issues/2503CiviReport does not localize custom fields of type Money2021-04-05T07:39:54ZjaapjansmaCiviReport does not localize custom fields of type Money**How to reproduce**
1. Set localization settings: **decimal separator** to `,` and **thousand separator** to `.`
2. Add a custom group for cases
3. Add a custom field of **Type** `Money` and **Field type** `Text`
4. Enter a value such ...**How to reproduce**
1. Set localization settings: **decimal separator** to `,` and **thousand separator** to `.`
2. Add a custom group for cases
3. Add a custom field of **Type** `Money` and **Field type** `Text`
4. Enter a value such as `1,234.56` which is localized `1.234,56`
5. Create a new CiviCase detail report. Check the column to display the custom field.
**Expected result**
Value of the custom field displayed as `1.234,56`
**Actual result**
Value of custom field is displayed as `1,234.56` (not the turn around of decimal and thousand separators).
**Linked PR**
https://github.com/civicrm/civicrm-core/pull/199625.37.0https://lab.civicrm.org/dev/core/-/issues/2502SearchBuilder: contact search for tags with child-tags broken when using '=' ...2021-04-09T02:27:08ZBjörn EndresSearchBuilder: contact search for tags with child-tags broken when using '=' operatorOverview
----------------------------------------
Searching for contact tags using the '=' operator is broken, if they have child tags: you get an SQL error. This problem was probably introduced with #1795.
Reproduction steps
---------...Overview
----------------------------------------
Searching for contact tags using the '=' operator is broken, if they have child tags: you get an SQL error. This problem was probably introduced with #1795.
Reproduction steps
----------------------------------------
1. Create tag A
1. Create tag B as a child of tag A
1. Go to the search builder
1. Search for contacts, tag(s), '=', tag A (see screenshot)
1. Got an error "**Fatal error: DB error**".
![search_generator](/uploads/b5a3d95d0b9372d7fc9ca4dd5a4c744c/search_generator.png)
Current behaviour
----------------------------------------
DB Error with a something like
```
civicrm_entity_tag-60659ba2e61bf`.tag_id IN ( )
```
in the generated SQL.
Expected behaviour
----------------------------------------
Search should work :)
Environment information
----------------------------------------
Could reproduce this on [dmaster](https://dmaster.demo.civicrm.org), version ``5.37.alpha1``. Also reproduces on ``5.35.1``.
Comments
----------------------------------------
I think I know the culprit and am working on a patch.5.37.0https://lab.civicrm.org/dev/core/-/issues/2501Calling UFGroup.create to update a profile without specifying is_active=1 dis...2021-04-08T04:48:50ZDaveDCalling UFGroup.create to update a profile without specifying is_active=1 disables the profileThis doesn't seem too recent (happens in 5.24 too) but seems like a bug given that the api spec for UFGroup specifically declares is_active=1 as the default and other api calls don't seem to work this way.
I'm not sure yet where in the ...This doesn't seem too recent (happens in 5.24 too) but seems like a bug given that the api spec for UFGroup specifically declares is_active=1 as the default and other api calls don't seem to work this way.
I'm not sure yet where in the code it's happening.5.38.0https://lab.civicrm.org/dev/core/-/issues/2498Regression dedupe threshold2021-04-19T01:19:26ZeileenRegression dedupe threshold![image](/uploads/f3b9df6dfa67374428ea472501166a73/image.png)
From https://github.com/civicrm/civicrm-core/commit/246500181dbfcdc72a528ad4893e51f295fc3124
Code looks right on first glance to am debugging![image](/uploads/f3b9df6dfa67374428ea472501166a73/image.png)
From https://github.com/civicrm/civicrm-core/commit/246500181dbfcdc72a528ad4893e51f295fc3124
Code looks right on first glance to am debugging5.36.1https://lab.civicrm.org/dev/core/-/issues/2497After updating to 5.35.1 addressess couldn't be saved2021-04-01T07:57:45ZJanecAfter updating to 5.35.1 addressess couldn't be savedOverview
----------------------------------------
When we tried to save an address we got a: "server could not be found" error.
Reproduction steps
----------------------------------------
1. Go to a contact.
1. Change it's address
1. Cl...Overview
----------------------------------------
When we tried to save an address we got a: "server could not be found" error.
Reproduction steps
----------------------------------------
1. Go to a contact.
1. Change it's address
1. Click save.
Current behaviour
----------------------------------------
An error is produced and the changes to the address are not saved.
Expected behaviour
----------------------------------------
Changes should be saved and confirmation should be given to the user.
Environment information
----------------------------------------
* __CiviCRM:__ 5.31(?) -> 5.35.1
* __PHP:__ 7.2
* __CMS:__ Drupal 7.73
Comments
----------------------------------------
Could be a thing that's only applicable to our set up.
Changing
`$address = CRM_Core_BAO_Address::legacyCreate($params, TRUE) `
into
`$address = CRM_Core_BAO_Address::create($params, TRUE); `
in: CRM/Contact/Form/Inline/Address.php @ line 172
Solved it
[civicrm-address-remove-legacy.patch](/uploads/7bf6d83f5462656e267b3cae7a4e6e74/civicrm-address-remove-legacy.patch).https://lab.civicrm.org/dev/core/-/issues/2496Api support for afform contribution pages2023-07-17T05:03:23ZeileenApi support for afform contribution pagesAfform what do we need to do to offer Contribution form
Contribution entity - but just basic fields
Payment block
To do the latter we need an api like
PaymentForm with the actions
```
->submit
->getfields
->validate
```
```payment_pro...Afform what do we need to do to offer Contribution form
Contribution entity - but just basic fields
Payment block
To do the latter we need an api like
PaymentForm with the actions
```
->submit
->getfields
->validate
```
```payment_processor_id ``` would be required for each of these (getfields would reply with metadata that would indicate what additional fields are required)
That should be the minimum for a totally basic processor (like paypal std, paypalpro, dummy, payment express, Mollie, Sagepay recurring in Omnipay, some IATS)
All the complexity comes in when we start adding in the js - e.g Stripe, Paypal checkout, some IATS. I think it makes much more sense to get the easy ones working first - but once they are then for phase 2 we will need at minimum
- An api function to get the JS to add with some details about it
- A js object on the pay to return the total_amount from the form - this is something Matt has been working on & would need to be thought through across forms - eg. CRM.PaymentForm.getAmount() should do the same on the angular form and on the core civicrm forms. We probably need to articulate what other functions it may or may not need
On submission we would do
```
Order.create (this can just be the same params as Contribution.create in the initial submit. At minimum it is basically contribution.create with a forced ‘Pending’ status
PaymentProcessor.pay
Payment.create
```
These exist in apiv3 & I think we would use them for nowhttps://lab.civicrm.org/dev/core/-/issues/2495Proposal - populate all requested smart groups at once2023-07-17T05:03:24ZeileenProposal - populate all requested smart groups at onceThis is tied to https://lab.civicrm.org/dev/core/-/issues/2480 but I wanted to split it off as it is related specifically to the existing load() mechanism and where I think we could tweak existing code slightly to make a function that wo...This is tied to https://lab.civicrm.org/dev/core/-/issues/2480 but I wanted to split it off as it is related specifically to the existing load() mechanism and where I think we could tweak existing code slightly to make a function that would be more efficient to support APIv4
What currently happens
load() is called for each **requested** group in turn.
If you want contacts in 8 different groups is is called 8 times.
(take a moment to realise I'm not talking about populating all groups in this gl - only those groups specifically required for the action)
it follows these steps
1) checks if it has already been loaded within this php process
2) checks if the cache date on the civicrm_group is NULL or longer ago than the smart group cache timeout
3) creates a temp table
4) populates the temp table with the contents of the saved search
5) acquires a mysql lock for the group - if not aquired it returns at this point
6) removes records for those groups from the group_contact_cache table
7) inserts rows from the temp table into the group_contact_cache table
8) drops the temp table
9) updates the group contact cache table.
10) releases the lock
What I think we could do is have a new function
```
public function loadGroups($groupIds) {
}
```
which would start with ids as a list of groups to resolve but
1) filter out any ids already loaded in this php process
2) filter out any groups with expired caches
3) attempt to acquire locks on each group, filter out any groups for which a lock cannot be aquired.
3) create a temp table
4) populate the temp table with the contents of each of the filtered groups (one by one still - we could test UNION later on if we want - out of scope)
5) remove records from the group contact table for all the groups
6) insert rows from the temp table for all the groups
7) drop the temp table
8) update the group contact cache table for all groups
9) release all the locks
While this is potentially less php cycles the thing I'm really trying to get to is less inserts into the civicrm_group_contact cache table as these can clash with each other or with delete actions. There is a risk that the number of rows inserted at once would be too many (although this already exists & it's unclear whether this WOULD actually make it worse) - we might iterate through the temporary table only inserting say 50k rows per query if this turns out to be an issue in testing.
I think @mattwire developed the process whereby we build a temp table first and then insert and I believe it has been a big improvement. I think this would get us further. I would want to focus on testing / implementing it in apiv4 rather than everywhere but api v4 could then figure out what groups it wants and request that they be loaded before joining onto the group_contact_cache table (in practice
LEFT JOIN civicrm_group_contact_cache UNION civicrm_group_contact if not all groups are smart groups).
I've also been pondering the (out of scope!!!) idea of really splitting out the temporary table building from the insert into smart group contact cache - perhaps the table could be durable & the query could use the temporary table itself and we could store the temporary table name & timestamp and somehow fire off a non-browser process to reap it into the group_contact_cache and drop the table.
@pfigel @colemanw @seamuslee @mattwire @totten @bgm @sluc23 @BjoernEhttps://lab.civicrm.org/dev/core/-/issues/2493CiviCRM 5.35.1, truncation of money values where $1,000 donation is recorded...2023-09-27T05:03:13ZeileenCiviCRM 5.35.1, truncation of money values where $1,000 donation is recorded as $1 which appears to be truncating at the thousand separator when using the Australian (AU) and US Locale@justinfreeman said to the AU-NZ crowd that there is a bug with money in order api but we can't do anything without a proper bug report
placeholder for regression to be clarified
I wasn't able to replicate based on the limited info pro...@justinfreeman said to the AU-NZ crowd that there is a bug with money in order api but we can't do anything without a proper bug report
placeholder for regression to be clarified
I wasn't able to replicate based on the limited info provided
Follow-up, here's the bug report:
> With CiviCRM 5.35.1, we (Agileware) have observed on two separate sites. The truncation of money values where $1,000 donation is recorded as $1 which appears to be truncating at the thousand separator. This has been observed on both WordPress and Drupal sites.
>
> Drupal site with Xero integration (Contribution stuff API) and a WordPress site using Caldera Form integration (Order API).
Agileware Ref: CIVICRM-1692https://lab.civicrm.org/dev/core/-/issues/2492Search-kit sql error I managed to create2023-07-30T05:03:19ZeileenSearch-kit sql error I managed to createI was able to configure this in the UI (with the custom field in the INNER JOIN) but it died on attempting to search by it
![image](/uploads/40e097316a3ed1ad97d73a411a9347e5/image.png)
Generated mysql - note requires table wmf_donor w...I was able to configure this in the UI (with the custom field in the INNER JOIN) but it died on attempting to search by it
![image](/uploads/40e097316a3ed1ad97d73a411a9347e5/image.png)
Generated mysql - note requires table wmf_donor with field lifetime_usd_total
```
SELECT GROUP_CONCAT(DISTINCT `Contribution_Contact_contact_id_01`.`display_name` SEPARATOR "�" ) AS `GROUP_CONCAT_DISTINCT_Contribution_Contact_contact_id_01_display_name`
FROM civicrm_contribution a
LEFT JOIN `wmf_donor` `wmf_donor` ON Contribution_Contact_contact_id_01.id = wmf_donor.entity_id
INNER JOIN `civicrm_contact` `Contribution_Contact_contact_id_01` ON `a`.`contact_id` = `Contribution_Contact_contact_id_01`.`id` AND `a`.`receive_date` BETWEEN "2019-07-01 00:00:00" AND "2020-06-30 23:59:59" AND `wmf_donor`.`lifetime_usd_total` >= "1000"
LEFT JOIN `civicrm_contribution` `Contribution_Contact_contact_id_01_Contact_Contribution_contact_id_01` ON `Contribution_Contact_contact_id_01`.`id` = `Contribution_Contact_contact_id_01_Contact_Contribution_contact_id_01`.`contact_id` AND `Contribution_Contact_contact_id_01_Contact_Contribution_contact_id_01`.`receive_date` BETWEEN "2020-07-01 00:00:00" AND "2021-06-30 23:59:59"
WHERE (`a`.`is_test` = "0") AND (`a`.`is_template` = "0") AND (`Contribution_Contact_contact_id_01_Contact_Contribution_contact_id_01`.`id` IS NULL)
GROUP BY `a`.`contact_id`
LIMIT 60
OFFSET 0
```
Note I tested the alternatives that might have been constructed - they 'worked' in the sense that they completed with error on my demo sites. On the server I had to kill them as they were taking more than a couple of minutes by the time I killed them. I don't think this query can work without the group filter TBH
```
SELECT GROUP_CONCAT(DISTINCT `Contribution_Contact_contact_id_01`.`display_name` SEPARATOR "�" ) AS `GROUP_CONCAT_DISTINCT_Contribution_Contact_contact_id_01_display_name`
FROM civicrm_contribution a
** INNER JOIN `civicrm_contact` `Contribution_Contact_contact_id_01` ON `a`.`contact_id` = `Contribution_Contact_contact_id_01`.`id` AND `a`.`receive_date` BETWEEN "2019-07-01 00:00:00" AND "2020-06-30 23:59:59"
INNER JOIN `wmf_donor` `wmf_donor` ON a.contact_id = wmf_donor.entity_id AND `wmf_donor`.`lifetime_usd_total` >= "1000"**
LEFT JOIN `civicrm_contribution` `Contribution_Contact_contact_id_01_Contact_Contribution_contact_id_01` ON `Contribution_Contact_contact_id_01`.`id` = `Contribution_Contact_contact_id_01_Contact_Contribution_contact_id_01`.`contact_id` AND `Contribution_Contact_contact_id_01_Contact_Contribution_contact_id_01`.`receive_date` BETWEEN "2020-07-01 00:00:00" AND "2021-06-30 23:59:59"
WHERE (`a`.`is_test` = "0") AND (`a`.`is_template` = "0") AND (`Contribution_Contact_contact_id_01_Contact_Contribution_contact_id_01`.`id` IS NULL)
GROUP BY `a`.`contact_id`
LIMIT 60
OFFSET 0
``
and
```
SELECT GROUP_CONCAT(DISTINCT `Contribution_Contact_contact_id_01`.`display_name` SEPARATOR "�" ) AS `GROUP_CONCAT_DISTINCT_Contribution_Contact_contact_id_01_display_name`
FROM civicrm_contribution a
** INNER JOIN `wmf_donor` `wmf_donor` ON a.contact_id = wmf_donor.entity_id AND `wmf_donor`.`lifetime_usd_total` >= "1000"
INNER JOIN `civicrm_contact` `Contribution_Contact_contact_id_01` ON `a`.`contact_id` = `Contribution_Contact_contact_id_01`.`id` AND `a`.`receive_date` BETWEEN "2019-07-01 00:00:00" AND "2020-06-30 23:59:59"**
LEFT JOIN `civicrm_contribution` `Contribution_Contact_contact_id_01_Contact_Contribution_contact_id_01` ON `Contribution_Contact_contact_id_01`.`id` = `Contribution_Contact_contact_id_01_Contact_Contribution_contact_id_01`.`contact_id` AND `Contribution_Contact_contact_id_01_Contact_Contribution_contact_id_01`.`receive_date` BETWEEN "2020-07-01 00:00:00" AND "2021-06-30 23:59:59"
WHERE (`a`.`is_test` = "0") AND (`a`.`is_template` = "0") AND (`Contribution_Contact_contact_id_01_Contact_Contribution_contact_id_01`.`id` IS NULL)
GROUP BY `a`.`contact_id`
LIMIT 60
OFFSET 0
```https://lab.civicrm.org/dev/core/-/issues/2491Search-kit - UI grokking issue2023-07-16T05:03:24ZeileenSearch-kit - UI grokking issueThis overlaps previously discussed issues @colemanw but I just spent 30 minutes trying to figure out what was wrong with this search & I think this illustrates it well
![image](/uploads/7e51b9b6517fc328525edde7367ebc80/image.png)
It tu...This overlaps previously discussed issues @colemanw but I just spent 30 minutes trying to figure out what was wrong with this search & I think this illustrates it well
![image](/uploads/7e51b9b6517fc328525edde7367ebc80/image.png)
It turns out the issue is that the Date Received in this 'without' is actually referencing the first contribution table not the second - so I'm searching for the equivalent of 'contributions between 2019 & 2020 that are not also between 2021 & 2022'