CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-07-04T12:55:04Zhttps://lab.civicrm.org/dev/core/-/issues/2195Custom fields of type money and drop-down list with round numbers don't updat...2023-07-04T12:55:04ZDaveDCustom fields of type money and drop-down list with round numbers don't update existing values when you change the option valueCame up reviewing https://github.com/civicrm/civicrm-core/pull/18959
For example:
1. Edit or create a custom field group for contacts.
1. Add a custom field of type money and make it a drop-down select list.
1. For the values, make cho...Came up reviewing https://github.com/civicrm/civicrm-core/pull/18959
For example:
1. Edit or create a custom field group for contacts.
1. Add a custom field of type money and make it a drop-down select list.
1. For the values, make choice 1 have value 10 and choice 2 have value 20.
1. Create a contact and pick one of the choices. Save.
1. Note that in the database in the civicrm_value_XX table this has been stored as 10.00, with the decimals. Note that in civicrm_option_value that the choice is stored as 10, without the decimals.
1. Go and edit the option value and change the value to anything else.
1. What's supposed to happen is that the contact's custom field values should be updated to match. But because of the mismatch above it doesn't change it in civicrm_value_XX and it now appears blank when viewing the contact.
I don't think this is recent.https://lab.civicrm.org/dev/core/-/issues/3440Check for matching contact on contact add form sends hardcoded fields to dupl...2023-07-06T07:25:40ZdarrickCheck for matching contact on contact add form sends hardcoded fields to duplicatecheck api callOverview
----------------------------------------
If a custom Supervised rule is created using any field not in ['first_name', 'last_name', 'nick_name', 'household_name', 'organization_name', 'email'] then clicking on the **Check for Mat...Overview
----------------------------------------
If a custom Supervised rule is created using any field not in ['first_name', 'last_name', 'nick_name', 'household_name', 'organization_name', 'email'] then clicking on the **Check for Matching Contact** button will return no results.
Reproduction steps
----------------------------------------
1. Click on **Contacts -> Find and Merge Duplicate Contacts**.
2. Click on **Add Individual Rule**
3. Add a rule with field phone, weight 10, threshold 10
4. Click on **Change rule** and set to **Supervised**
5. Click on **Contacts -> New Individual**
6. Add a contact with First Name: Bob, Last Name: Dobbs and phone 666.666.6666
7. Save the contact.
8. Click on **Contacts -> New Individual**
9. Add a contact with First Name: Bob, Last Name: Dobbs and phone 666.666.6666
10. Click on **Check for Matching Contact**
Current behaviour
----------------------------------------
A popup displays "Similar contact if found" after entering the First Name and after entering the Last Name in step 9.
After entering the duplicate phone number nothing happens.
After clicking on **Check for Matching Contact** nothing happens.
Expected behavior
----------------------------------------
After entering either First Name or Last Name nothing should happen unless the entered field is included in the Supervised rule.
A popup displaying "Similar contact if found" should happen after the phone number is entered and also after clicking on **Check for Matching Contact**
Comments
----------------------------------------
I ran across this while looking to see if I could fix any other outstanding bugs related dedupe. Was working on this one: (https://lab.civicrm.org/dev/core/-/issues/2966) I wasn't able to reproduce their issue.
It may still be useful to hard code those fields so by default the form always matches on name or email when entering those fields but then the additional fields will also be searched when added using the custom rule. As any fields not needed for the custom rule will just be ignored.https://lab.civicrm.org/dev/core/-/issues/4388Order of constructor params for QuickForms2023-07-11T16:25:57Zluke.stewartOrder of constructor params for QuickFormsThere is a bit of chaos in param order here as encounted on related issue - is this something that we should be concerned about in this point of the lifecycle using Quickforms? Maybe if it's not broken we shouldn't fix?
Below list is a...There is a bit of chaos in param order here as encounted on related issue - is this something that we should be concerned about in this point of the lifecycle using Quickforms? Maybe if it's not broken we shouldn't fix?
Below list is a result of grep and then manual copy and paste and spacing to try and match up like orderings.
```
xbutton.php: function __construct($elementName = null, $elementContent = null, $attributes = null)
advmultiselect.php: function __construct($elementName = null, $elementLabel = null,
element.php: function __construct($elementName=null, $elementLabel=null, $attributes=null)
file.php: function __construct($elementName=null, $elementLabel=null, $attributes=null)
input.php: function __construct($elementName=null, $elementLabel=null, $attributes=null)
password.php: function __construct($elementName=null, $elementLabel=null, $attributes=null)
textarea.php: function __construct($elementName=null, $elementLabel=null, $attributes=null)
text.php: function __construct($elementName=null, $elementLabel=null, $attributes=null)
hiddenselect.php:function __construct($elementName=null, $elementLabel=null, $options=null, $attributes=null)
select.php: function __construct($elementName=null, $elementLabel=null, $options=null, $attributes=null)
autocomplete.php:function __construct($elementName=null, $elementLabel=null, $options=null, $attributes=null)
date.php: function __construct($elementName=null, $elementLabel=null, $options=array(), $attributes=null)
hierselect.php: function __construct($elementName=null, $elementLabel=null, $attributes=null, $separator=null)
static.php: function __construct($elementName=null, $elementLabel=null, $text=null)
checkbox.php: function __construct($elementName=null, $elementLabel=null, $text='', $attributes=null)
advcheckbox.php: function __construct($elementName=null, $elementLabel=null, $text=null, $attributes=null, $values=null)
radio.php: function __construct($elementName=null, $elementLabel=null, $text=null, $value=null, $attributes=null)
button.php: function __construct($elementName=null, $value=null, $attributes=null)
reset.php: function __construct($elementName=null, $value=null, $attributes=null)
submit.php: function __construct($elementName=null, $value=null, $attributes=null)
link.php: function __construct($elementName=null, $elementLabel=null, $href=null, $text=null, $attributes=null)
group.php: function __construct($elementName=null, $elementLabel=null, $elements=null, $separator=null, $appendName = true, $attributes = null)
image.php: function __construct($elementName=null, $src='', $attributes=null)
header.php: function __construct($elementName=null, $text = null)
hidden.php: function __construct($elementName=null, $value='', $attributes=null)
Page.php: function __construct($formName, $method = 'post', $target = '', $attributes = null)
Controller.php: function __construct($name, $modal = true)
html.php: function __construct($text = null)
```https://lab.civicrm.org/dev/core/-/issues/3714CiviCRM Contribution Page, total amount calculation is sometimes off by one o...2023-07-18T05:40:43Zjustinfreeman (Agileware)CiviCRM Contribution Page, total amount calculation is sometimes off by one or two cents, which causes incorrect amount to be paid or transaction to fail depending on the Payment Processor being usedWhen using a **CiviCRM Contribution Page** the total amount calculation is sometimes off by one or two cents, which causes incorrect amount to be paid or transaction to fail depending on the Payment Processor being used.
In the case of ...When using a **CiviCRM Contribution Page** the total amount calculation is sometimes off by one or two cents, which causes incorrect amount to be paid or transaction to fail depending on the Payment Processor being used.
In the case of Stripe, the _correct amount_ is _pre-authorised_. When the payment is _confirmed_ the _incorrect amount_ is used and because the two amounts do not match, the **transaction fails**. As a result this bug prevents payments from working with Stripe as a payment processor for certain amounts.
# Steps to reproduce
1. Enable tax and invoicing
2. Add e.g. a 10% GST account to Member Dues
3. Create a membership prices set with one field that has three checkbox options:
4. 318.18182 (350 incl tax)
5. 22.72727 (25 incl tax)
6. 24.54545 (27 after tax - note that this would actually work out to 27.01 with the older premature rounding based off-by one issue)
7. Use this price set on a contribution page
8. Submit a test using this contribution page.
9. Note in the summary that the line totals are 350.00, 25.00, and 27.00
## Expected result
Total should be **$402**
## Actual result
Total is **$401.99** - which is one cent less than the expect value, and two cents less than the value you would get with premature rounding.
## Tested environment
Bug has been confirmed using **CiviCRM 5.52.alpha1** on https://wpmaster.demo.civicrm.org/
## Workaround
Adjust the amount to account for the rounding error. In this case, change the amount: 24.54545 to 24.5363636364 which then changes the total to $401.99. This is then matches the total on the Thank You page.
![image](/uploads/ddf3c7cbeaf3cdc4700079f79b01c7d6/image.png)
![image](/uploads/5eb33f84dfa49cad5697570a019c6936/image.png)
# Contribution page with membership options selected - correct total is shown
![image](/uploads/f35b520763e5e9f3545585f27ff42812/image.png)
# Thank you page - incorrect total is shown
![image](/uploads/d6fa209c43cbf2d721d4cf085882b1f8/image.png)
# Set up - Membership Types
![image](/uploads/01450b1f2a777f3c279e694d5ea7e007/image.png)
# Set up - Priceset
![image](/uploads/672f5c3de5b72e7817dc063388f08c38/image.png)
Agileware Ref: CIVICRM-2007https://lab.civicrm.org/dev/core/-/issues/1926Finish allowing use of SSL to connect to database2023-07-24T03:44:31ZDaveDFinish allowing use of SSL to connect to databaseFollowing on to complete what's left for https://lab.civicrm.org/dev/core/-/issues/1137, there are a couple confusing things so I'm listing them to try to get a handle on them:
| thing | comment
| ---- | ----
| [CRM_Utils_File](https://...Following on to complete what's left for https://lab.civicrm.org/dev/core/-/issues/1137, there are a couple confusing things so I'm listing them to try to get a handle on them:
| thing | comment
| ---- | ----
| [CRM_Utils_File](https://github.com/civicrm/civicrm-core/blob/6eced9e6cedd66769757a30415a4c6c99fd0d231/CRM/Utils/File.php#L327) | This block seems to do almost the same thing as CRM_Core_DAO::init(), but not completely, but I think also has a "bug" where it wipes the sql_mode instead of updating it. This is called from e.g. the upgrade process, but it's not clear yet why that would require its own thing, since civi is already installed. Will test this out and see.
| [CRM_Utils_System::authenticate](https://github.com/civicrm/civicrm-core/blob/da6c68c61ab13ba434a5061701c3acfd8676cc29/CRM/Utils/System.php#L727) | It calls DB::connect assuming it's already available, and then does nothing with it, and then a couple milliseconds later the UF class does it again. Is this just leftover from refactoring, maybe as suggested in the comment a couple lines above. Additionally, only drupal 7 seems to use it, but it's not clear if it's really necessary and why can't it just use cms functions in the cms class, which it does in drupal 8 and the others.
| install/ | Would like to leave this out if the future plan is to move to a civicrm-setup variant. Otherwise it means adding fields to the form for example.https://lab.civicrm.org/dev/core/-/issues/4463Authentication tokens: session already active - same user2023-08-03T14:23:35Zaydunsaidan.saunders@squiffle.ukAuthentication tokens: session already active - same userOverview
----------------------------------------
If a user clicks an authenticated link a second time the result is an error message stating: `HTTP 401 Cannot login. Session already active.`
See [this chat](https://chat.civicrm.org/civ...Overview
----------------------------------------
If a user clicks an authenticated link a second time the result is an error message stating: `HTTP 401 Cannot login. Session already active.`
See [this chat](https://chat.civicrm.org/civicrm/pl/n4cr3jownifexnjda3k45qhura)
and https://lab.civicrm.org/dev/core/-/issues/4462
Reproduction steps
----------------------------------------
1. Create a FormBuilder form and enable the Token option
1. Send a mail using the form token. This link includes "_authx=Bearer..."
1. Click the link on the received mail. - Should work and creates an authenticated session.
1. Click the link again - fails with `HTTP 401 Cannot login. Session already active.`
Expected behaviour
----------------------------------------
There are two scenarios depending on whether it is the same user. This bug issue relates to the case where the user is the same. See #4464 for handling this where the user is different.
It the user is the same, it should just continue to the form without error.https://lab.civicrm.org/dev/core/-/issues/4464Authentication tokens: session already active - different user2023-08-04T12:51:21Zaydunsaidan.saunders@squiffle.ukAuthentication tokens: session already active - different userOverview
----------------------------------------
See #4463 and https://lab.civicrm.org/dev/core/-/issues/4462 This improvement issue relates to handling the case where the new user is different to that of the existing session.
From [t...Overview
----------------------------------------
See #4463 and https://lab.civicrm.org/dev/core/-/issues/4462 This improvement issue relates to handling the case where the new user is different to that of the existing session.
From [this chat](https://chat.civicrm.org/civicrm/pl/n4cr3jownifexnjda3k45qhura):
@totten says:
- if the active session is same user, then no problems. just proceed.
- if the active session user differs from the requested user, then you kinda have to choose between:
- killing the old session and starting a new one
- changing the userID of the live session. (but then you're liable to leak session-data in hard-to-predict ways)
- aborting the request
For the already active session with different user - I think it would be ideal to show a prompt and confirm that they want to logout/switch-user.https://lab.civicrm.org/dev/core/-/issues/4458Error when viewing contact-info profile without "view deleted contacts" permi...2023-08-07T12:11:55ZcolemanwError when viewing contact-info profile without "view deleted contacts" permissionThe change in b7edabe813db467aff6dd1ea083d798089198655 switched the profile to use APIv4 to fetch email id for constructing an email link. The API call looks like this:
```php
$emailID = Email::get()->setOrderBy(['is_primary' => 'DES...The change in b7edabe813db467aff6dd1ea083d798089198655 switched the profile to use APIv4 to fetch email id for constructing an email link. The API call looks like this:
```php
$emailID = Email::get()->setOrderBy(['is_primary' => 'DESC'])->setWhere([['contact_id', '=', $this->_id], ['email', '=', $email], ['on_hold', '=', FALSE], ['contact_id.is_deceased', '=', FALSE], ['contact_id.is_deleted', '=', FALSE], ['contact_id.do_not_email', '=', FALSE]])->execute()->first()['id'];
```
It was reported on SE that this fails for users without "view deleted contacts", however I'm unable to reproduce.
See https://civicrm.stackexchange.com/questions/45313/invalid-field-contact-id-is-deceased-apiv4
This should have already been double-fixed by:
- [Revert "Add permission metadata to contact is_deleted field" #22203](https://github.com/civicrm/civicrm-core/pull/22203)
- [APIv4 - Silently ignore non-permissioned fields instead of throwing exceptions #20724](https://github.com/civicrm/civicrm-core/pull/20724)https://lab.civicrm.org/dev/core/-/issues/2597Tabset for manage contribution pages or manage events requires Classname to e...2023-08-07T15:13:57ZlarsssandergreenTabset for manage contribution pages or manage events requires Classname to equal pathIf you add a tab to a tabset for Configure Event, it will work fine with the last word of the class name different from the last word of the path, except for trying to save, which will save, but redirect to Manage Events instead of stayi...If you add a tab to a tabset for Configure Event, it will work fine with the last word of the class name different from the last word of the path, except for trying to save, which will save, but redirect to Manage Events instead of staying on the same tab. This is because [code here](https://github.com/civicrm/civicrm-core/blob/5db0bc3c1f54eaca4307f103a73bda596ae914d6/CRM/Event/Form/ManageEvent.php#L377) expects lastWord from CRM_myExtension_Form_lastWord to be the same as the path, i.e. /civicrm/event/manage/lastword. Everything else works as expected if these are different, except Save.
Looks like this same issue exists for [Contribution Pages](https://github.com/civicrm/civicrm-core/blob/5db0bc3c1f54eaca4307f103a73bda596ae914d6/CRM/Contribute/Form/ContributionPage.php#L427
) and probably elsewhere.
For now, I've added a warning to the docs for [the tabset hook](https://lab.civicrm.org/documentation/docs/dev/-/merge_requests/908). It would be good to fix this so the code linked above uses the path for the url instead of the class, at some point.https://lab.civicrm.org/dev/core/-/issues/4482Action schedule effective start and end dates ignored when using an absolute ...2023-08-09T13:56:33ZjamieAction schedule effective start and end dates ignored when using an absolute trigger dateHere's an action schedule that did not go as expected:
![image](/uploads/8205f1e5dbba0c9f0b056b72d4f6c261/image.png)
Notice how there is an absolute date as the "When (trigger) date" *that does not allow you to enter a time* and also a...Here's an action schedule that did not go as expected:
![image](/uploads/8205f1e5dbba0c9f0b056b72d4f6c261/image.png)
Notice how there is an absolute date as the "When (trigger) date" *that does not allow you to enter a time* and also an "Effective Start Date" that *does allow you to enter a time.*
In the [code](https://github.com/civicrm/civicrm-core/blob/4e4de633c452fd2e9a36df61bdbdb258b9f88ed7/Civi/ActionSchedule/RecipientBuilder.php#L418) we seem to ignore the effective start and end dates when an absolute date is specified.
So, this email was sent at 12:05 am on August 5th instead of 8:05 am on August 5th.
I get it. If you are specifying an absolute start date, why would you need an effective start date?
Well, in this case it's because you are fine-tuning with a time.
I see three options:
1. Change the UI so that the effective start and end dates are greyed out if you enter an absolute date. This seems like the easiest to implement.
2. Change the code so that we don't ignore the effective start and end date when using an absolute date. I'm not crazy about messing with the Action Schedule code. But, this seems like it could be pulled off without a lot of work or risk.
3. Allow setting an absolute date with a time attached to it (and also doing number two). I would rather have a root canal than be responsible for this one. But, it does seem like the best optionhttps://lab.civicrm.org/dev/core/-/issues/2595Change Log Tab Excrutiatingly Slow - Poorly Performing Query and Fix (from 8 ...2023-08-14T13:09:44ZgordanChange Log Tab Excrutiatingly Slow - Poorly Performing Query and Fix (from 8 minutes down to 4 seconds)The page takes about 8 minutes to return which is absurdly slow for anything expected to be remotely interactive. It is so slow that my client is referring to it as the "Triangle of Doom".
I tracked it down to this query:
```
INSERT IG...The page takes about 8 minutes to return which is absurdly slow for anything expected to be remotely interactive. It is so slow that my client is referring to it as the "Triangle of Doom".
I tracked it down to this query:
```
INSERT IGNORE INTO civicrm_tmp_e_logsummary_ffa3ac146126d178ade367f2a5d17bf5
SELECT activity_id, IF (entity_log_civireport.log_action = 'Insert' AND extra_table.activity_type_id = 51 , GROUP_CONCAT(entity_log_civireport.contact_id), 1) , entity_log_civireport.log_action as log_civicrm_entity_log_action, 'log_civicrm_activity_contact' as log_civicrm_entity_log_type, entity_log_civireport.log_user_id as log_civicrm_entity_log_user_id, entity_log_civireport.log_date as log_civicrm_entity_log_date, modified_contact_civireport.display_name as log_civicrm_entity_altered_contact, modified_contact_civireport.id as log_civicrm_entity_altered_contact_id, entity_log_civireport.log_conn_id as log_civicrm_entity_log_conn_id, modified_contact_civireport.is_deleted as log_civicrm_entity_is_deleted, altered_by_contact_civireport.display_name as altered_by_contact_display_name
FROM staging_civicrm.log_civicrm_activity_contact entity_log_civireport
JOIN civicrm_contact modified_contact_civireport ON (entity_log_civireport.contact_id = modified_contact_civireport.id )
JOIN staging_civicrm.log_civicrm_activity extra_table ON extra_table.id = entity_log_civireport.activity_id
LEFT JOIN civicrm_contact altered_by_contact_civireport ON (entity_log_civireport.log_user_id = altered_by_contact_civireport.id)
WHERE modified_contact_civireport.id = 338520 AND
entity_log_civireport.log_action != 'Initialization'
GROUP BY entity_log_civireport.log_conn_id,
entity_log_civireport.log_user_id,
EXTRACT(DAY_MICROSECOND FROM entity_log_civireport.log_date),
entity_log_civireport.id
ORDER BY entity_log_civireport.log_date DESC;
EXPLAIN shows:
```
```
+------+-------------+-------------------------------+--------+---------------+---------+---------+---------------------------------------------------+---------+-------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+--------+---------------+---------+---------+---------------------------------------------------+---------+-------------------------------------------------+
| 1 | SIMPLE | modified_contact_civireport | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | extra_table | ALL | NULL | NULL | NULL | NULL | 3014020 | |
| 1 | SIMPLE | entity_log_civireport | ALL | NULL | NULL | NULL | NULL | 5518537 | Using where; Using join buffer (flat, BNL join) |
| 1 | SIMPLE | altered_by_contact_civireport | eq_ref | PRIMARY | PRIMARY | 4 | staging_civicrm.entity_log_civireport.log_user_id | 1 | Using where |
+------+-------------+-------------------------------+--------+---------------+---------+---------+---------------------------------------------------+---------+-------------------------------------------------+
```
The fix passes:
First pass:
```
ALTER TABLE log_civicrm_activity_contact ADD INDEX index_activity_id (activity_id);
```
With no further changes, this alone makes the above query go from 8 minutes to 1m45s.
Explain plain:
```
+------+-------------+-------------------------------+--------+-------------------+-------------------+---------+---------------------------------------------------+---------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+--------+-------------------+-------------------+---------+---------------------------------------------------+---------+---------------------------------+
| 1 | SIMPLE | modified_contact_civireport | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | extra_table | ALL | NULL | NULL | NULL | NULL | 3014020 | Using where |
| 1 | SIMPLE | entity_log_civireport | ref | index_activity_id | index_activity_id | 5 | staging_civicrm.extra_table.id | 1 | Using where |
| 1 | SIMPLE | altered_by_contact_civireport | eq_ref | PRIMARY | PRIMARY | 4 | staging_civicrm.entity_log_civireport.log_user_id | 1 | Using where |
+------+-------------+-------------------------------+--------+-------------------+-------------------+---------+---------------------------------------------------+---------+---------------------------------+
```
Second pass:
```
ALTER TABLE log_civicrm_activity ADD INDEX index_id (id);
```
Change the JOIN order explicitly and add a hint for the query optimizer to not re-order the JOINs:
```
SELECT STRAIGHT_JOIN activity_id, IF (entity_log_civireport.log_action = 'Insert' AND extra_table.activity_type_id = 51 , GROUP_CONCAT(entity_log_civireport.contact_id), 1) , entity_log_civireport.log_action as log_civicrm_entity_log_action, 'log_civicrm_activity_contact' as log_civicrm_entity_log_type, entity_log_civireport.log_user_id as log_civicrm_entity_log_user_id, entity_log_civireport.log_date as log_civicrm_entity_log_date, modified_contact_civireport.display_name as log_civicrm_entity_altered_contact, modified_contact_civireport.id as log_civicrm_entity_altered_contact_id, entity_log_civireport.log_conn_id as log_civicrm_entity_log_conn_id, modified_contact_civireport.is_deleted as log_civicrm_entity_is_deleted, altered_by_contact_civireport.display_name as altered_by_contact_display_name
FROM civicrm_contact modified_contact_civireport
JOIN staging_civicrm.log_civicrm_activity_contact entity_log_civireport ON entity_log_civireport.contact_id = modified_contact_civireport.id
JOIN staging_civicrm.log_civicrm_activity extra_table ON extra_table.id = entity_log_civireport.activity_id
LEFT JOIN civicrm_contact altered_by_contact_civireport ON entity_log_civireport.log_user_id = altered_by_contact_civireport.id
WHERE modified_contact_civireport.id = 338520 AND
entity_log_civireport.log_action != 'Initialization'
GROUP BY entity_log_civireport.log_conn_id,
entity_log_civireport.log_user_id,
EXTRACT(DAY_MICROSECOND FROM entity_log_civireport.log_date),
entity_log_civireport.id
ORDER BY entity_log_civireport.log_date DESC;
```
New EXPLAIN:
```
+------+-------------+-------------------------------+--------+-------------------+----------+---------+---------------------------------------------------+---------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+--------+-------------------+----------+---------+---------------------------------------------------+---------+---------------------------------+
| 1 | SIMPLE | modified_contact_civireport | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | entity_log_civireport | ALL | index_activity_id | NULL | NULL | NULL | 5518537 | Using where; Using filesort |
| 1 | SIMPLE | extra_table | ref | index_id | index_id | 5 | staging_civicrm.entity_log_civireport.activity_id | 1 | |
| 1 | SIMPLE | altered_by_contact_civireport | eq_ref | PRIMARY | PRIMARY | 4 | staging_civicrm.entity_log_civireport.log_user_id | 1 | Using where |
+------+-------------+-------------------------------+--------+-------------------+----------+---------+---------------------------------------------------+---------+---------------------------------+
```
This gets it down to 4 seconds!
Since the first index we started with is no longer getting used in the final variant, we can just not add it.
Summary:
To fix "Change Log" tab taking forever to load, the following fix is needed:
Add index:
```
ALTER TABLE log_civicrm_activity ADD INDEX index_id (id);
```
Make the code emit the query modified as follows:
```
INSERT IGNORE INTO civicrm_tmp_e_logsummary_ffa3ac146126d178ade367f2a5d17bf5
SELECT STRAIGHT_JOIN activity_id, IF (entity_log_civireport.log_action = 'Insert' AND extra_table.activity_type_id = 51 , GROUP_CONCAT(entity_log_civireport.contact_id), 1) , entity_log_civireport.log_action as log_civicrm_entity_log_action, 'log_civicrm_activity_contact' as log_civicrm_entity_log_type, entity_log_civireport.log_user_id as log_civicrm_entity_log_user_id, entity_log_civireport.log_date as log_civicrm_entity_log_date, modified_contact_civireport.display_name as log_civicrm_entity_altered_contact, modified_contact_civireport.id as log_civicrm_entity_altered_contact_id, entity_log_civireport.log_conn_id as log_civicrm_entity_log_conn_id, modified_contact_civireport.is_deleted as log_civicrm_entity_is_deleted, altered_by_contact_civireport.display_name as altered_by_contact_display_name
FROM civicrm_contact modified_contact_civireport
JOIN staging_civicrm.log_civicrm_activity_contact entity_log_civireport ON entity_log_civireport.contact_id = modified_contact_civireport.id
JOIN staging_civicrm.log_civicrm_activity extra_table ON extra_table.id = entity_log_civireport.activity_id
LEFT JOIN civicrm_contact altered_by_contact_civireport ON entity_log_civireport.log_user_id = altered_by_contact_civireport.id
WHERE modified_contact_civireport.id = 338520 AND
entity_log_civireport.log_action != 'Initialization'
GROUP BY entity_log_civireport.log_conn_id,
entity_log_civireport.log_user_id,
EXTRACT(DAY_MICROSECOND FROM entity_log_civireport.log_date),
entity_log_civireport.id
ORDER BY entity_log_civireport.log_date DESC;
```
Not only does this speed it up from 8 minutes down to 4 seconds, it also doesn't wreak havoc with row locking where every row scanned gets locked by the transaction engine, potentially resulting in a massive query pile-up in the database.https://lab.civicrm.org/dev/core/-/issues/2611Change default_value column for custom fields from varchar to text2023-08-20T02:32:58ZlarsssandergreenChange default_value column for custom fields from varchar to textA custom field can be of many types, including note. For a custom field that is a note, you may want to use a default value that is longer than 255 characters, but currently the default value is stored as a varchar(255). It would make se...A custom field can be of many types, including note. For a custom field that is a note, you may want to use a default value that is longer than 255 characters, but currently the default value is stored as a varchar(255). It would make sense to change this column to text, so that longer values can be entered.
In the custom fields UI, this field would be changed to a textarea when note is selected.
For us, the use case would be storing the content of a short waiver, so that the exact value of the waiver at the time of signing can be stored on the participant for an event, but I can imagine there may be other use cases.
Will submit PR if supported.https://lab.civicrm.org/dev/core/-/issues/2641Users affected by ACLs intermittently lose access to contacts2023-08-22T04:01:30ZjhungerfordUsers affected by ACLs intermittently lose access to contactsOverview
----------------------------------------
When a user's permission to view Contacts is affected by multiple ACLs, they will intermittently lose access to some of them, especially directly after editing a record.
Reproduction ste...Overview
----------------------------------------
When a user's permission to view Contacts is affected by multiple ACLs, they will intermittently lose access to some of them, especially directly after editing a record.
Reproduction steps
----------------------------------------
1. Create a test User who is not an Administrator and does not have "view all contacts" or "edit all contacts" permission
2. Create two Groups (of type Access Control List), and add the test User to both groups
3. Create two Groups (not of ACL type) with distinct names like "Contact Group1" and "Contact Group2"
4. Create two Contacts and add one to each of the Groups created in step 3
5. Create two ACLs, each giving users in one of the ACL groups access to one of the Contact groups
6. Log in as the test User and repeatedly edit the Nickname (or any other data) on both of the test Contacts until you encounter a failure ("Error You do not have the necessary permission to view this contact").
It will rarely take more than ten edits before you encounter a failure, and the failures will never occur for one of the ACL groups.
Current behaviour
----------------------------------------
Sometimes it works, sometimes the user loses permission during the save or reload operation, resulting in the error message below. Our users reported that usually the data would be saved, but sometimes not (e.g. when saving some Activities on a Case). I have not been able to replicate the data loss scenario.
```
Error
You do not have the necessary permission to view this contact.
```
Expected behaviour
----------------------------------------
The data should be saved without error messages appearing (and this is what happens some of the time).
Environment information
----------------------------------------
* __Browser:__ Mostly tested on Chromium, most recently version 90.0.4430.93
* __CiviCRM:__ Tested on 5.33.5 and prior ESR versions
* __PHP:__ 7.3
* __CMS:__ Drupal 7.80
* __Database:__ 10.3.27-MariaDB
* __Web Server:__ Apache
Comments
----------------------------------------
We have mitigated the problem with this patch (minus the watchdog line with the FIXME comment):
https://github.com/AsylumSeekersCentre/civicrm-core/commit/030ecb6e31ed17b9fd01716cb772f13bb60bf455
With this in place, I've been unable to reproduce the error in manual testing. If it still happens, it's very rare. However, I'm not sure that this is a very "proper" solution so I'm opening the issue without a pull request for discussion.
The watchdog line from that patch usually produces output like this:
```
( ( `contact_a`.id IN ( SELECT contact_id FROM civicrm_group_contact WHERE group_id IN (5, 6) AND status = 'Added' ) ) ) AND (contact_a.is_deleted = 0)
```
On the occasions when the permission failure occurs, it instead looks like this:
```
( ( `contact_a`.id IN ( SELECT contact_id FROM civicrm_group_contact WHERE group_id IN (5) AND status = 'Added' ) ) ) AND (contact_a.is_deleted = 0)
```
It has somehow lost one of the groups it should be checking for. It always seems to retain the same one, which I think is why it works if the user is only affected by one ACL, and the bug only shows up if users are affected by multiple ACLs.https://lab.civicrm.org/dev/core/-/issues/4442Add action to convert smart group to regular group2023-08-31T16:33:25ZyashodhaAdd action to convert smart group to regular groupThere are cases when smart groups have actually run their course. The list is not going to change after a while so no need keeping the group smart esp when the criteria is quite complex and increases the load time. The idea here is keep ...There are cases when smart groups have actually run their course. The list is not going to change after a while so no need keeping the group smart esp when the criteria is quite complex and increases the load time. The idea here is keep only those groups that are needed as smart/dynamic so that it prevents unnecessary process of re-calculating the groups that just shouldn't be if deletion of the group is not an option.
In such cases, provide the ability to convert smart group to regular group.
Proposal : Provide action link for smart groups to convert to regular group.
- refresh the smart group
- move from the group contact cache to group contacts
- unset the saved search on the groupyashodhayashodhahttps://lab.civicrm.org/dev/core/-/issues/2572Prevent unintentional removal from group for anonymous users filing out profiles2023-09-01T19:44:06ZlarsssandergreenPrevent unintentional removal from group for anonymous users filing out profilesCurrently, if an anonymous contact submits a profile that contains a groups field and the profile submission is deduped on submission with an existing contact, the group selections in the profile overwrite the contact's previous groups, ...Currently, if an anonymous contact submits a profile that contains a groups field and the profile submission is deduped on submission with an existing contact, the group selections in the profile overwrite the contact's previous groups, for the groups that are included in the profile (i.e. the public groups). The result is that if someone is subscribed to a mailing list and fills in a profile anonymously and doesn't check the same group name, they will be unsubscribed from the group. This is definitely not what users expect! I think it's clear that checking the box will add you to the group, but I doubt many expect that they will be removed from a group if they don't check the box that they've checked in the past.
This is also not what happens with other profile fields that are left blank, nor is it what happens if two contacts are merged manually.
I propose to ignore group checkboxes that are not checked in a profile submitted anonymously. No change for profiles submitted with known cids (as these will have the appropriate checkboxes already checked, allowing the contact to unsubscribe by unchecking them if desired). Currently, all group ids in the profile are being [passed here](https://github.com/civicrm/civicrm-core/blob/5db0bc3c1f54eaca4307f103a73bda596ae914d6/CRM/Contact/BAO/GroupContact.php#L534), removing the contact from groups that were unchecked. Will submit PR if supported.
EDIT: Here is some discussion on StackExchange: [Unwanted unsubscribe in profile](https://civicrm.stackexchange.com/questions/39403/unwanted-unsubscribe-in-profile/39406), [Anonymous user filing out a profile unintentionally remove themselves from groups](https://civicrm.stackexchange.com/questions/29337/anonymous-user-filing-out-a-profile-unintentionally-remove-themselves-from-group)https://lab.civicrm.org/dev/core/-/issues/4552Add scheduled job form doesn't go to the right place after save or delete2023-09-06T09:36:22ZDaveDAdd scheduled job form doesn't go to the right place after save or deleteI'm not sure how often people use the UI to add scheduled jobs - I was just testing something and came across this. But what happens is it will create the job but then just go to a blank form. After deleting a scheduled job, it does dele...I'm not sure how often people use the UI to add scheduled jobs - I was just testing something and came across this. But what happens is it will create the job but then just go to a blank form. After deleting a scheduled job, it does delete it but just reloads the confirmation screen.https://lab.civicrm.org/dev/core/-/issues/4544Older sites missing index for `is_deceased` on `civicrm_contact`2023-09-06T13:56:12ZwmortadaOlder sites missing index for `is_deceased` on `civicrm_contact`This is a follow up from Eileen's comment about a missing index in https://lab.civicrm.org/dev/core/-/issues/4539#note_150479.
The index for `is_deceased` was added in CiviCRM 4.7 and is defined in [xml/schema/Contact/Contact.xml](https...This is a follow up from Eileen's comment about a missing index in https://lab.civicrm.org/dev/core/-/issues/4539#note_150479.
The index for `is_deceased` was added in CiviCRM 4.7 and is defined in [xml/schema/Contact/Contact.xml](https://github.com/civicrm/civicrm-core/blob/f068222ac25a5a393e9238b617ae91a4d9d29797/xml/schema/Contact/Contact.xml#L722-L726). However, there doesn't appear to be an upgrade task to add this index to existing sites. As a result any site installed before CiviCRM 4.7 is likely to be missing this index. Checking our client sites, I can see a mix of sites that do and don't have it.
The MR where this index was added is: https://github.com/civicrm/civicrm-core/pull/10489
There was a question from Tim about whether there should be an upgrader. There was a later comment from Eileen about whether this index should be removed.
So I guess the question is whether we want this index or not:
- if we **do** want this index we should add an upgrade task so that older sites have this index
- if we **don't** want this index we should add an upgrade task to remove it from sites that have ithttps://lab.civicrm.org/dev/core/-/issues/4557Proposal - Allow empty profile field labels instead of null2023-09-08T15:11:53ZshaneonabikeProposal - Allow empty profile field labels instead of nullPresently, when you create a Custom Profile and add fields you need add a label. In some cases, having a large question makes the label kind of awkward and crammed. It makes for a bad UX experience I believe.
So the workaround by most o...Presently, when you create a Custom Profile and add fields you need add a label. In some cases, having a large question makes the label kind of awkward and crammed. It makes for a bad UX experience I believe.
So the workaround by most of the designers I work with is to simply remove the ```label``` and then drop the question in the Pre text. This works just dandy, except that CiviCRM is adding ```null``` for empty texts. The only way to prevent this is to modify the label in the DB.
I would like to propose that we remove the code that is setting this text to null. I could make a quick patch. I realize the label is being used to create the actual field name, but in the case of the Profile the field name is already created.
![Selection_007](/uploads/48ab9decd2a6b1c3bb5a380a8278f12d/Selection_007.png)https://lab.civicrm.org/dev/core/-/issues/3559CiviCRM bounce processing, processed emails are never deleted from the mailbo...2023-09-08T22:34:57Zjustinfreeman (Agileware)CiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little valueCiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little value.
Possible solutions:
1. Once a bounce email has been processed it should be deleted from the mailbox,...CiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little value.
Possible solutions:
1. Once a bounce email has been processed it should be deleted from the mailbox, or
1. A new Scheduled Job process should be implemented which deletes all processed email which is older than 30 days
Relates to https://lab.civicrm.org/dev/mail/-/issues/108
Agileware Ref: CIVICRM-1977https://lab.civicrm.org/dev/core/-/issues/3586Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process ...2023-09-12T02:13:33Zjustinfreeman (Agileware)Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process any emails if a single email is sent to the bounce mailbox with an invalid returnPathCiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service ema...CiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service email" causing all other emails to remain unprocessed. CiviCRM mail reports are therefore incorrect and ultimately the CiviCRM Fetch Bounces scheduled job fails to perform the job it was designed to do.
Example invalid returnPaths as observed in production:
```
x2108-arfsfufwyfgkvni2x7ww5wvac3eifc2wx3vr1fcwbbysieqppis4ytnyip09xcohck9s1vrm77nbg4xh431tthdsmtmfvfpa@rspf.mxthunderstruck.net
```
```
SRS0=3Ypr33=QM=crm.nothappyjohn.org.au=bounce+b.25211.6039846.eb7e7e8ba282f9d0@airquotes.com.au
```
This problem has been highlighted by the introduction of "Scheduled Job Failures" feature which raises the visibility of this type of problem.
Bounce mailboxes have been observed with 10k to 50k unprocessed emails, depending on the time when the "Denial of service email" was received.
Agileware Ref: CIVICRM-1970