CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-06-18T00:43:47Zhttps://lab.civicrm.org/dev/core/-/issues/3311Proposal - store metadata on membership renewal on line item2023-06-18T00:43:47ZeileenProposal - store metadata on membership renewal on line item~~UPDATE - discussion on this is converging on adding a new json field 'metdata' to the line item table, permitting data related to that line item to be stored at point of order, and used when confirming. (UPDATE - this is not really the...~~UPDATE - discussion on this is converging on adding a new json field 'metdata' to the line item table, permitting data related to that line item to be stored at point of order, and used when confirming. (UPDATE - this is not really the convergence now - proposal updated below)~~
We have a few issues open ( https://lab.civicrm.org/dev/membership/-/issues/28 , https://lab.civicrm.org/dev/core/-/issues/1402 , https://lab.civicrm.org/dev/financial/-/issues/53 ) that are to my mind blocked by us not having a data model that holds the user's intent when renewing. The focus of this gl is the data model and exposing it via the back office renewal form & doing what is needed in the apis.
**The problem**
When renewing an expired membership the back office user has a fairly blunt way of influencing the calculated dates - they can set the 'renewal date'. However, if the payment is not made in the same form submission that intent is not captured anywhere and cannot be respected when completing the transaction. Likewise the receipt text and number of renewal terms are not captured.
**The proposal data model**
1) add membership_num_terms to civicrm_line_item table.
2) add 'payment_completion_metadata' to the ~~civicrm_line_item~~ civicrm_contribution table for data that is only used in the order->payment flow. Limit what can be stored here to tested values used in that flow - likely to be the message details and effective date the renewal form uses but doesn't sotre.
~~1. New field added to civicrm_line_item called 'metadata' which stores information required to complete the membership renewal - e.g. the field might store ``json_encode(['start_date' => '2010-01-01', 'end_date' => '2020-01-01', 'membership_status_id' => 'Current', 'membership_num_terms' => 2']~~
~~- note this is the new proposal - but I'm wondering what happens if it's renewed in the meantime & then paid~~
~~- New status 'Pending renewal' used when there is a need to record the dates (is_current_member = 0, is_reserved = 1, is_admin = 1 too I think)~~
~~- When a contribution linked to a membership in 'Pending renewal' is completed the status is updated but the dates are not changed~~
~~- If a membership has 'Pending renewal' status and status_override_end_date is set then any processing of that membership after that date would result in it being reverted to previous dates and status - this information should be available in the membership_log table.~~
**Proposed form exposure**
No proposed form changes - this is just storing data already submitted in the renewal form.
~~- the logic could be used outside of existing core forms but within core I think this is something we are exposing to the back-office user doing a renewal - they already have considerable ability to edit dates. So I think this is a form layer thing not some site-wide setting.
- when renewing an expired membership the renewal form would present the user with what will happen to the start date, end date etc if payment is received on that day and offer a checkbox to specify dates - if that is checked then date fields would be exposed an on save the Pending Renewal status would be used with the selected dates.
- potentially the user can also enter the date the 'offer' is available until - ie status_override_end_date~~
**Edge cases**
1. Multiple terms
We would add an extra field membership_num_terms to the line items column
~~https://lab.civicrm.org/dev/membership/-/issues/28 throws up the edge case that more than one renewal term might be selected. Both @jptillman and I assumed that the right way to represent that would be setting qty = n and then picking that when confirming. @andrewhunt didn't think that was the right assumption https://github.com/civicrm/civicrm-core/pull/18618. If we come down on NOT setting qty then we ALSO need a way to record intent for non-expired renewals - this could be an extra membership_num_terms column on the line_item table. (I'm still inclined to my original assumption but more points of view needed).~~
2. About to expire rather than actually expired
Out of scope for now - just do what the form currently does
~~I think we would still manage this in the form ie - "the membership is due to to expire in 4 days, if payment is received after that then .... ' and present the same options as for an actually expired membership~~https://lab.civicrm.org/dev/core/-/issues/2382Allow disabling of confirmation page for paid events2022-10-16T01:24:38ZlarsssandergreenAllow disabling of confirmation page for paid eventsCurrently, a paid event must have a confirmation page. In simple paid event registration cases, it would be sometimes be preferable to skip the confirmation page (and potential drop-off of registrants or confusion from those who don't re...Currently, a paid event must have a confirmation page. In simple paid event registration cases, it would be sometimes be preferable to skip the confirmation page (and potential drop-off of registrants or confusion from those who don't realize that the confirmation page wasn't the end of the process) and simply register the participant directly.
Note that you can skip the confirmation page on a contribution page and this is a very similar situation (people are making a selection and filling in their payment details). We find our supporters are likely to be confused by multiple, verbose pages and the confirmation page definitely adds complexity that isn't needed in some cases. Are there any downsides to giving users the option to set up events without a confirmation page?https://lab.civicrm.org/dev/core/-/issues/2410Image custom field type2023-01-30T20:19:36ZMichael McAndrewImage custom field typeOverview
----------------------------------------
We want to add a custom field type of image that can be used to store images as part of a CiviCRM entity.
In terms of how this might be different to the file field type, I think the main...Overview
----------------------------------------
We want to add a custom field type of image that can be used to store images as part of a CiviCRM entity.
In terms of how this might be different to the file field type, I think the main consideration is how these fields would be displayed (resizing, thumbnails, etc.)
Example use-case
----------------------------------------
1. Store (one or more) images related to a contact as part of the contact record.
2. *** feel free to add more***
Current behaviour
----------------------------------------
At the moment, there is a single core field for a contact image (e.g. a headshot of an indiviual or a company logo). No other images can be added. Images can be added as files but are not displayed as one would expect an image to be displayed.
Proposed behaviour
----------------------------------------
People should be able to define image fields and upload images to them. The original should be retained and resized images should also be created (e.g. for thumbnails).
(Wireframes, mockups and more thinking required.)
Comments
----------------------------------------
Some relevant content (previous workarounds, etc.):
- https://drupal.stackexchange.com/questions/220599/how-to-add-image-field-in-civicrm
- https://civicrm.stackexchange.com/questions/15900/how-to-add-image-field-in-civicrmhttps://lab.civicrm.org/dev/core/-/issues/2481Discussion drop support for php 7.2 and mysql 5.62023-01-04T18:01:36ZeileenDiscussion drop support for php 7.2 and mysql 5.6We have been dropping support for EOL php & mysql on ESR releases because that gives sites the option of delaying for another 6 months for just $120. So far we are not aware of any sites opting for the ESR for that reason but we are crea...We have been dropping support for EOL php & mysql on ESR releases because that gives sites the option of delaying for another 6 months for just $120. So far we are not aware of any sites opting for the ESR for that reason but we are creating the option.
With 5.39 coming up we should consider whether we want to drop support for
php 7.2 (EOL Dec 2020)
mysql 5.6 (EOL Feb 2021)
Note that historically supporting older versions of mysql has been low effort although some features are not supported. However, it might be holding us back from using new features
Supporting older versions of php has historically been more problematic as it has been at the expense of supporting newer versions. In particular we often find packages don't support latest and EOL versions.
So dropping support for older php versions is more helpful than older mysql versions.
Last ESR we considered dropping support for php 7.1 and mysql 5.6. We did the former and not the latter. At the time the stats were:
mysql 5.6 : 805
php 7.1 : 658.
Maria DB 10.1 : 943
Currently they are
mysql 5.6 : 603
php 7.2 : 2846
Maria DB 10.1 : 526
Based on those numbers I think it's clearly too early for php 7.2 and I would be OK targetting both for the NEXT ESR (5.45)https://lab.civicrm.org/dev/core/-/issues/2484Create apiv4 payment api2023-07-13T08:07:37ZeileenCreate apiv4 payment apiIn order to be able to search for payments we need to be able to search for payment rows (rows with is_payment = TRUE) in the financial_trxn table and for the entity bridge to be able to find it's way from there to the contribution table...In order to be able to search for payments we need to be able to search for payment rows (rows with is_payment = TRUE) in the financial_trxn table and for the entity bridge to be able to find it's way from there to the contribution table.
This gives us the ability to say find all contributions with a payment with a payment instrument of 'Check' or find the contribution where the payment's trxn value is y
The scope I would be looking for is
1) can use any of the fields in financial_trxn as a filter
2) can logically link to the contribution (probably through bridge entities)
3) forces 'is_payment' as a filter (only gets payments)
4) the create action calls Payment.create
5) the update & delete actions are blocked
This effectively gives us parity with the v3 api albeit with more options provided through the apiv4 layer itself.
Note that I think this is likely blocked on other feature requests which @mattwire will add notes onhttps://lab.civicrm.org/dev/core/-/issues/2494Search-kit wishlist - exposed join filters2022-11-07T04:17:02ZeileenSearch-kit wishlist - exposed join filtersSo on my effort to get a workable lybunt I find that what I really want is to be able to expose the join filter in the afform -ie
```
SELECT Contacts
WITH contributions > {{{total_amount}}} IN DATE RANGE {{select_date}}
WITHOUT contribu...So on my effort to get a workable lybunt I find that what I really want is to be able to expose the join filter in the afform -ie
```
SELECT Contacts
WITH contributions > {{{total_amount}}} IN DATE RANGE {{select_date}}
WITHOUT contributions IN DATE RANGE {{select_date}}
```
Obviously the other filter issue (https://lab.civicrm.org/dev/core/-/issues/2405) is a prerequisite on thishttps://lab.civicrm.org/dev/core/-/issues/2546Add Settings, Disable, Delete buttons to group contacts listing page2023-09-14T15:09:10ZlarsssandergreenAdd Settings, Disable, Delete buttons to group contacts listing pageCurrently, you can only change settings, disable or delete a group from the Manage Groups page. It would make sense to be able to change the settings at least on the page that lists all the contacts in a group (disable and delete are les...Currently, you can only change settings, disable or delete a group from the Manage Groups page. It would make sense to be able to change the settings at least on the page that lists all the contacts in a group (disable and delete are less important, but I think they might as well be added as well). I often find I click into a group when I want to make it a mailing list and then realize I have to go back to the Manage Groups page and click settings.
I think this would make the most sense at the top of the page, maybe right below the "Add Contacts to this group" button or alternately, just above select records. Relatedly, I would also propose to put the "Edit Smart Group Criteria for this group" and "Add Contacts to this group" buttons on the same line, as they are currently on two.
For reference:
![image](/uploads/b7ed25111565401ddf2ae295795f3b8c/image.png)
I'll give this a go if it is supported.https://lab.civicrm.org/dev/core/-/issues/2559Cannot get Auth Code in Oauth2 from Microsoft Azure Application2023-01-10T14:28:11ZspalmstromCannot get Auth Code in Oauth2 from Microsoft Azure ApplicationOverview
----------------------------------------
You cannot get an Auth Code in Oath2 from a single-tenant Microsoft Azure Application because the access token string is
```
https://login.microsoftonline.com/common/oauth2/v2.0/token
``...Overview
----------------------------------------
You cannot get an Auth Code in Oath2 from a single-tenant Microsoft Azure Application because the access token string is
```
https://login.microsoftonline.com/common/oauth2/v2.0/token
```
when it should be:
```
https://login.microsoftonline.com/<tenant ID>/oauth2/v2.0/token
```
Reproduction steps
----------------------------------------
1. Click on Admin -> Oauth2 Administration
1. Select Microsoft Exchange Online
1. Click on Add token and enter an MS account
Current behaviour
----------------------------------------
```
AADSTS50194: Application '226037fb-d13a-4f81-ba32-561601248bea'(MissionAssist Mail) is not configured as a multi-tenant application. Usage of the /common endpoint is not supported for such applications created after '10/15/2018'. Use a tenant-specific endpoint or configure the application to be multi-tenant.
```
Expected behaviour
----------------------------------------
A token should be added.
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ _Edge_ but probably not relevant
* __CiviCRM:__ _5.36.1_ <!-- If this problem relates to an upgrade, then specify both old and new versions -->
* __PHP:__ _7.4.16__ but probably not relevant
* __CMS:__ _Drupal 9.1.7_ but probably not relevant.
* __Database:__ _MySQL 8.0.24_ but probably not relevant
* __Web Server:__ _IIS_ but probably not relevant.
Comments
----------------------------------------
It would be good if the setup could prompt for the tenant ID>5.59.0https://lab.civicrm.org/dev/core/-/issues/2562Simplify when scheduled reminders are sent or not sent for events2023-08-07T15:14:25ZlarsssandergreenSimplify when scheduled reminders are sent or not sent for eventsFor event scheduled reminders, you can set emails to be sent on specific days or N hours or days before or after an event (or before or after registration ends or starts). For emails sent on a specific day, this is straightforward. They ...For event scheduled reminders, you can set emails to be sent on specific days or N hours or days before or after an event (or before or after registration ends or starts). For emails sent on a specific day, this is straightforward. They are sent on that day and anyone who registers after that day does not receive the email.
But for emails scheduled N hours or days before or after an event, the behaviour is less clear. My testing indicates contacts registered soon after an event has ended can still receive emails scheduled to go out 24 hours before the event start or 30 hours before the event end (but contacts registered somewhat later won't receive these emails). Similarly, contacts registered a day after an event ends will receive an email scheduled to go out an hour after the event start. I can't find any documentation that tells users what will happen when they schedule these reminders, though there are a few questions on Stack Exchange wondering about what actually does happen (opinions vary, which is a bad sign).
I think the clearest and best option would be to simply send scheduled reminders at the time they are scheduled and not afterwards. For one, this is simple and probably what users assume happens. It's certainly what our users think. Secondly, let's say you have an event with a confirmation email, one scheduled reminder a day before the event and one 2 hours before the event: with the current system a person who registers at the last minute will receive three emails in short order, which doesn't make sense. Thirdly, if a user registers someone in the backend for an event after the event has ended, just to record their attendance, they probably don't want that person to automatically receive emails previously scheduled to go out right after the event start or similar (this could be avoided by using a different status for the post-event registration and the scheduled reminder, but that's a pretty fine point that users won't think about).
In short, I think there are so many potential pitfalls with a system that sends scheduled reminders after the time set on the reminder has passed, especially when none of this behaviour is apparent to users when they set up the reminders. To me, this outweighs the potential convenience of having reminders sent to late registrants, so we should simply not send scheduled reminders whose time has passed.
I haven't dug into the code yet to see how this all works, but I'll do that and submit a PR as long as this is supported.
EDIT: removed aside on timing of scheduled reminders.https://lab.civicrm.org/dev/core/-/issues/2565HTML editor for additional event confirmation email text2023-08-02T21:16:19ZlarsssandergreenHTML editor for additional event confirmation email textIt would be nice to be able to have an HTML editor for the additional event confirmation text entered on the Manage Event Page that gets inserted into the System Template email template. The benefit would be to be able to easily add link...It would be nice to be able to have an HTML editor for the additional event confirmation text entered on the Manage Event Page that gets inserted into the System Template email template. The benefit would be to be able to easily add links, etc. I understand this will be a complicated one, but it does seem worthwhile as almost every other similar field is HTML, so this one really stands out. [See previous issue.](https://lab.civicrm.org/dev/core/-/issues/852#note_58464)
How it works now: Text from the event confirmation email text is added into the event confirmation message template with some minimal tags added. I see a <p> wrapper around the text and <br /> inserted for newlines, at a cursory glance.
Here's my guess at what needs to happen:
1. Add HTML editor to Event Configuration
2. Add HTML editor on the New Event Registration page (which is also Add Participant from events or Add Event Registration for a contact), so that the message can be edited before sending.
3. Inject content as HTML into Message Templates, i.e. remove code that adds tags, wrap code in div or something.
4. Convert all existing content in the DB with the same process now taken to inject the text into the Message Template or handle the content injection into the Message Template with two cases based on text/HTML.
The first three should be relatively simple, but the last one there seems like the tricky part. How has this has been handled in the past?https://lab.civicrm.org/dev/core/-/issues/2572Prevent unintentional removal from group for anonymous users filing out profiles2023-09-01T19:44:06ZlarsssandergreenPrevent unintentional removal from group for anonymous users filing out profilesCurrently, if an anonymous contact submits a profile that contains a groups field and the profile submission is deduped on submission with an existing contact, the group selections in the profile overwrite the contact's previous groups, ...Currently, if an anonymous contact submits a profile that contains a groups field and the profile submission is deduped on submission with an existing contact, the group selections in the profile overwrite the contact's previous groups, for the groups that are included in the profile (i.e. the public groups). The result is that if someone is subscribed to a mailing list and fills in a profile anonymously and doesn't check the same group name, they will be unsubscribed from the group. This is definitely not what users expect! I think it's clear that checking the box will add you to the group, but I doubt many expect that they will be removed from a group if they don't check the box that they've checked in the past.
This is also not what happens with other profile fields that are left blank, nor is it what happens if two contacts are merged manually.
I propose to ignore group checkboxes that are not checked in a profile submitted anonymously. No change for profiles submitted with known cids (as these will have the appropriate checkboxes already checked, allowing the contact to unsubscribe by unchecking them if desired). Currently, all group ids in the profile are being [passed here](https://github.com/civicrm/civicrm-core/blob/5db0bc3c1f54eaca4307f103a73bda596ae914d6/CRM/Contact/BAO/GroupContact.php#L534), removing the contact from groups that were unchecked. Will submit PR if supported.
EDIT: Here is some discussion on StackExchange: [Unwanted unsubscribe in profile](https://civicrm.stackexchange.com/questions/39403/unwanted-unsubscribe-in-profile/39406), [Anonymous user filing out a profile unintentionally remove themselves from groups](https://civicrm.stackexchange.com/questions/29337/anonymous-user-filing-out-a-profile-unintentionally-remove-themselves-from-group)https://lab.civicrm.org/dev/core/-/issues/3204Add recurring contributions to contribution reports2022-09-16T21:11:08ZlarsssandergreenAdd recurring contributions to contribution reportsIt would be very convenient to be able to do filter by recurring contributions for reports, add recurring contributions as a column or even group by.
[Here's a PR that adds this for the Contribution Summary report.](https://github.com/c...It would be very convenient to be able to do filter by recurring contributions for reports, add recurring contributions as a column or even group by.
[Here's a PR that adds this for the Contribution Summary report.](https://github.com/civicrm/civicrm-core/pull/20168) I will also implement in other relevant reports once this has been reviewed.https://lab.civicrm.org/dev/core/-/issues/2595Change Log Tab Excrutiatingly Slow - Poorly Performing Query and Fix (from 8 ...2023-08-14T13:09:44ZgordanChange Log Tab Excrutiatingly Slow - Poorly Performing Query and Fix (from 8 minutes down to 4 seconds)The page takes about 8 minutes to return which is absurdly slow for anything expected to be remotely interactive. It is so slow that my client is referring to it as the "Triangle of Doom".
I tracked it down to this query:
```
INSERT IG...The page takes about 8 minutes to return which is absurdly slow for anything expected to be remotely interactive. It is so slow that my client is referring to it as the "Triangle of Doom".
I tracked it down to this query:
```
INSERT IGNORE INTO civicrm_tmp_e_logsummary_ffa3ac146126d178ade367f2a5d17bf5
SELECT activity_id, IF (entity_log_civireport.log_action = 'Insert' AND extra_table.activity_type_id = 51 , GROUP_CONCAT(entity_log_civireport.contact_id), 1) , entity_log_civireport.log_action as log_civicrm_entity_log_action, 'log_civicrm_activity_contact' as log_civicrm_entity_log_type, entity_log_civireport.log_user_id as log_civicrm_entity_log_user_id, entity_log_civireport.log_date as log_civicrm_entity_log_date, modified_contact_civireport.display_name as log_civicrm_entity_altered_contact, modified_contact_civireport.id as log_civicrm_entity_altered_contact_id, entity_log_civireport.log_conn_id as log_civicrm_entity_log_conn_id, modified_contact_civireport.is_deleted as log_civicrm_entity_is_deleted, altered_by_contact_civireport.display_name as altered_by_contact_display_name
FROM staging_civicrm.log_civicrm_activity_contact entity_log_civireport
JOIN civicrm_contact modified_contact_civireport ON (entity_log_civireport.contact_id = modified_contact_civireport.id )
JOIN staging_civicrm.log_civicrm_activity extra_table ON extra_table.id = entity_log_civireport.activity_id
LEFT JOIN civicrm_contact altered_by_contact_civireport ON (entity_log_civireport.log_user_id = altered_by_contact_civireport.id)
WHERE modified_contact_civireport.id = 338520 AND
entity_log_civireport.log_action != 'Initialization'
GROUP BY entity_log_civireport.log_conn_id,
entity_log_civireport.log_user_id,
EXTRACT(DAY_MICROSECOND FROM entity_log_civireport.log_date),
entity_log_civireport.id
ORDER BY entity_log_civireport.log_date DESC;
EXPLAIN shows:
```
```
+------+-------------+-------------------------------+--------+---------------+---------+---------+---------------------------------------------------+---------+-------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+--------+---------------+---------+---------+---------------------------------------------------+---------+-------------------------------------------------+
| 1 | SIMPLE | modified_contact_civireport | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | extra_table | ALL | NULL | NULL | NULL | NULL | 3014020 | |
| 1 | SIMPLE | entity_log_civireport | ALL | NULL | NULL | NULL | NULL | 5518537 | Using where; Using join buffer (flat, BNL join) |
| 1 | SIMPLE | altered_by_contact_civireport | eq_ref | PRIMARY | PRIMARY | 4 | staging_civicrm.entity_log_civireport.log_user_id | 1 | Using where |
+------+-------------+-------------------------------+--------+---------------+---------+---------+---------------------------------------------------+---------+-------------------------------------------------+
```
The fix passes:
First pass:
```
ALTER TABLE log_civicrm_activity_contact ADD INDEX index_activity_id (activity_id);
```
With no further changes, this alone makes the above query go from 8 minutes to 1m45s.
Explain plain:
```
+------+-------------+-------------------------------+--------+-------------------+-------------------+---------+---------------------------------------------------+---------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+--------+-------------------+-------------------+---------+---------------------------------------------------+---------+---------------------------------+
| 1 | SIMPLE | modified_contact_civireport | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | extra_table | ALL | NULL | NULL | NULL | NULL | 3014020 | Using where |
| 1 | SIMPLE | entity_log_civireport | ref | index_activity_id | index_activity_id | 5 | staging_civicrm.extra_table.id | 1 | Using where |
| 1 | SIMPLE | altered_by_contact_civireport | eq_ref | PRIMARY | PRIMARY | 4 | staging_civicrm.entity_log_civireport.log_user_id | 1 | Using where |
+------+-------------+-------------------------------+--------+-------------------+-------------------+---------+---------------------------------------------------+---------+---------------------------------+
```
Second pass:
```
ALTER TABLE log_civicrm_activity ADD INDEX index_id (id);
```
Change the JOIN order explicitly and add a hint for the query optimizer to not re-order the JOINs:
```
SELECT STRAIGHT_JOIN activity_id, IF (entity_log_civireport.log_action = 'Insert' AND extra_table.activity_type_id = 51 , GROUP_CONCAT(entity_log_civireport.contact_id), 1) , entity_log_civireport.log_action as log_civicrm_entity_log_action, 'log_civicrm_activity_contact' as log_civicrm_entity_log_type, entity_log_civireport.log_user_id as log_civicrm_entity_log_user_id, entity_log_civireport.log_date as log_civicrm_entity_log_date, modified_contact_civireport.display_name as log_civicrm_entity_altered_contact, modified_contact_civireport.id as log_civicrm_entity_altered_contact_id, entity_log_civireport.log_conn_id as log_civicrm_entity_log_conn_id, modified_contact_civireport.is_deleted as log_civicrm_entity_is_deleted, altered_by_contact_civireport.display_name as altered_by_contact_display_name
FROM civicrm_contact modified_contact_civireport
JOIN staging_civicrm.log_civicrm_activity_contact entity_log_civireport ON entity_log_civireport.contact_id = modified_contact_civireport.id
JOIN staging_civicrm.log_civicrm_activity extra_table ON extra_table.id = entity_log_civireport.activity_id
LEFT JOIN civicrm_contact altered_by_contact_civireport ON entity_log_civireport.log_user_id = altered_by_contact_civireport.id
WHERE modified_contact_civireport.id = 338520 AND
entity_log_civireport.log_action != 'Initialization'
GROUP BY entity_log_civireport.log_conn_id,
entity_log_civireport.log_user_id,
EXTRACT(DAY_MICROSECOND FROM entity_log_civireport.log_date),
entity_log_civireport.id
ORDER BY entity_log_civireport.log_date DESC;
```
New EXPLAIN:
```
+------+-------------+-------------------------------+--------+-------------------+----------+---------+---------------------------------------------------+---------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+--------+-------------------+----------+---------+---------------------------------------------------+---------+---------------------------------+
| 1 | SIMPLE | modified_contact_civireport | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | entity_log_civireport | ALL | index_activity_id | NULL | NULL | NULL | 5518537 | Using where; Using filesort |
| 1 | SIMPLE | extra_table | ref | index_id | index_id | 5 | staging_civicrm.entity_log_civireport.activity_id | 1 | |
| 1 | SIMPLE | altered_by_contact_civireport | eq_ref | PRIMARY | PRIMARY | 4 | staging_civicrm.entity_log_civireport.log_user_id | 1 | Using where |
+------+-------------+-------------------------------+--------+-------------------+----------+---------+---------------------------------------------------+---------+---------------------------------+
```
This gets it down to 4 seconds!
Since the first index we started with is no longer getting used in the final variant, we can just not add it.
Summary:
To fix "Change Log" tab taking forever to load, the following fix is needed:
Add index:
```
ALTER TABLE log_civicrm_activity ADD INDEX index_id (id);
```
Make the code emit the query modified as follows:
```
INSERT IGNORE INTO civicrm_tmp_e_logsummary_ffa3ac146126d178ade367f2a5d17bf5
SELECT STRAIGHT_JOIN activity_id, IF (entity_log_civireport.log_action = 'Insert' AND extra_table.activity_type_id = 51 , GROUP_CONCAT(entity_log_civireport.contact_id), 1) , entity_log_civireport.log_action as log_civicrm_entity_log_action, 'log_civicrm_activity_contact' as log_civicrm_entity_log_type, entity_log_civireport.log_user_id as log_civicrm_entity_log_user_id, entity_log_civireport.log_date as log_civicrm_entity_log_date, modified_contact_civireport.display_name as log_civicrm_entity_altered_contact, modified_contact_civireport.id as log_civicrm_entity_altered_contact_id, entity_log_civireport.log_conn_id as log_civicrm_entity_log_conn_id, modified_contact_civireport.is_deleted as log_civicrm_entity_is_deleted, altered_by_contact_civireport.display_name as altered_by_contact_display_name
FROM civicrm_contact modified_contact_civireport
JOIN staging_civicrm.log_civicrm_activity_contact entity_log_civireport ON entity_log_civireport.contact_id = modified_contact_civireport.id
JOIN staging_civicrm.log_civicrm_activity extra_table ON extra_table.id = entity_log_civireport.activity_id
LEFT JOIN civicrm_contact altered_by_contact_civireport ON entity_log_civireport.log_user_id = altered_by_contact_civireport.id
WHERE modified_contact_civireport.id = 338520 AND
entity_log_civireport.log_action != 'Initialization'
GROUP BY entity_log_civireport.log_conn_id,
entity_log_civireport.log_user_id,
EXTRACT(DAY_MICROSECOND FROM entity_log_civireport.log_date),
entity_log_civireport.id
ORDER BY entity_log_civireport.log_date DESC;
```
Not only does this speed it up from 8 minutes down to 4 seconds, it also doesn't wreak havoc with row locking where every row scanned gets locked by the transaction engine, potentially resulting in a massive query pile-up in the database.https://lab.civicrm.org/dev/core/-/issues/2597Tabset for manage contribution pages or manage events requires Classname to e...2023-08-07T15:13:57ZlarsssandergreenTabset for manage contribution pages or manage events requires Classname to equal pathIf you add a tab to a tabset for Configure Event, it will work fine with the last word of the class name different from the last word of the path, except for trying to save, which will save, but redirect to Manage Events instead of stayi...If you add a tab to a tabset for Configure Event, it will work fine with the last word of the class name different from the last word of the path, except for trying to save, which will save, but redirect to Manage Events instead of staying on the same tab. This is because [code here](https://github.com/civicrm/civicrm-core/blob/5db0bc3c1f54eaca4307f103a73bda596ae914d6/CRM/Event/Form/ManageEvent.php#L377) expects lastWord from CRM_myExtension_Form_lastWord to be the same as the path, i.e. /civicrm/event/manage/lastword. Everything else works as expected if these are different, except Save.
Looks like this same issue exists for [Contribution Pages](https://github.com/civicrm/civicrm-core/blob/5db0bc3c1f54eaca4307f103a73bda596ae914d6/CRM/Contribute/Form/ContributionPage.php#L427
) and probably elsewhere.
For now, I've added a warning to the docs for [the tabset hook](https://lab.civicrm.org/documentation/docs/dev/-/merge_requests/908). It would be good to fix this so the code linked above uses the path for the url instead of the class, at some point.https://lab.civicrm.org/dev/core/-/issues/2611Change default_value column for custom fields from varchar to text2023-08-20T02:32:58ZlarsssandergreenChange default_value column for custom fields from varchar to textA custom field can be of many types, including note. For a custom field that is a note, you may want to use a default value that is longer than 255 characters, but currently the default value is stored as a varchar(255). It would make se...A custom field can be of many types, including note. For a custom field that is a note, you may want to use a default value that is longer than 255 characters, but currently the default value is stored as a varchar(255). It would make sense to change this column to text, so that longer values can be entered.
In the custom fields UI, this field would be changed to a textarea when note is selected.
For us, the use case would be storing the content of a short waiver, so that the exact value of the waiver at the time of signing can be stored on the participant for an event, but I can imagine there may be other use cases.
Will submit PR if supported.https://lab.civicrm.org/dev/core/-/issues/2669Target contact missing for automatically generated case activities2023-02-23T16:00:45ZAndreasandreas.howiller@civiservice.deTarget contact missing for automatically generated case activitiesOverview
----------------------------------------
Some actions on cases leave automatically generated activities in the case activity log, but have no target contacts. Examples: Change case title, change case tags. As a result, they cann...Overview
----------------------------------------
Some actions on cases leave automatically generated activities in the case activity log, but have no target contacts. Examples: Change case title, change case tags. As a result, they cannot be edited through the user interface. While this is a minor issue, some users may want to leave notes about the activity (for example, why they changed a case title) and therefore need to edit it.
Reproduction steps
----------------------------------------
1. Open any case in CiviCRM and change the case title.
2. Tap edit for the corresponding activity in the activity timeline.
3. Tap save.
Current behaviour
----------------------------------------
The dialog will freeze while the CiviCRM logo keeps turning. A fatal error "DB Error: constraint violation" will be filed in the logs.
If you add any contact as a target contact in the database or in the edit screen the error won't appear anymore.
What is confusing, too: When editing the activity the client contact already wrongly shows up as target contact in this dialog and if you'd like to change it there you need to search the contact (the link to choose the client contact below the search field won't do the change here).
Expected behaviour
----------------------------------------
The Client contact is filed as target contact by default and connected problems disappear. E.g. editable activities can be saved after edit through UI now.
Environment information
----------------------------------------
* __CiviCRM:__ _5.35.2_ and _5.38.0_
* __CMS:__ _WordPress 5.7.2_https://lab.civicrm.org/dev/core/-/issues/3552Set time limit for bounce types with hold_threshold > 12023-09-05T10:00:02ZMichael McAndrewSet time limit for bounce types with hold_threshold > 1Bounce types can be split into those with a hold_threshold = 1 where a single bounce should cause this contact to be put on hold (e.g. mailbox non existent), and those where more than one bounce is required (mailbox full).
We don't curr...Bounce types can be split into those with a hold_threshold = 1 where a single bounce should cause this contact to be put on hold (e.g. mailbox non existent), and those where more than one bounce is required (mailbox full).
We don't currently set a time limit on those where more than one bounce is required, which is an oversight. For example, the bounce threshold for Mailbox full is 3. This means that as soon as we have received more than 3 mailbox full messages for a contact, their email will be put on hold. This might make sense if it was 3 in the last year, but doesn't really make sense if it was three in the last 10 years.
Similar arguments can be made for other bounce types, like Away, which is set to 30. If I send a fair amount of email to people that are out of the office quite a lot, over time a significant chunk of these will send me > 30 out of the office messages.
Hence we should only consider messages received in a certain time period. I am going to suggest one year. I would be up for making it user configurable (number of days) if people feel strongly about it.
Let me know what you think.https://lab.civicrm.org/dev/core/-/issues/2761Proposal: Change Pay Later Billing Address to look up a profile rather than a...2022-10-04T21:07:14ZBarijohnProposal: Change Pay Later Billing Address to look up a profile rather than a coded block.Overview
----------------------------------------
Currently on the pay later option when you use the option of Billing Address the fields are hard coded. This would be better to use a profile then it could be edited by the end user witho...Overview
----------------------------------------
Currently on the pay later option when you use the option of Billing Address the fields are hard coded. This would be better to use a profile then it could be edited by the end user without having to mess with the code.
Example use-case
----------------------------------------
Click on Pay Later Option and enable Billing Fields. This shows multiple fields that might not be required depending on the part of the world. So in that case there could be a reserved profile just for billing fields. This could have a help field to show where this profile is saved, so it would easily amended.
Current behaviour
----------------------------------------
Currently this is set in Core/Payment.php and /civicrm/templates/CRM/Core/BillingBlock.tpl (based on communications with our devs).
Proposed behaviour
----------------------------------------
Make this block look up a billing address profile instead.
Comments
----------------------------------------
Currently this field asks for information that isn't required in the UK and most of our clients would like to be able to edit this directly.https://lab.civicrm.org/dev/core/-/issues/2863Agree / document how tokens should be rendered from outside core2023-09-24T22:52:19ZeileenAgree / document how tokens should be rendered from outside coreIdeally we would have one recommended api way to do this. This would be a way that is usable via api explorer and all the ways we use the api and core code would be converted to do things using that same api method
There is [a documenta...Ideally we would have one recommended api way to do this. This would be a way that is usable via api explorer and all the ways we use the api and core code would be converted to do things using that same api method
There is [a documentation pr here](https://lab.civicrm.org/documentation/docs/dev/-/merge_requests/962) - at the moment I don't think it meets this need.
**Permissions & non-workflow rendering**
There is a question as to whether the api would support non-workflow messaging (in which case the api would likely be more like `Message::render()`. The argument against is that the permissions are more complex. However some notes on that
1) the most common use case is probably when checkPermissions = FALSE in php code.
2) there is an appropriate permission already in core
```
render templates' => [
$prefix . ts('render templates'),
ts('Render open-ended template content. (Additional constraints may apply to autoloaded records and specific notations.)'),
],
```
3) currently the v3 MessageTemplate.send api allows access to the core function - it required a messageTemplateID to be set - but this can probably be faked to allow 'any text'https://lab.civicrm.org/dev/core/-/issues/2864Meta - token usage 5.43 standardisation effort2022-09-02T01:38:26ZeileenMeta - token usage 5.43 standardisation effortWe are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can ...We are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can be around the one release - documentation is [here](https://lab.civicrm.org/-/ide/project/documentation/docs/sysadmin/tree/case/-/docs/upgrade/version-specific.md/)
The goal is to consolidate such that
1. All token rendering is done through the token processor, token hooks are redundant and deprecated
2. All functions on CRM_Utils_Token are unused in core and deprecated except for those in the legacy BAO mailing classes.
3. We freeze, deprecate and phase out in legacy BAO classes in favour of flexmailer
4. All core classes that render tokens do so using consistent token naming & output conventions (rather than dates being formatted in various ways and fields sometimes outputting labels and other times values or name fields https://lab.civicrm.org/dev/core/-/issues/2650)
5. As much as possible we render via tokens rather than smarty in workflow message templates (that couples them with the entity rather than any form flow)
6. pdf & email & scheduled reminder tasks consistently offer the same entity-specific, contact & domain tokens
7. We can start to define obvious missing tokens such as `{domain.now}` (done), `{contribution.balance}` and formatted addresses
Task list / streams - this gives some idea of where things are progressing or blocked
- [x] standardise contact tokens to support/ advertise label style (e.g `{contact.prefix_id:label}`
- [x] make Contact tokens work with partial contact inputs - currently they only work with no contact array passed in, or an array with all possible values - currently blocked on a caching bug https://github.com/civicrm/civicrm-core/pull/21790
- [x] split contact tokens into own class & extend entity tokens like the other classes(blocked on stdise contact tokens)
- [x] switch contact tokens to use apiv4
- [x] work through standardising location token loading (this could go further - ie I had thought about adding loading for billing
- [x] date formatting standardisation [documentation](https://docs.civicrm.org/user/en/latest/common-workflows/tokens-and-mail-merge/#date) https://lab.civicrm.org/dev/core/-/issues/2746
- [x] locale sensitive standardised money formatting https://lab.civicrm.org/dev/core/-/issues/2638 & [pr](https://github.com/civicrm/civicrm-core/pull/21783)
- [x] switch badge tokens to use token processor
- [x] event tokens cleanup & loading fixes + remove unindexed query, move the `fee_amount` and `balance` tokens to the participant tokens class, fix event class to listen to either 'participantId' or 'eventId'. Participant class should run first & load eventId if required. Events should be cached in a static cache as it would be very rare to message thousands of events in one run.
- [x] Recurring contribution tokens work (currently availble in only the recurring_edit template with a pr to extend to cancelled recurring https://test.civicrm.org/job/CiviCRM-Core-PR/44632/
- [x] pdf letter is sane (https://lab.civicrm.org/dev/core/-/issues/2790) enough to extend to deliver tokens for other entities (participant https://lab.civicrm.org/dev/core/-/issues/780)
- [x] email letter extend to deliver tokens for other entities (membership & participant) (https://lab.civicrm.org/dev/core/-/issues/2862)
- [x] scheduled reminders supports participant tokens https://lab.civicrm.org/dev/core/-/issues/779
- [x] domain tokens are consistently available & render https://lab.civicrm.org/dev/core/-/issues/2838 - done but a gap on case tasks as they prioritise being able to filter tokens by case type
- [x] install flexmailer on new installs https://lab.civicrm.org/dev/core/-/issues/2836
- [ ] fix remaining places that process tokens other than via the payment processor
- [x] eliminate replaceGreeting - currently [blocked on apiv4 caching bug](https://github.com/civicrm/civicrm-core/pull/21790)
- [ ] fix CRM_Contact_BAO_Contact_Utils::updateGreeting to not call `getTokenDetails`
- [ ] confirm civicrm_api3_mailing_preview is replaced by flexmailer
- [x] remove getTokenDetails call from CRM_Contribute_Form_Task_PDFLetter - note this appears to be an expensive way to call the contact api & tokens are unused https://github.com/civicrm/civicrm-core/pull/21805
- [x] remove extra rendering in BAO_Pledge https://github.com/civicrm/civicrm-core/pull/21789
- [x] removed getTokenDetails from updatePledgeStatus https://github.com/civicrm/civicrm-core/pull/21847
- [x] remove getTokenDetails from transitionParticipants
- [ ] remove getTokenDetails from sendCancellation & cancelParticipant in selfsvc flow
- [ ] deliverGroup - replaced by flexmailer?
- [ ] MailingPage::preview - replaced by flexmailer?
- [ ] remove hook:;tokenValues from label task
- [ ] getAnonymousTokenDetails ???
- [ ] confirm BAO_Mailing::compose is only via flexmailer
- [x] communicate rc status via dev list
- [x] communicate changes via blog
- [ ] upgrade message https://lab.civicrm.org/dev/core/-/issues/2871
- [ ] docs update to these instructions / check it can still all be done https://docs.civicrm.org/dev/en/latest/step-by-step/create-custom-case-token/
**Currenly out of scope**
- [ ] Token.getlist api - until we have this we have an issue with audience-filtering and with switching tokens depending on the schema. Also case tasks still have to render tokens weirdly without this because they currently filter by case type with partial success https://lab.civicrm.org/dev/core/-/issues/2788
- [ ] Clarify how extensions should render tokens https://lab.civicrm.org/dev/core/-/issues/2863
- [ ] encourage flexmailer with status checks on existing installs https://lab.civicrm.org/dev/core/-/issues/2836
Analysis of current token rendering it core:
Method | Used in | Status
-- | -- | --
Use the token processor | scheduled reminders, civimail with flexmailer in use, activity pdf task | This is the method we are consolidating all paths to use.
Use the functions in CRM_Utils_Token | PDF tasks other than activity | Migrated (PRs on [membership](https://github.com/civicrm/civicrm-core/pull/21521) and [contribution](https://github.com/civicrm/civicrm-core/pull/21524). Also work on fixing up the [pdf task class](https://lab.civicrm.org/dev/core/-/issues/2790) supporting[add participant tokens](https://lab.civicrm.org/dev/core/-/issues/780) (once we’ve finished [the standardisation of those](https://github.com/civicrm/civicrm-core/pull/21587))
| Email tasks | Migrated to token processor. Added tokens for other entities (participant, membership https://lab.civicrm.org/dev/core/-/issues/1396) [also see](https://lab.civicrm.org/dev/core/-/issues/2862)
| Export tasks (when merging contacts in export) | Migrated to token processor
| Profile link rendering | Migrated to token processor
| Storing contact greetings (eg. email_greeting_display) | In progress. Currently resolving inconsistent token syntax and load issues as this needs to be able to use partial pre-loads
Use random ad hoc code | Event badges | [migrated](https://github.com/civicrm/civicrm-core/pull/21587) - this is the only place outside of scheduled reminders where event or participant tokens were used in core
| Send SMS task (BAO_Activity::sendSMS) | migrated
| Labels task CRM_Contact_Form_Task_Label | argh
| CRM_Event_Form_SelfSvcTransfer | Still pretty argh - but this is a workflow message template so the fix is a bit different
| Transition participants | Also a workflow - possibly redundant as seems to be mostly about contact tokens which are resolved in the send function anyway
| Pledge acknowledgements | Also appear to be redundant
| CiviMail with flexmailer not used (many functions & classes) | Push the switch to flexmailer.
| Various unsubscribe / resubscribe type actions | Probably out of scope for this round