Development issueshttps://lab.civicrm.org/groups/dev/-/issues2023-11-23T07:20:57Zhttps://lab.civicrm.org/dev/translation/-/issues/53Frequency units frequently fail to be translated properly2023-11-23T07:20:57ZStéphane Lussierstephane@symbiotic.coopFrequency units frequently fail to be translated properlyOn numerous occasions, the internal value of frequency units (eg: year, month) are sent for display to a template without being translated first.
One such issue may be found in [CRM/Contribute/Form/Contribution/Main.php](https://lab.ci...On numerous occasions, the internal value of frequency units (eg: year, month) are sent for display to a template without being translated first.
One such issue may be found in [CRM/Contribute/Form/Contribution/Main.php](https://lab.civicrm.org/dev/core/-/blob/master/CRM/Contribute/Form/Contribution/Main.php#L550). In this case, a simple fix would usually suffice:
```patch
--- a/civicrm/CRM/Contribute/Form/Contribution/Main.php
+++ b/civicrm/CRM/Contribute/Form/Contribution/Main.php
@@ -548,7 +548,7 @@ class CRM_Contribute_Form_Contribution_Main extends CRM_Contribute_Form_Contribu
if (!empty($form->_values['is_recur_interval']) || $className == 'CRM_Contribute_Form_Contribution') {
$unit .= "(s)";
}
- $form->assign('frequency_unit', $unit);
+ $form->assign('frequency_unit', ts($unit));
}
else {
$form->assign('one_frequency_unit', FALSE);
```
However, this patch in incomplete. There are multiple cases where a similar fix would need to be applied.
It's also not uncommon to find singular/plural tests in existing templates. The following example comes from [templates/CRM/Contribute/Form/Contribution/ThankYou.tpl](https://lab.civicrm.org/dev/core/-/blob/master/templates/CRM/Contribute/Form/Contribution/ThankYou.tpl#L123):
```smarty
{if $frequency_interval > 1}
<strong>{ts 1=$frequency_interval 2=$frequency_unit}This membership will be renewed automatically every %1 %2(s).{/ts}</strong>
{else}
<strong>{ts 1=$frequency_unit}This membership will be renewed automatically every %1.{/ts}</strong>
{/if}
```
This solution looks a bit strange to me... It assumes that adding an `(s)` to an internal keyword will make it plural. That may work in most situations, but it would prevent the already translated keyword to be altered upon the phrase translation.
Shouldn't the keyword be fully translated _and_ pluralized (if necessary) _before_ being sent to the translation string?
Could the `ts` function take care of translating and pluralizing substrings?
I don't mean to make a huge issue out of a small one, but I wonder: Is there a pattern that we can agree on for these sorts of situations?https://lab.civicrm.org/dev/joomla/-/issues/23Frontend form field labels are not styled well on Joomla 32020-01-14T14:55:04ZAndrew ThompsonFrontend form field labels are not styled well on Joomla 3As shown in this screenshot, the field labels have an ugly grey background and white text applied to the `.label` CSS class, at least on the Joomla 3 default 'Protostar' site (frontend) template.
![image](/uploads/4e7a8e4a2f0d000f75a6ea8...As shown in this screenshot, the field labels have an ugly grey background and white text applied to the `.label` CSS class, at least on the Joomla 3 default 'Protostar' site (frontend) template.
![image](/uploads/4e7a8e4a2f0d000f75a6ea84d8046f42/image.png)
Of course this could be overridden by a savvy user with custom CSS but it does degrade the out-of-box experience for a new Joomla 3 user.https://lab.civicrm.org/dev/core/-/issues/863Functionality of Remove Case Role activity type is messed up2022-10-23T16:00:50ZDaveDFunctionality of Remove Case Role activity type is messed upI think it's been like this for a few years. I can look into it later but recording here for reference. Examples of stuff below, and I can reproduce on dmaster.demo.
1. In the case roles section on manage case, click the trash can icon ...I think it's been like this for a few years. I can look into it later but recording here for reference. Examples of stuff below, and I can reproduce on dmaster.demo.
1. In the case roles section on manage case, click the trash can icon at the end of a row to remove a case role. It does remove the role, but there's supposed to be a Remove Case Role activity filed on the case.
2. In the Other Relationships section on manage case, add a new relationship. It gets correctly added but adds a Remove Case Role activity to the case. Besides being the wrong activity, no activity should be added here since that add button is just a shortcut to adding a regular relationship in the usual way, that exists outside of the life of the case.https://lab.civicrm.org/dev/core/-/issues/4208Generating invoices generates an error with version >= 0.12.6 of wkhtmltopdf.2023-04-04T08:44:14ZeileenGenerating invoices generates an error with version >= 0.12.6 of wkhtmltopdf.Full detail in https://github.com/civicrm/civicrm-core/pull/23055 - am creating this to track that issue as the patch is not mergeable as isFull detail in https://github.com/civicrm/civicrm-core/pull/23055 - am creating this to track that issue as the patch is not mergeable as ishttps://lab.civicrm.org/dev/core/-/issues/2929Geocoding failures kill contributions2023-10-23T16:08:42ZandrewcormickdockeryGeocoding failures kill contributionsOverview
----------------------------------------
Last night between about 6pm and midnight AEDT, it appears that Nominatim (the Open Street Map API provider) had an intermittent outage. Some requests were receiving a 502 error. We hav...Overview
----------------------------------------
Last night between about 6pm and midnight AEDT, it appears that Nominatim (the Open Street Map API provider) had an intermittent outage. Some requests were receiving a 502 error. We have configured this as our Geocoding provider in CiviCRM under /civicrm/admin/setting/mapping?reset=1
People who were using CiviCRM in ways which caused the Geomapping provider to be called were receiving fatal errors as a result. This included people who were making donations (i.e. entering their address on the form which in turn was handed to the Geomapping provider for geocoding).
Current behaviour
----------------------------------------
When a request to the Geomapping provider fails (for example with a 502 error), the generating transaction within CiviCRM returns a fatal error and does not complete. This includes financial transactions, and indeed any administrative function that involves adding or updating a contact's address.
Expected behaviour
----------------------------------------
A Geomapping provider failure should never cause financial transactions to fail. I can't think of any reason why adding or updating an address should fail either. Fatal errors should only occur when an issue occurs which genuinely prevent a transaction from succeeding. A payment can still be processed regardless of whether that person's address can be geocoded or not. I expect a 502 Bad Gateway error from Nominatim would be logged, but never cause an address to fail to be added/updated. Obviously that would occur without the latitude/longitude being recorded. I can't even see the utility in informing the user. It's a system admin problem.
Environment information
----------------------------------------
* __CiviCRM:__ _5.35.2_
* __PHP:__ _7.3__
* __CMS:__ _Drupal 7.82_
* __Database:__ _MariaDB 10.4.21_
* __Web Server:__ _Nginx 1.10.3_
Comments
----------------------------------------
_Anything else you would like the reviewer to note._https://lab.civicrm.org/dev/core/-/issues/4721Get rid of `civicrm_option_value.domain_id`2023-10-26T11:41:06ZcolemanwGet rid of `civicrm_option_value.domain_id`Background/Motivation
------
Most tables with a `domain_id` column have that field *required* for every record.
`civicrm_option_value` is an outlier; most option groups don't care about domains. In fact there are only two that do:
```p...Background/Motivation
------
Most tables with a `domain_id` column have that field *required* for every record.
`civicrm_option_value` is an outlier; most option groups don't care about domains. In fact there are only two that do:
```php
/**
* $_domainIDGroups array maintains the list of option groups for whom
* domainID is to be considered.
*
* FIXME: Hardcoded list = bad. It would be better to make this a column in the civicrm_option_group table
* @var array
*/
public static $_domainIDGroups = [
'from_email_address',
'grant_type',
];
```
The `civicrm_option_value.domain_id` column and that lovely accompanying hardcoded list were added [in this commit](https://github.com/civicrm/civicrm-svn/commit/406091dc969907753a56fd7fae73320b1b012e67) as part of [this issue](https://issues.civicrm.org/jira/browse/CRM-5546). The assumption at the time seemed to be that the list might grow in the future, but that's proven to not be the case. Instead, we have this extra column making the already overloaded `civicrm_option_value` table even more complex to deal with. It would have been better at the time to move those two option groups into their own tables. It may not be too late to do so.
Proposal
-----------
1. Create a new table `civicrm_grant_type` - manage that table from the `civigrant` extension.
2. Create a new table `civicrm_from_email_address`.
3. Migrate all options into the new tables. Update pseudoconstant metadata in the schema.
4. Add some legacy handling at a few key points like `civicrm_api()` & `CRM_Core_OptionGroup::values()` to fetch options from the new table, for backward compatibility.
5. Deprecate the `civicrm_option_value.domain_id` column and eventually drop it.https://lab.civicrm.org/dev/core/-/issues/3083getNonDeductibleAmountFromPriceSet should take into account the financial typ...2023-12-05T21:36:19ZherbdoolgetNonDeductibleAmountFromPriceSet should take into account the financial type of the price field optionOverview
----------------------------------------
`getNonDeductibleAmountFromPriceSet()` gets called in a couple places to calculate the non-deductible amount of contributions. While the calling method will take into account the financi...Overview
----------------------------------------
`getNonDeductibleAmountFromPriceSet()` gets called in a couple places to calculate the non-deductible amount of contributions. While the calling method will take into account the financial type of the contribution when assigning the non-deductible amount, `getNonDeductibleAmountFromPriceSet()` does *not* take into account the financial type of the price set option. I believe it should be.
Reproduction steps
----------------------------------------
1. Create price set and add two fields: one with financial type Donation (deductible) = `$10` and other Event Fee (non-deductible) = `$20`. Make the Event Fee have a random non-deductible amount of say `$2.50`.
2. Add a contribution in the backend to a contact and choose this new price set.
3. Look at the resulting contribution.
Current behaviour
----------------------------------------
The contribution has non-deductible = `$2.50` for a `$30` contribution.
Expected behaviour
----------------------------------------
I expect the non-deducible to be `$20`, the full price of the Event Fee. The contribution should be setting the non-deductible to be equal that of the merchandise price field, regardless of the actual non-deductible amount for that field. If a price field is non-deductible then it is 100% non-deductible based on the financial type.https://lab.civicrm.org/dev/core/-/issues/4304getUFLocale() is not setting the proper locale2023-05-24T06:37:15ZshaneonabikegetUFLocale() is not setting the proper locale## Overview
Presently, the function ```getUFLocale()``` obtains the interface language for Wordpress with integration with WPML. The present formula is using incorrect ```apply_filters``` to generate the wrong locale for front-endusers....## Overview
Presently, the function ```getUFLocale()``` obtains the interface language for Wordpress with integration with WPML. The present formula is using incorrect ```apply_filters``` to generate the wrong locale for front-endusers.
This was discovered while working validating the pull request [#289](https://github.com/civicrm/civicrm-wordpress/pull/289) for better WPML integration for front-end users.
## What should happen
The link should be generated in the language that the current user has set.
## What is the problem
```php
// Maybe override with the locale that WPML reports.
elseif (defined('ICL_LANGUAGE_CODE')) {
$languages = apply_filters('wpml_active_languages', NULL);
foreach ($languages as $language) {
if ($language['active']) {
$locale = $language['default_locale'];
break;
}
}
}
```
According to the docs, ```apply_filters('wpml_active_languages', NULL)``` only retrieves a list of languages, but this has no bearing on the user's current language. The value ```$language['active']``` refers to whether the language is active or not. In my case, I have seen this reported as _false_ in some cases where languages are active - go figure :shrug: .
So we need to use ```apply_filters('wpml_current_language')``` [to obtain](https://wpml.org/wpml-hook/wpml_current_language/) the users front-end language.
I'll post a patch for this and link it to this one.
cc @kcristianohttps://lab.civicrm.org/dev/core/-/issues/3883Group - remove contacts action removes contacts who are not in group and repo...2022-10-25T11:51:49ZlarsssandergreenGroup - remove contacts action removes contacts who are not in group and reports incorrect numbers removedOverview
----------------------------------------
[removeContactsFromGroup](https://github.com/civicrm/civicrm-core/blob/843958bc0022cec4145c8796ae49ccb2fd9d3c28/CRM/Contact/BAO/GroupContact.php#L156) doesn't check if a contact is in a g...Overview
----------------------------------------
[removeContactsFromGroup](https://github.com/civicrm/civicrm-core/blob/843958bc0022cec4145c8796ae49ccb2fd9d3c28/CRM/Contact/BAO/GroupContact.php#L156) doesn't check if a contact is in a group before removing or deleting them, resulting in some strange behaviour. The reported counts shown to the user of contacts removed are also incorrect. Subscription history is set when it doesn't need to be.
This function is also used by Api3 and for unsubscribes.
Current behaviour
----------------------------------------
If we're removing, no matter what the current status of the contact is in the group (Added, Pending, Removed or none at all), the status is set to Removed. If a contact had no relationship to the group at all, they will be set to Removed (and show on the Contact Groups tab as removed, which is confusing for the user as they were never in the group). In all cases, subscription history is added with status Removed.
Counts of contacts removed reported to user are incorrect.
If we're deleting, the row is simply deleted from the table. In all cases, subscription history is added with status Deleted.
Expected behaviour
----------------------------------------
If we're removing:
- If a contact is currently Added or Pending, it is set to Removed and subscription history is added.
- If a contact is Removed or has no group status, nothing happens.
This is somewhat confusing for smart groups. If a contact is not in a smart group because they don't meet the smart group criteria, do we want to set them to Removed, so they are effectively pre-removed if they later meet the smart group criteria? I think we do want this to happen for mailing groups because this function is used for unsubscribes. We don't want a contact to later become part of a smart group that was used for a mailing that they unsubscribed from, so I think we do want to set status to Removed no matter what.
Counts reported to user should also be correct based on what actually happened.
If we're deleting:
- If a contact is currently Added, Pending or Removed, it is deleted from the table and subscription history is added. For smart groups, this resets the contact to being a smart member of the group or not based on the smart group criteria.
- If a contact has no group status, nothing happens.
Fix
----------------------------------------
If this makes sense, I'll submit a PR.
Confusingly, removeContactsFromGroup is also used to _add_ contacts to a group [here.](https://github.com/civicrm/civicrm-core/blob/843958bc0022cec4145c8796ae49ccb2fd9d3c28/CRM/Contact/Page/View/GroupContact.php#L173) I will change that to use addContactsToGroup for Added or Pending. This will result in a change to the op for the hooks though (from `CRM_Utils_Hook::pre('edit', 'GroupContact'...` to `CRM_Utils_Hook::pre('create', 'GroupContact'...`. [The docs](https://docs.civicrm.org/dev/en/latest/hooks/hook_civicrm_pre/) say create is correct for a contact being added to a group, so it appears this is not correct as it stands. Definitely would value an expert opinion on this aspect.https://lab.civicrm.org/dev/core/-/issues/4370Group opt in and subscribe/unsubscribe in form builder2023-09-01T19:44:08ZMichael McAndrewGroup opt in and subscribe/unsubscribe in form builder# Group subscribe and unsubscribe in form builder
This issue is an attempt to add functionality to handle group subscribe and unsubscribe in form builder as per the roadmap https://lab.civicrm.org/dev/core/-/issues/3761.
## Scenarios
...# Group subscribe and unsubscribe in form builder
This issue is an attempt to add functionality to handle group subscribe and unsubscribe in form builder as per the roadmap https://lab.civicrm.org/dev/core/-/issues/3761.
## Scenarios
Here are a few scenarios that we'd like to cater for...
### Opt in only flow
This flow is useful to give people the option to opt in to a group as part of another flow, e.g. when I register for an event, I might want to opt into an events newsletter. Note: I don't typically want to opt *out* of such a newsletter in these situations, and an opt out is likey unexpected and undesirable.
Opt in flows are also useful to create newsletter sign ups that contain multiple groups/mailing lists that people can sign up for.
We render a checkbox and group name. When the checkbox is ticked, we add a contact to the group (this might be a no-op if they are already in the group). If the checkbox is not ticked, we do nothing (i.e. we do not unsubscribe anyone that is already subscribed to the group).
We render a separate component for each group that the contact wants to opt into (rather than a single component that contains all groups).
By default checkboxes are not ticked when the form is loaded. This aligns nicely for workflows (e.g. GDPR compliant workflows) where opting in should be a conscious decision. However, it should be possible to set the checkbox to be ticked by default if desired. This may be appropriate in certain situations, e.g. it would be reasonable to pre-check a 'Please send me members newsletter' checkbox in a membership form if you wanted most people to receive the members newsletter but also wanted to give the option to opt out.
## Subscribe and unsubscribe flow
This allows people to both add themselves and remove themselves from groups.
This is typically used in 'Manage my notifications/communication preferences' screens and similar. It allows people to see what groups they are currently subscribed to, and subscribe and unsubscribe from them as appropriate.
We render a separate UI element for each group that the contact wants to opt into. When the checkbox is ticked, we add a contact to the group (if they are not already in it). If the checkbox is not ticked, we remove them from the group (if they are already in it).
## Automatic subscribe / unsubscribe flow
Sometimes we might want to automatically sign someone up to a mailing list without presenting a checkbox to tick. For example...
![image](/uploads/e5774b8f0d319fa33cee79f471d7049b/image.png)
One solution in this situation would be to add a hidden and checked subscribe or opt in element. In the situation that submitting the form should unsubscribe you, adding an unchecked subscribe/unsubscribe component should do the trick.
## The data model and form processing
I'm not 100% sure what the mark up for this should look like. An email join looks like this:
```html
<fieldset af-fieldset="Individual1" class="af-container" af-title="Individual 1">
<afblock-name-individual></afblock-name-individual>
<div af-join="Email">
<af-field name="email" />
</div>
</fieldset>
```
But group subscriptions and un-subscriptions don't work quite the same way.
In terms of the API, I think that what we want to do can probably be achieved with something like the save api:
```php
$results = \Civi\Api4\GroupContact::save()
->addRecord([
'group_id' => 2,
'contact_id' => 203,
'status' => 'Added',
])
->setMatch([
'group_id',
'contact_id',
])
->execute();
```
But I think there are a few tweaks that we'd want to do, e.g. we ideally do not want to record a status=removed when submitting a form with an unsubscribe/subscribe if they were not in the group in the first place.
And I am not sure how the save api gets called in formbuilder.
In terms of what the html might look like, I am wondering if it makes sense to create some components that encapsulate some of the logic, so that it ends up looking like this.
```html
<fieldset af-fieldset="Individual1" class="af-container" af-title="Individual 1">
<afblock-name-individual></afblock-name-individual>
<af-group group_id="2" />
</fieldset>
```
It might make sense to add a mode property that could be set to control the display and behaviour, e.g. `<af-group group_id="2" mode='opt-in'/>`
- normal - opts in and out
- opt-in - only allows opting in
- auto-add - hidden and adds people on submission
- auto-remove - hidden and removes people on submission
And a default property for opt in workflows, e.g. <af-group group_id="2" mode='opt-in' default=true/>
The above component would be responsible for generating the checkbox and text and for generating appropriate values when the form is submitted.
I can't see a way of doing it with the existing components. I might be missing something conceptual but here is my attempt. It made me think that a new component would probably make things easier.
```html
<fieldset af-fieldset="Individual1" class="af-container" af-title="Individual 1">
<afblock-name-individual></afblock-name-individual>
<div af-join="GroupContact">
<af-field name="group_id" /> <?-- do we also need to join on group to get the name? -->
<af-field name="status" /> <?-- Would need to translate a checkbox to value?'Added':'Removed'? -->
</div>
</fieldset>
```
## Email confirmation
At the moment, double opt in is closely linked to subscribing to an email group. IIRC in the past we have used something like
```php
<?php
civicrm_api3('MailingEventSubscribe', 'create', ...);
```
But this has a few drawbacks, including having to confirm multiple email subscriptions for a single form. Since we are also working on https://lab.civicrm.org/dev/core/-/issues/4232, we think it probably makes sense to leverage that for any logged out forms which will require that an email needs to be confirmed before any form processing occurs.https://lab.civicrm.org/dev/drupal/-/issues/86Group Role sync not mapping users but duplicating users2019-09-13T07:05:05ZacaselliGroup Role sync not mapping users but duplicating usersHi,
The group role sync module doesn't seem to work properly anymore. When I create a new user and I log in for the first time, instead of mapping the Drupal user id with the civicrm user id, the module maps the drupal user id to a new ...Hi,
The group role sync module doesn't seem to work properly anymore. When I create a new user and I log in for the first time, instead of mapping the Drupal user id with the civicrm user id, the module maps the drupal user id to a new user that has everything blank apart from the email.
I checked in the database, and in the table uf_match to confirm and here is (I think) what the module does.
When a user logs in, the checking of the existing user doesn't work anymore, so a new user is created with only the email field and that new user civicrm id is mapped with the drupal id in the uf_match table.
This is causing a lot of problems in our website. I updated to the latest version, 5.17.3 but that didn't fix it either. I noticed this error since version 5.11https://lab.civicrm.org/dev/core/-/issues/1321group.get API (v3- but lets fix for v4) fails to find groups of one group_typ...2023-08-04T20:49:54ZRichgroup.get API (v3- but lets fix for v4) fails to find groups of one group_type if group has multiple typesWe used (5.15) to be able to filter for mailing groups with the API and now we can't. I'm not sure when this came in.
e.g. on a buildkit at 58e7f004e2
```
cv api Group.get return='id,title,group_type'
```
Returns
```
{
"is_error":...We used (5.15) to be able to filter for mailing groups with the API and now we can't. I'm not sure when this came in.
e.g. on a buildkit at 58e7f004e2
```
cv api Group.get return='id,title,group_type'
```
Returns
```
{
"is_error": 0,
"version": 3,
"count": 4,
"values": {
"1": {
"id": "1",
"title": "Administrators",
"group_type": [
"1"
]
},
"2": {
"id": "2",
"title": "Newsletter Subscribers",
"group_type": [
"1",
"2"
]
},
"3": {
"id": "3",
"title": "Summer Program Volunteers",
"group_type": [
"1",
"2"
]
},
"4": {
"id": "4",
"title": "Advisory Board",
"group_type": [
"1",
"2"
]
}
}
}
```
But
```
cv api Group.get return='id,title,group_type' group_type='Mailing List'
cv api Group.get return='id,title,group_type' group_type=2
```
returns none.
<del>On 5.15 Mailing groups are returned properly.</del>https://lab.civicrm.org/dev/core/-/issues/4701Hidden fields do their job too well2023-10-20T07:52:02ZJonGoldHidden fields do their job too well"Hidden" fields are meant to be hidden on profiles, and hidden (but showable) by default on FormBuilder.
It turns out they're hidden on all QuickForm pages. :eye: :mag_right:
Attached are two screenshots of a contribution with a hidden..."Hidden" fields are meant to be hidden on profiles, and hidden (but showable) by default on FormBuilder.
It turns out they're hidden on all QuickForm pages. :eye: :mag_right:
Attached are two screenshots of a contribution with a hidden custom field "Outreach Method", which has a value of "email". Please ignore the other custom fields - I didn't make this db :rolling_eyes:
Now in the longer term we can assume these pages will be replaced by FB and it will be a moot point, but for now, it seems like the correct behavior for hidden fields should be "visible on the back end". There are certainly instances where that wouldn't be desirable, but for now I think we need to accommodate the most common use case. Which this will be until we remove the unfortunate overlap with "reserved" custom fields.
![Selection_2050](/uploads/460114d6d1e110685b27f06a595a90cf/Selection_2050.png)
![Selection_2051](/uploads/31e8e7ba5e89e136030a7b476d306863/Selection_2051.png)https://lab.civicrm.org/dev/core/-/issues/4196hook_civicrm_links - are the docs wrong or the code?2023-03-23T20:11:09ZJonGoldhook_civicrm_links - are the docs wrong or the code?The docs for `hook_civicrm_links` provides this signature:
```php
hook_civicrm_links(string $op, string $objectName, int $objectID, array &$links, int &$mask, array &$values)
```
However, the type hinting isn't always correct. I've fo...The docs for `hook_civicrm_links` provides this signature:
```php
hook_civicrm_links(string $op, string $objectName, int $objectID, array &$links, int &$mask, array &$values)
```
However, the type hinting isn't always correct. I've found:
* PCP pages pass `NULL` for `$mask`.
* The extensions page passes a string for `$objectID`.
Should we change the docs or the code?https://lab.civicrm.org/dev/core/-/issues/2607How to handle removed files2023-12-15T15:29:01Zaydunsaidan.saunders@squiffle.ukHow to handle removed filesSee [this SE question](https://civicrm.stackexchange.com/questions/39531/database-upgrade-script-failing-call-to-undefined-method-crm-upgrade-incrementa/39546#39546)
A file was removed from the installation tarball but [the Joomla upgr...See [this SE question](https://civicrm.stackexchange.com/questions/39531/database-upgrade-script-failing-call-to-undefined-method-crm-upgrade-incrementa/39546#39546)
A file was removed from the installation tarball but [the Joomla upgrade process](https://docs.civicrm.org/sysadmin/en/latest/upgrade/joomla/#install-the-extension) installs into existing directories meaning the 'removed' file was still present. In this case, it caused the upgrade to fail and will cause problems for any Joomla upgrades following the documented method.
More generally, the same problem applies to any files no longer included in releases.
Some possibilities:
- in the release notes say that files have been removed so don't follow the normal upgrade process
- add an upgrade step that removes old files (but note this problem occurred early in the upgrade process)
- change the upgrade instructions for Joomlahttps://lab.civicrm.org/dev/core/-/issues/2565HTML editor for additional event confirmation email text2023-08-02T21:16:19ZlarsssandergreenHTML editor for additional event confirmation email textIt would be nice to be able to have an HTML editor for the additional event confirmation text entered on the Manage Event Page that gets inserted into the System Template email template. The benefit would be to be able to easily add link...It would be nice to be able to have an HTML editor for the additional event confirmation text entered on the Manage Event Page that gets inserted into the System Template email template. The benefit would be to be able to easily add links, etc. I understand this will be a complicated one, but it does seem worthwhile as almost every other similar field is HTML, so this one really stands out. [See previous issue.](https://lab.civicrm.org/dev/core/-/issues/852#note_58464)
How it works now: Text from the event confirmation email text is added into the event confirmation message template with some minimal tags added. I see a <p> wrapper around the text and <br /> inserted for newlines, at a cursory glance.
Here's my guess at what needs to happen:
1. Add HTML editor to Event Configuration
2. Add HTML editor on the New Event Registration page (which is also Add Participant from events or Add Event Registration for a contact), so that the message can be edited before sending.
3. Inject content as HTML into Message Templates, i.e. remove code that adds tags, wrap code in div or something.
4. Convert all existing content in the DB with the same process now taken to inject the text into the Message Template or handle the content injection into the Message Template with two cases based on text/HTML.
The first three should be relatively simple, but the last one there seems like the tricky part. How has this has been handled in the past?https://lab.civicrm.org/dev/core/-/issues/4047htmlspecialchars() issue on PHP8 and CiviReport2023-01-04T07:44:17ZJonGoldhtmlspecialchars() issue on PHP8 and CiviReport**Note** I've been seeing this `htmlspecialchars()` error on this line frequently on PHP 8. This is just the one instance I investigated thoroughly.
When attempting to use a particular CiviReport instance, we get a WSOD with this error...**Note** I've been seeing this `htmlspecialchars()` error on this line frequently on PHP 8. This is just the one instance I investigated thoroughly.
When attempting to use a particular CiviReport instance, we get a WSOD with this error in watchdog:
```
TypeError: htmlspecialchars(): Argument #1 ($string) must be of type string, array given in htmlspecialchars() (line 144 of /var/www/mysite/vendor/civicrm/civicrm-packages/HTML/Common.php)
#0 /home/jon/local/mysite/vendor/civicrm/civicrm-packages/HTML/Common.php(144): htmlspecialchars()
#1 /home/jon/local/mysite/vendor/civicrm/civicrm-packages/HTML/QuickForm/input.php(154): HTML_Common->_getAttrString()
#2 /home/jon/local/mysite/vendor/civicrm/civicrm-packages/HTML/QuickForm/Renderer/Array.php(307): HTML_QuickForm_input->toHtml()
#3 /home/jon/local/mysite/vendor/civicrm/civicrm-packages/HTML/QuickForm/Renderer/ArraySmarty.php(189): HTML_QuickForm_Renderer_Array->_elementToArray()
#4 /home/jon/local/mysite/vendor/civicrm/civicrm-core/CRM/Core/Form/Renderer.php(87): HTML_QuickForm_Renderer_ArraySmarty->_elementToArray()
#5 /home/jon/local/mysite/vendor/civicrm/civicrm-packages/HTML/QuickForm/Renderer/Array.php(221): CRM_Core_Form_Renderer->_elementToArray()
#6 /home/jon/local/mysite/vendor/civicrm/civicrm-packages/HTML/QuickForm/element.php(415): HTML_QuickForm_Renderer_Array->renderElement()
#7 /home/jon/local/mysite/vendor/civicrm/civicrm-packages/HTML/QuickForm.php(1705): HTML_QuickForm_element->accept()
#8 /home/jon/local/mysite/vendor/civicrm/civicrm-core/CRM/Core/Form.php(1136): HTML_QuickForm->accept()
#9 /home/jon/local/mysite/vendor/civicrm/civicrm-core/CRM/Core/QuickForm/Action/Display.php(95): CRM_Core_Form->toSmarty()
#10 /home/jon/local/mysite/vendor/civicrm/civicrm-core/CRM/Core/QuickForm/Action/Display.php(83): CRM_Core_QuickForm_Action_Display->renderForm()
#11 /home/jon/local/mysite/vendor/civicrm/civicrm-packages/HTML/QuickForm/Controller.php(203): CRM_Core_QuickForm_Action_Display->perform()
#12 /home/jon/local/mysite/vendor/civicrm/civicrm-packages/HTML/QuickForm/Page.php(103): HTML_QuickForm_Controller->handle()
#13 /home/jon/local/mysite/vendor/civicrm/civicrm-core/CRM/Core/Controller.php(355): HTML_QuickForm_Page->handle()
#14 /home/jon/local/mysite/vendor/civicrm/civicrm-core/CRM/Utils/Wrapper.php(98): CRM_Core_Controller->run()
#15 /home/jon/local/mysite/vendor/civicrm/civicrm-core/CRM/Report/Page/Instance.php(74): CRM_Utils_Wrapper->run()
#16 /home/jon/local/mysite/vendor/civicrm/civicrm-core/CRM/Core/Invoke.php(319): CRM_Report_Page_Instance->run()
#17 /home/jon/local/mysite/vendor/civicrm/civicrm-core/CRM/Core/Invoke.php(69): CRM_Core_Invoke::runItem()
#18 /home/jon/local/mysite/vendor/civicrm/civicrm-core/CRM/Core/Invoke.php(36): CRM_Core_Invoke::_invoke()
#19 /home/jon/local/mysite/web/modules/contrib/civicrm/src/Civicrm.php(88): CRM_Core_Invoke::invoke()
#20 /home/jon/local/mysite/web/modules/contrib/civicrm/src/Controller/CivicrmController.php(80): Drupal\civicrm\Civicrm->invoke()
#21 [internal function]: Drupal\civicrm\Controller\CivicrmController->main()
#22 /home/jon/local/mysite/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php(123): call_user_func_array()
#23 /home/jon/local/mysite/web/core/lib/Drupal/Core/Render/Renderer.php(580): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()
#24 /home/jon/local/mysite/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php(124): Drupal\Core\Render\Renderer->executeInRenderContext()
#25 /home/jon/local/mysite/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php(97): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext()
#26 /home/jon/local/mysite/vendor/symfony/http-kernel/HttpKernel.php(169): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()
#27 /home/jon/local/mysite/vendor/symfony/http-kernel/HttpKernel.php(81): Symfony\Component\HttpKernel\HttpKernel->handleRaw()
#28 /home/jon/local/mysite/web/core/lib/Drupal/Core/StackMiddleware/Session.php(58): Symfony\Component\HttpKernel\HttpKernel->handle()
#29 /home/jon/local/mysite/web/core/lib/Drupal/Core/StackMiddleware/KernelPreHandle.php(48): Drupal\Core\StackMiddleware\Session->handle()
#30 /home/jon/local/mysite/web/core/modules/page_cache/src/StackMiddleware/PageCache.php(106): Drupal\Core\StackMiddleware\KernelPreHandle->handle()
#31 /home/jon/local/mysite/web/core/modules/page_cache/src/StackMiddleware/PageCache.php(85): Drupal\page_cache\StackMiddleware\PageCache->pass()
#32 /home/jon/local/mysite/web/core/lib/Drupal/Core/StackMiddleware/ReverseProxyMiddleware.php(48): Drupal\page_cache\StackMiddleware\PageCache->handle()
#33 /home/jon/local/mysite/web/core/lib/Drupal/Core/StackMiddleware/NegotiationMiddleware.php(51): Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle()
#34 /home/jon/local/mysite/vendor/stack/builder/src/Stack/StackedHttpKernel.php(23): Drupal\Core\StackMiddleware\NegotiationMiddleware->handle()
#35 /home/jon/local/mysite/web/core/lib/Drupal/Core/DrupalKernel.php(707): Stack\StackedHttpKernel->handle()
#36 /home/jon/local/mysite/web/index.php(19): Drupal\Core\DrupalKernel->handle()
#37 {main}
```
I tracked it down to my `form_values` which is this:
```
a:27:{s:6:"fields";a:4:{s:9:"sort_name";s:1:"1";s:8:"event_id";s:1:"1";s:9:"status_id";s:1:"1";s:7:"role_id";s:1:"1";}s:12:"sort_name_op";s:3:"has";s:15:"sort_name_value";s:0:"";s:8:"email_op";s:3:"has";s:11:"email_value";s:0:"";s:11:"event_id_op";s:2:"in";s:14:"event_id_value";a:0:{}s:6:"sid_op";s:2:"in";s:9:"sid_value";a:0:{}s:6:"rid_op";s:2:"in";s:9:"rid_value";a:0:{}s:34:"participant_register_date_relative";s:1:"0";s:30:"participant_register_date_from";s:0:"";s:28:"participant_register_date_to";s:0:"";s:6:"eid_op";s:2:"in";s:9:"eid_value";a:0:{}s:11:"custom_4_op";s:2:"in";s:14:"custom_4_value";a:0:{}s:16:"blank_column_end";s:0:"";s:11:"description";s:44:"Provides lists of participants for an event.";s:13:"email_subject";s:0:"";s:8:"email_to";s:0:"";s:8:"email_cc";s:0:"";s:10:"permission";s:16:"access CiviEvent";s:6:"groups";s:0:"";s:7:"options";N;s:9:"domain_id";i:1;}
```
Unserialized:
```
array (
'fields' =>
array (
'sort_name' => '1',
'event_id' => '1',
'status_id' => '1',
'role_id' => '1',
),
'sort_name_op' => 'has',
'sort_name_value' => '',
'email_op' => 'has',
'email_value' => '',
'event_id_op' => 'in',
'event_id_value' => array (),
'sid_op' => 'in',
'sid_value' => array (),
'rid_op' => 'in',
'rid_value' => array (),
'participant_register_date_relative' => '0',
'participant_register_date_from' => '',
'participant_register_date_to' => '',
'eid_op' => 'in',
'eid_value' => array (),
'custom_4_op' => 'in',
'custom_4_value' => array (),
'blank_column_end' => '',
'description' => 'Provides lists of participants for an event.',
'email_subject' => '',
'email_to' => '',
'email_cc' => '',
'permission' => 'access CiviEvent',
'groups' => '',
'options' => NULL,
'domain_id' => 1,
)
```
The issue is `custom_4`, which is save as `in` `array()`. However, this is (and always has been) a free-entry text field, so "IN" should never have been an option. Pre-PHP8, this was ignored, now it's a fatal error.
I see @seamuslee made a [partial fix](https://github.com/civicrm/civicrm-packages/pull/346) but I'll submit a patch to handle the "empty array" case.JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/4564Idea: Smart groups without the cache2023-09-13T12:20:47ZAndrew WestIdea: Smart groups without the cacheJust throwing this out there as something we could maybe fund:
Current behaviour
----------------------------------------
Smart groups always write to the civicrm_group_contact_cache table
Proposed behaviour
---------------------------...Just throwing this out there as something we could maybe fund:
Current behaviour
----------------------------------------
Smart groups always write to the civicrm_group_contact_cache table
Proposed behaviour
----------------------------------------
Smart groups optionally write to the civicrm_group_contact_cache table (default: yes)
Overview
----------------------------------------
As I understand it, whenever you request a smart group it stores its contacts in civicrm_group_contact_cache. This is for good reason: it's a cache that means the smart groups aren't regenerated every time they're requested, which can be slow.
It also simplifies the code: you can effectively treat smart groups the same as non-smart groups - as it's just grabbing a list of contacts from a table.
But it's possible to set the cache to expire after 0 seconds, if you need your smart groups to be up to date: ie. they are regenerated each time they're requested. I think a fair few orgs do this.
If you have a 0s cache, you sometimes don't need the data written to a table at all. You just need a transient list of contacts that doesn't need to persist past the end of the request. And it's pretty slow to use the cache table: the 'INSERT INTO' command is slow relative to the original SELECT query. This time can add up if you're querying a bunch of groups at once, or have nested smart groups, or have a lot of contacts in the groups.
So I was tentatively wondering if there could be an API4 'no-smart-group-cache' parameter, that just runs the underlying query and stores it all in memory. I imagine this would only work for API4 smart groups, as it's super-easy to get their underlying search criteria.
The default would remain to use the cache, for full backwards-compatibility. But the parameter could be used in custom code.
Two questions:
1. Anyone else interested in this?
2. How hard would it be / how much of a can of worms is it?https://lab.civicrm.org/dev/core/-/issues/4045Ideas for (funded) speed improvements in mailings2023-08-28T14:44:15ZAndrew WestIdeas for (funded) speed improvements in mailingsWe'd be interested in funding some speed improvements when building mailing recipients in CiviMail. They'd inherently feed through to Mosaico, I think.
We find the mailings screens pretty snappy apart from selecting recipients. There a...We'd be interested in funding some speed improvements when building mailing recipients in CiviMail. They'd inherently feed through to Mosaico, I think.
We find the mailings screens pretty snappy apart from selecting recipients. There are few bottlenecks that it'd be nice to get rid of:
**Improve the dropdown**
When you have lots of groups and mailings the dropdown doesn't work very well. It stalls, with multiple spinners appearing, or wipes out your text while typing, or says 'none found' for a bit before updating, or alternates between 'Searching' and 'Loading', and generally takes an achingly long time to find the groups in its own dropdown.
![image](/uploads/91dcd7a0a2c2e7d3230c4016d1d49db0/image.png)
Compare this to the smart group dropdown in Search Kit, which is blindingly fast.
Maybe this could be replaced with the new API4-based entityref? Possibly split into two (or even four?): Groups to include / Groups to Exclude / Mailings to include / Mailings to exclude?
**Improve generating the list of recipients**
A complicated selection of groups can easily take 30 seconds to update and save, for us. This is very frustrating when adding one smart group at a time in the dropdown. Possible fixes:
- Refactor CRM_Mailing_BAO_Mailing::getRecipients(). At the moment this function seems to create a lot of temp tables, and uses a bunch of heavy queries to compare them against each other. Again, Search Kit can do complicated includes/excludes way faster. So I figure this could be modernised.
- Possibly the slow part here is emptying/filling civicrm_mailing_recipients each time the criteria are changed. Maybe add a 'getRecipientCountPreview' function to get a count of recipients without actually populating the table? This would let people experiment in the mailing screen quickly. Then only generate the actual recipient list when the mailing is scheduled or saved as a draft?
Any other ideas? Or thoughts on whether the above makes sense?
We could fund maybe 15hrs on this atm. If anyone wants to join us that'd be great too :-)https://lab.civicrm.org/dev/core/-/issues/1625IDN-Domain Emails - dont pass email check.2022-09-29T14:42:46ZRar9IDN-Domain Emails - dont pass email check.See Issue https://issues.civicrm.org/jira/browse/CRM-15975 and
Email validation is too restrictive (rejects IDN addresses) https://issues.civicrm.org/jira/browse/CRM-16313
Emails with these Characters should pass
https://www.denic.de/...See Issue https://issues.civicrm.org/jira/browse/CRM-15975 and
Email validation is too restrictive (rejects IDN addresses) https://issues.civicrm.org/jira/browse/CRM-16313
Emails with these Characters should pass
https://www.denic.de/en/know-how/idn-domains/idn-character-list/
Last tested with Civicrm 5.22.1 + Drupal 7
Php 7.2x Intl enabled.