Development issueshttps://lab.civicrm.org/groups/dev/-/issues2024-03-15T02:23:37Zhttps://lab.civicrm.org/dev/core/-/issues/3752Membership terms present in price set not get used in the offline form signup.2024-03-15T02:23:37ZsunilMembership terms present in price set not get used in the offline form signup.Overview
----------------------------------------
Number of Terms present in priceset not get used while signing up using offline form.
Reproduction steps
----------------------------------------
1. Crete Price and field of Select opti...Overview
----------------------------------------
Number of Terms present in priceset not get used while signing up using offline form.
Reproduction steps
----------------------------------------
1. Crete Price and field of Select option.
1. Use Membership type `Student` which is 1 year duration.
1. Use the same membership type with different terms.
1. Now, go to the contact and sign up for membership using priceset.
1. Use Membership with 2 or 3 year terms.
1. Result : it extends by only 1 year.
Current behaviour
----------------------------------------
Irrespective of different terms of membership, membership only extends by only one year.
Expected behaviour
----------------------------------------
Membership should extend as per terms set in the price set field option.https://lab.civicrm.org/dev/financial/-/issues/200Contribution status’s are ambiguous (WIP)2022-09-10T05:42:18ZJamie Novick - CompucoContribution status’s are ambiguous (WIP)**Background:**
Contribution status’s are ambiguous in CiviCRM and can lead to confusion over the current state of an invoice or contribution and whether there are amounts owed or owing to the contributor. This leads to a wealth of prob...**Background:**
Contribution status’s are ambiguous in CiviCRM and can lead to confusion over the current state of an invoice or contribution and whether there are amounts owed or owing to the contributor. This leads to a wealth of problems about what the intention of the user really is in a number of scenarios when they change the contribution status.
Examples:
| Example | Narrative | Video |
| ------ | ------ | ------ |
| Example 1a: Overpayment / pending refund | A contribution which is created for $100 donation, but pending. We then record an overpayment of $120. Hypothetically the contribution is overpaid by $20 and should be pending refund. This is not shown in CiviCRM. There is nothing showing the user that there is an overpayment of $20.| ![1-Overpayment](/uploads/74eca492a8d044aa17033dd681922c4a/1-Overpayment.webm)
| Example 1b: Change contribution status to refunded | We can further demonstrate this by taking the same contribution and then changing the status of the contribution to refunded. This raises the question of what this means? Does this reflect the users intention to: Cancel the original debt of $100 (i.e. to credit note the original amounts owed and clear the balance Or reflect the fact a refund was made for the payment of $120, but that they are still owed the original $100 Or both? CiviCRM has refunded the original value of the invoice, i.e. $100, even though the payment of $120 was received. As such we still owe $20 but this is not shown in CiviCRM.|![2-Change_status_to_refunded](/uploads/bf9eae010687771d440b4baa0d528d1e/2-Change_status_to_refunded.webm)
| Example 2: A typo entered in the wrong contribution amount | Add a contribution for $100 and set as completed. Realised that actually the contribution was $120. System will automatically add the additional $20 payment. Note: If this was a change in the line items of an invoice, it would not be safe to assume we received the additional payment. The aim of the user might be to state the obligation to pay was $120. Also it probably wasn’t 2 separate payments and hence we should reverse out the first payment (-100) and then make a second payment for the full amount (+120) | ![Updating_a_contribution_amount_error](/uploads/bda0cc37c9c5911b54bb429367bdffbf/Updating_a_contribution_amount_error.webm)
Much of this stems from the contribution status field being both be the driver of changes to the contribution (i.e. by changing the status to completed or refunded) or changed itself (i.e. the result of changes to the position of the contribution) by recording payments or refunds.
As such this is a proposal to simplify this workflow so that the the contribution status is only the result of the status of the contribution, and can no longer be used to change the status of the contribution.
We would retain the ability to make changes to the contribution status via the API for backwards compatibility until extensions can adopt the new model.
**Further ramblings (feel free to skip this part if you are not too interested in accounting principles)**
Accounting has simple and sounds principles for dealing with all of this.
“Obligations” are managed by 2 documents: Invoices and Credit notes. An invoice would stipulate what is obligated. This can be reduced by creating a credit note, which would then reduce the obligated amount.
It’s worth noting that for sales tax (for example VAT) purposes invoices should be sequentially numbered and should not be edited once they are generated. Hence usually you would not just simply delete an invoice to reduce the obligated amount, you would create a credit note for some or all of the amount, apply it to the invoice and send that to the customer to show the reduce in their balance owed to you.
As such I think in CiviCRM we need to fully enforce the split of concepts of the “obligation” and the actual payments (or refunded) amounts. Currently we have started on the journey towards that where Contributions (and line items on them) represent the obligations in most cases, and payments (or financial transactions of type payment) are the payments.
The contribution status should be a reflection of the net of that position. Does the customer owe us money (= pending or partially paid*) or is the balance settled (= completed) or do we owe them money (= pending refund)?
</details>
**Proposed changes:**
1. Recalculate the contribution status whenever a contribution is created or updated
We should recalculate the contribution status whenever a contribution is created or updated based on the following rules:
| Ref | Calculation | Status |
| ------ | ------ | ------ |
| 1 | If there are no payments or refunds | Pending (arguably there would be a better label for this but in order to avoid making actual changes to the contribution status options this would seem to be the best fit) |
| 2 | If the sum of the line items (including any tax line items) > (sum of payments received - sum of refunds paid out) | Partially paid |
| 3 | If the sum of the line items (including any tax line items) < (sum of payments received - sum of refunds paid out) | Pending refund |
| 4 | If the sum of the line items (including any tax line items) = (sum of payments received - sum of refunds paid out) | Completed |
2. Changes to the User interface:
| Screen | How it looks currently | How it will work | Changes required |
| ------ | ------ | ------ |------ |
| New contribution screen | ![image](/uploads/568a11ebd7cc0b44edc36077d06f62d8/image.png) | ![Create_new_contribution](/uploads/cb028a89f3c3cc2c3798128efaa4e0d6/Create_new_contribution.png) | We split the "obligation" with the "payment". As such we remove the contribution status field from this screen and instead allow people to decide whether they want to record the "payment" at the same time as recording the "obligation" by checking the "Record payment" checkbox. The contribution status would then be calculated using the logic above. i. |
| View contribution screen (one line item) | ![Screenshot_2022-08-29_at_11.26.55](/uploads/02854f20d41d8ffec5159fa1977627e0/Screenshot_2022-08-29_at_11.26.55.png) | | No Changes |
| View contribution screen (multiple line items) || | |
| Edit contribution screen (one line item) | ![Screenshot_2022-08-29_at_11.30.42](/uploads/6faaed62b313698c1a4346956f48e719/Screenshot_2022-08-29_at_11.30.42.png)| ![Screenshot_2022-08-29_at_11.37.56](/uploads/68293bdc8d48c548606793c6a64d159d/Screenshot_2022-08-29_at_11.37.56.png)| Status field becomes read only and we introduce a button to cancel the contribution. See below for new screen for cancelling a contribution|
| Edit contribution screen (multiple line items) || | |
| New membership screen | ![image](/uploads/568a11ebd7cc0b44edc36077d06f62d8/image.png) | cell | ------ |
| View membership screen (one line item) | | ||
| View membership screen (multiple line items) | | ||
| Edit membership screen (one line item) | | ||
| Edit membership screen (multiple line items) | | ||
| New event participant screen | ![image](/uploads/568a11ebd7cc0b44edc36077d06f62d8/image.png) | ![New_participant](/uploads/7837cf2271b10bb5824d11483221037a/New_participant.png) | Change the checkbox "Record Payment" to "Record Contribution" OR have a toggle to allow the admin to decide whether they are recording a "Paid" or "Free ticket". We only need the financial type and contribution date if we are only recording the contribution (note in https://lab.civicrm.org/dev/core/-/issues/1403 we have agreed to change label of contribution received date to contribution date). The amounts / lines should come from above. |
| View event participant screen (one line item) | | ||
| View event participant screen (multiple line items) | | ||
| Edit event participant screen (one line item) | | ||
| Edit event participant screen (multiple line items) | | ||
- Removehttps://lab.civicrm.org/dev/user-interface/-/issues/48Swap out CiviCRM logos2023-10-07T10:15:12Zjoshjosh@civicrm.orgSwap out CiviCRM logosRequest to replace the current logos displayed in-app and in the "empowered by" display with current CiviCRM logos. Attached files match the current logo dimensions and file names.
![civi99.svg](/uploads/a78c0fe05886784ef83ad0c4ab6c05e0...Request to replace the current logos displayed in-app and in the "empowered by" display with current CiviCRM logos. Attached files match the current logo dimensions and file names.
![civi99.svg](/uploads/a78c0fe05886784ef83ad0c4ab6c05e0/civi99.svg)
![civi99](/uploads/d629d77f01db2750813f08178c264cbb/civi99.png)
![logo_words_small](/uploads/8c5a775cba670ed1854bcf86468e282c/logo_words_small.png)https://lab.civicrm.org/dev/core/-/issues/3740Set default lock level to READ COMMITTED2023-10-20T07:57:25ZJoeMurraySet default lock level to READ COMMITTEDDrupal recently changed the default lock level slightly down from REPEATABLE READ to READ COMMITTED, which has the effect of avoiding some db locks. See https://www.drupal.org/node/3269885 We have seen over recent years some issues with ...Drupal recently changed the default lock level slightly down from REPEATABLE READ to READ COMMITTED, which has the effect of avoiding some db locks. See https://www.drupal.org/node/3269885 We have seen over recent years some issues with deadlocks, primarily on rebuilding group contact cache when there are hierarchies of smart groups. Should CiviCRM be more explicit about what isolation level we expect in order to get more repeatable results, and should we follow Drupal in its move to READ COMMITTED? I tend to think so.
See responses in MM chat from @demeritcowboy
https://chat.civicrm.org/civicrm/pl/gnmhfdxes7rgb8k7ppoaezo45ahttps://lab.civicrm.org/dev/core/-/issues/3738Feature request - APIv4 Get action, provide ability to define aliases for the...2024-03-15T14:56:37Zjustinfreeman (Agileware)Feature request - APIv4 Get action, provide ability to define aliases for the select fields so that external integrations with CiviCRM do not break when the field name changes OR a different field is selectedThis is a Feature request. It would be great if you could define an **alias** for each _select_ **field** when using a **APIv4 Get action**. This alias should then be used as the output in the results for the Get action.
This would enab...This is a Feature request. It would be great if you could define an **alias** for each _select_ **field** when using a **APIv4 Get action**. This alias should then be used as the output in the results for the Get action.
This would enable external CiviCRM integrations to be more robust, by allowing the fields selected in the Get action to be **changed** and but continue to use the **same alias**, thus not changing the structure of the results, the external integration would continue to work.
The other benefit would be that very long CiviCRM field names could be reduced to something shorter or more logical. eg. Instead of "email.email" the alias could just be set to "email". Or instead of "email_greeting_id:label" the alias could just be "email_greeting".
![image](/uploads/2ccae5082e4f50f5fc924d8e6998ffaa/image.png)https://lab.civicrm.org/dev/core/-/issues/3730Search Kit: data exported to spreadsheet as strings where numbers expected2023-08-25T14:47:13ZAndreasandreas.howiller@civiservice.deSearch Kit: data exported to spreadsheet as strings where numbers expectedOverview
----------------------------------------
When exporting spreadsheets from Search Kit in some cases data types seem to be exported incorrectly as strings where they should be numbers.
Reproduction steps
------------------------...Overview
----------------------------------------
When exporting spreadsheets from Search Kit in some cases data types seem to be exported incorrectly as strings where they should be numbers.
Reproduction steps
----------------------------------------
1. Create Search Kit search for **Contributions** on demo.
2. Add columns for **Total Amount** and **Date Received** and click **Search**.
3. Export results via **Action** → **Download Spreadsheet** and choose **.ods** (or **.xlsx**).
4. Open document with office software.
Current behaviour
----------------------------------------
While the Contribution ID column is correctly formatted as a number, Total Amount and Date Received are formatted as **strings**.
Note: Numbers are also left-aligned in Search Kit exports (column A).
Here is what the file looks like in LibreOffice 6.4.7.2:
![grafik](/uploads/ede5de9d9898448c1e0d2676863cfc17/grafik.png)
…and here the corresponding XML in content.xml:
![grafik](/uploads/9487cc0e42f0980eb08e958bea23e7f4/grafik.png)
Expected behaviour
----------------------------------------
Numbers and Date fields are exported in correct data type.
Additional ideas / Comments
----------------------------------------
- it would probably be very useful to have **more Field Transformations** to have some choice for data types, esp. to be able to output money values with our without money signs
- maybe numbers should not be left-aligned anymore (as it is the standard behaviour in spreadsheets)
Environment information
----------------------------------------
* CiviCRM: 5.52.alpha1 and 5.50.3
* PHP: 7.4
* CMS: WordPress 6.0 and Drupal 9.3.14https://lab.civicrm.org/dev/core/-/issues/3714CiviCRM Contribution Page, total amount calculation is sometimes off by one o...2023-07-18T05:40:43Zjustinfreeman (Agileware)CiviCRM Contribution Page, total amount calculation is sometimes off by one or two cents, which causes incorrect amount to be paid or transaction to fail depending on the Payment Processor being usedWhen using a **CiviCRM Contribution Page** the total amount calculation is sometimes off by one or two cents, which causes incorrect amount to be paid or transaction to fail depending on the Payment Processor being used.
In the case of ...When using a **CiviCRM Contribution Page** the total amount calculation is sometimes off by one or two cents, which causes incorrect amount to be paid or transaction to fail depending on the Payment Processor being used.
In the case of Stripe, the _correct amount_ is _pre-authorised_. When the payment is _confirmed_ the _incorrect amount_ is used and because the two amounts do not match, the **transaction fails**. As a result this bug prevents payments from working with Stripe as a payment processor for certain amounts.
# Steps to reproduce
1. Enable tax and invoicing
2. Add e.g. a 10% GST account to Member Dues
3. Create a membership prices set with one field that has three checkbox options:
4. 318.18182 (350 incl tax)
5. 22.72727 (25 incl tax)
6. 24.54545 (27 after tax - note that this would actually work out to 27.01 with the older premature rounding based off-by one issue)
7. Use this price set on a contribution page
8. Submit a test using this contribution page.
9. Note in the summary that the line totals are 350.00, 25.00, and 27.00
## Expected result
Total should be **$402**
## Actual result
Total is **$401.99** - which is one cent less than the expect value, and two cents less than the value you would get with premature rounding.
## Tested environment
Bug has been confirmed using **CiviCRM 5.52.alpha1** on https://wpmaster.demo.civicrm.org/
## Workaround
Adjust the amount to account for the rounding error. In this case, change the amount: 24.54545 to 24.5363636364 which then changes the total to $401.99. This is then matches the total on the Thank You page.
![image](/uploads/ddf3c7cbeaf3cdc4700079f79b01c7d6/image.png)
![image](/uploads/5eb33f84dfa49cad5697570a019c6936/image.png)
# Contribution page with membership options selected - correct total is shown
![image](/uploads/f35b520763e5e9f3545585f27ff42812/image.png)
# Thank you page - incorrect total is shown
![image](/uploads/d6fa209c43cbf2d721d4cf085882b1f8/image.png)
# Set up - Membership Types
![image](/uploads/01450b1f2a777f3c279e694d5ea7e007/image.png)
# Set up - Priceset
![image](/uploads/672f5c3de5b72e7817dc063388f08c38/image.png)
Agileware Ref: CIVICRM-2007https://lab.civicrm.org/dev/core/-/issues/3711Permissions reset on upgrade or configuration change2022-08-25T21:41:06ZAdam WoodPermissions reset on upgrade or configuration changeApplies to CiviCRM running on Joomla.
The permission "See CiviCRM is installed" keeps resetting by itself. This definitely occurs whenever CiviCRM is upgraded (issue observed up to and including 5.50.4) and/or an extension is installed,...Applies to CiviCRM running on Joomla.
The permission "See CiviCRM is installed" keeps resetting by itself. This definitely occurs whenever CiviCRM is upgraded (issue observed up to and including 5.50.4) and/or an extension is installed, enabled/disabled, updated etc, and may occur at other times. I cannot discern the pattern!
This means that CiviCRM disappears from the 'Components' administrator menu unless you are logged in as Super Administrator.
Since CiviGrant was migrated to an extension, the same issue now applies to "access CiviGrant", "edit grants" and "delete in CiviGrant" - I have to keep re-applying these after each upgrade.https://lab.civicrm.org/dev/core/-/issues/3705CiviReport email sending fails if multiple recipient are specified in 'to' field2024-03-05T05:24:43ZKurund JalmiCiviReport email sending fails if multiple recipient are specified in 'to' field### Steps to replicate
1. Create report an instance
2. Configure `Email Delivery Settings`; add multiple emails in `to` field.
3. Configure this report instance to send emails via Scheduled Job
Emails are never sent
## Error in CiviCR...### Steps to replicate
1. Create report an instance
2. Configure `Email Delivery Settings`; add multiple emails in `to` field.
3. Configure this report instance to send emails via Scheduled Job
Emails are never sent
## Error in CiviCRM logs
```bash
[error] $Fatal Error Details = Array
(
[callback] => Array
(
[0] => CRM_Core_Error
[1] => exceptionHandler
)
[code] => 10006
[message] => Failed to send data [SMTP: Invalid response code received from SMTP server while sending email. This is often caused by a misconfigura
tion in Outbound Email settings. Please verify the settings at Administer CiviCRM >> Global Settings >> Outbound Email (SMTP). (code: 554, response: Tra
nsaction failed: Domain contains illegal character)]
[mode] => 16
[debug_info] =>
[type] => PEAR_Error
[user_info] =>
[to_string] => [pear_error: message="Failed to send data [SMTP: Invalid response code received from SMTP server while sending email. This is often caused by a misconfiguration in Outbound Email settings. Please verify the settings at Administer CiviCRM >> Global Settings >> Outbound Email (SMTP). (code: 554, response: Transaction failed: Domain contains illegal character)]" code=10006 mode=callback callback=CRM_Core_Error::exceptionHandler prefix="" info=""]
)
```
This happens because multiple emails gets converted in to invalid emails due to `CRM_Utils_Mail::formatRFC822Email()`https://lab.civicrm.org/dev/core/-/issues/3703Confirmation emails are not sent if it is a recurring payment2022-08-25T17:04:50ZMariaVConfirmation emails are not sent if it is a recurring payment**Current behavior apparently:**
One-time donations: A confirmation email comes right after you fill out the donation form
Recurring donations: A confirmation email comes each time after the payment is processed
This works quite well ...**Current behavior apparently:**
One-time donations: A confirmation email comes right after you fill out the donation form
Recurring donations: A confirmation email comes each time after the payment is processed
This works quite well for online payment processors like PayPal, where the first payment is made within seconds of submitting the donation form.
With "asynchronous payment processors" such as SEPA, however, this approach works quite poorly, because it can take several weeks before the next payment run happens (usually once a month).
Currently this does not work at all.
Moreover, from a fundraiser's point of view, it is a real disadvantage if these confirmations are sent with every payment: People will be reminded every time that they still haven't cancelled their recurring donations.
**In contrast, the behavior most of us would like to see would be:**
One-time and recurring donations: A confirmation email comes right after you fill out the donation form
Further confirmations for recurring donations will only come if this has been set in a configuration setting.
**My suggestion:**
There could be an additional radio button:
One option that will be choosed by default which explains that the mail is only sent for recurring donations when a payment has actually been received (and for one-time donations directly). And for this purpose, there will be another option that says that a confirmation will be sent once directly after the form has been submitted - regardless of whether it is a one-time or recurring donation.
This will ensure that this function will continue to work for everyone who actually wants to use it that way - even if there are probably not many.https://lab.civicrm.org/dev/core/-/issues/3702DOMDocument::loadHTML(): Tag af-field invalid in Entity when loading dashboard2022-10-20T20:07:11ZDaveDDOMDocument::loadHTML(): Tag af-field invalid in Entity when loading dashboardI'm not sure if this is new or not. This might be something coming up for me now with php8. The error comes up in phpquery which is now provided by vendor/rubaboquery which is new, but I'm not sure it's from any differences there. I thin...I'm not sure if this is new or not. This might be something coming up for me now with php8. The error comes up in phpquery which is now provided by vendor/rubaboquery which is new, but I'm not sure it's from any differences there. I think the error is technically legit.
What's happening is during hook_civicrm_angularModules, afform loads all the blocks, such as this one: https://github.com/civicrm/civicrm-core/blob/6fa8c627a2f406329370f2b03d656dbc565ccb9f/ext/afform/core/ang/afblockContactAddress.aff.html
At https://github.com/civicrm/civicrm-core/blob/6fa8c627a2f406329370f2b03d656dbc565ccb9f/ext/afform/core/Civi/Afform/Symbols.php#L38, it creates a DOMDocumentWrapper around it, which is provided by rubaboquery. Note that it considers the block html (as opposed to e.g. xml), so calls DOMDocument->loadHTML. So I believe what's happening is that loadHTML validates it against the html schema, and `<af-field>` is not a real html tag.
See also https://www.php.net/manual/en/domdocument.loadhtml.php#125526, although I'm not saying that is the best solution. I don't know at the moment.5.56.0https://lab.civicrm.org/dev/core/-/issues/3695Search Kit: Commas in event types break IN operator2024-03-06T22:16:53ZJonGoldSearch Kit: Commas in event types break IN operatorWhen you have an event type with a comma in it, you can't search for it using the `WHERE` clause using the `IN` operator.
### Steps to Replicate
* Create a new event type with a comma in the label (which will propagate to the name).
* C...When you have an event type with a comma in it, you can't search for it using the `WHERE` clause using the `IN` operator.
### Steps to Replicate
* Create a new event type with a comma in the label (which will propagate to the name).
* Create an event with the new event type.
* Create a SK search with a `WHERE` event type `is one of` the new event type.
### Expected Result
The new event will appear.
### Actual Result
Event doesn't appear.
See screenshots comparing `=` and `is one of`.
### Comments
SK assumes that `name` fields can't have commas, so it can explode the string on a comma delimiter. This is the API query for the screenshot above (below).
If event types were their own table, I'd suggest making the `name` field be snake-cased. However, this goes against a long-standing (and probably unwise) design decision around `civicrm_option_value` names, which can be used to match on import with external systems. So this may have to get fixed in SK.
```
{
"version": 4,
"select": [
"id",
"title",
"event_type_id:label"
],
"orderBy": {},
"where": [
[
"event_type_id:name",
"IN",
[
"Development",
"Administration"
]
]
],
"groupBy": [],
"join": [],
"having": []
}
```
![Selection_1550](/uploads/03d0a44e852b431289d54eb9b9951d6e/Selection_1550.png)
![Selection_1551](/uploads/afce4bfb38cd7c3e6cd9e979e0f0c463/Selection_1551.png)https://lab.civicrm.org/dev/core/-/issues/3682Feature request - `{domain.logo}`2023-09-25T19:36:31ZeileenFeature request - `{domain.logo}`Tim & I discussed the other day adding the `{domain.logo}` token. As always when this comes up we started with 'it would be very easy to do this minimal version' and by the end of the discussion the scope was un-manageable as unsponsored...Tim & I discussed the other day adding the `{domain.logo}` token. As always when this comes up we started with 'it would be very easy to do this minimal version' and by the end of the discussion the scope was un-manageable as unsponsored work.
However, I think the thing we CAN do is document it....
**The easy way**
`{domain.logo}`
refers to the image (if any) in the image_url field for the domain contact. That's it.
Probably we would need a version with `<img>` tags & one without.
**But then ... sizing & ^^ is too basic/ limited**
So next most complex is to use a setting. Once we start down this path we wind up having to develop a UI, handling sizing and it seems we might as well skip to....
**Mailing Component tokens**
In this version we would add a new component type to `civicrm_mailing_component` and rows of this type would be available as tokens using `{domain.{civicrm_mailing_component.name}}`. Placeholders ones for '{domain.logo}' and '{domain.header}' would be added. The existing UI could be used.
**And still spiralling**
- from there we got into trying to make it possible to nest the above, to use regions, to do the sort of thing the `entity_messages` token attempted to whereby we could store entity-related snippets (the idea was both to be able to insert a contribution specific block and to store entity-specific text to re-use - e.g to re-send an invoice with the original form-base message text)https://lab.civicrm.org/dev/core/-/issues/3648SMS limit is hardcoded at 460 but should be changeable somewhere2024-01-29T09:44:22ZjasonobrownSMS limit is hardcoded at 460 but should be changeable somewhereTwillio (and probably other sms providers) allows SMS messages up to 1600 characters, but Civi has the limit hardcoded at 460. I've verified that by simply going to CRM/SMS/Provider.php and changing MAX_SMS_CHAR from 460 to 1600 the sys...Twillio (and probably other sms providers) allows SMS messages up to 1600 characters, but Civi has the limit hardcoded at 460. I've verified that by simply going to CRM/SMS/Provider.php and changing MAX_SMS_CHAR from 460 to 1600 the system now accepts (and sends) messages up to 1600 characters. It seems logical that the character limit should able to be adjusted and stored somewhere rather than set as a static value in the code. We are currently using the larger limit, and would really like to see this implemented asap so that we can stop manually updating the code each time we update civi.https://lab.civicrm.org/dev/core/-/issues/3645Eventually google will require OAUTH for bounce processing and may require it...2022-12-08T20:55:07ZDaveDEventually google will require OAUTH for bounce processing and may require it for outbound SMTP through gmail serversAs noted on [stackexchange](https://civicrm.stackexchange.com/questions/34142/sending-mail-via-gmail-google-announce-username-password-authentication-will-b), Google sent out an email outlining plans to require OAUTH and discontinue supp...As noted on [stackexchange](https://civicrm.stackexchange.com/questions/34142/sending-mail-via-gmail-google-announce-username-password-authentication-will-b), Google sent out an email outlining plans to require OAUTH and discontinue support for just username/password.
The way I read it, this is mostly applying to POP/IMAP clients for reading mail, and they are *hinting* that someday they might require it for outbound SMTP, i.e. at Administer - System Settings - Outbound Mail you choose SMTP and have it set to use gmail servers. The relevant quote is
> No change is required for scanners or other devices using simple mail transfer protocol (SMTP) or LSAs to send emails. If you replace your device, look for one that sends email using OAuth.
(LSA means less secure apps, which is google-speak for username/password)
They are targeting June 2020 - Feb 2021 for IMAP etc, but as noted above they have not specifically set a date for changes regarding sending outbound emails.
It occurs to me though that bounce processing and the email processor use POP/IMAP. For the email processor you can work around it by just forwarding it to a non-gmail address and polling that, but bounce processing might be harder depending on setup.https://lab.civicrm.org/dev/core/-/issues/3613System Workflow Messages - Improve localization experience2024-01-14T14:55:57ZtottenSystem Workflow Messages - Improve localization experience## Background
Several CiviCRM components - such as CiviContribute, CiviCase, and CiviEvent - send automated email notifications. Many organizations find it useful to customize/tune these emails, and the emails may be administered in the...## Background
Several CiviCRM components - such as CiviContribute, CiviCase, and CiviEvent - send automated email notifications. Many organizations find it useful to customize/tune these emails, and the emails may be administered in the web UI ("Administer: Message Templates: System Workflow Messages").
In principle, this is a powerful tool in which an administrator may use Smarty templating to produce highly tuned messages -- and it requires only the ability to understand template notations (e.g. `{$contact.first_name}` or `{$event.start_date}`). However, in practice, this is often difficult -- real-world templates are long and complex, and they may be used with data of varying quality/character, and the workflow to test a template can be tedious.
## Goals
This epic (*overarching issue*) aims to improve the experience of maintaining system workflow templates. We are specifically funded to work on it from the following perspective: You have an organization which maintains multiple templates for use in multiple countries/locales -- periodically, as campaigns/operations are revised, you may need to engage with a few people to update+retest a few templates. We have three main themes to improve upon:
* __Drafting/Workflow__: Currently, Civi has exactly one variant/revision of a template, and it is always live. There is no way to develop a draft (new revisions) or to keep a record of past revisions.
* __~~Translation~~ Localization__: For any given message, you may need to compose different versions for different locales.
* __Previews/Samples__: Many templates are sensitive to fine-grained details. This is true in multiple ways, e.g. (1) the template adjusts based on *available data* ("Do we have a mailing address for this person? Do we have a full-name, or just an email? Is this one-off contribution or a recurring contribution?"), and (2) the template includes *precise codes* that must be well-formed (e.g. matching tags, matching quotes, using an email-friendly dialect of HTML/CSS).
## Related work
* https://lab.civicrm.org/extensions/msgtpltools
* https://lab.civicrm.org/community/feature-request/-/issues/26
* https://github.com/eileenmcnaughton/civi-data-translate
* https://lab.civicrm.org/extensions/l10nx
## Subtopics / Brainstorming
(Some of the descriptions here are terse - but they may get more explanation in the wireframe.)
* __Sample data presentation__: While composing a message, how do you show a preview based on sample data? Some ideas:
* __Edit-Preview Split-Panes__
* __Pro__: very amenable to live-update
* __Con__: needs wide screen (prob dialog). agreeable with O(10) sample records - but disagreeable with O(1000) records
* __Edit-Preview Accordion__
* __Pro__: fairly amenable to live-update. full width.
* __Con__: agreeable with O(10) sample records - but disagreeable with O(1000) records
* __Explore-Preview Panes in dialog__
* __Pro__: agreeable with O(1000) records
* __Con__: requires more navigating.
* __Comment__: live-update might be possible with separate popup - but more difficult
* __Comment__: might mitigate navigation work with a hotkey
* __Sample data source__: When viewing a preview based on some example record(s), where does the example come from?
* __Basic/prepackaged__: Define some JSON files, which are linked to the workflows.
* __Comment__: Requires metadata to say, "workflows $a+$b can use sample data $c+$d"
* __Pro__: Some templates have very nuanced purposes and may have special vars. This allows you to mock data that's attuned to edge-cases, and you can use these samples in any environment (dev-site/public-demo-site/live-site). No mixing-up real records and sample records. Can simulate complex data (e.g. "tpl for use-case with 2 contribution records")
* __Con__: More upfront work in making sample data. User cannot test against real data.
* __Comment__: Should have unit-test to ensure sample-data-structure stays in sync with real-data-structure
* __Data search__: Search for a live record (eg `Contribution` or `Membership`) to use as a sample.
* __Comment__: Requires metadata to say, "workflows $a+$b need data from entity $c"
* __Pro__: You can use real data, which can be subjectively attuned.
* __Con__: You have to find real data that's *useful*. May be awkward if you're trying to test edge-cases, unless you're willing to add mock/synthetic records to the DB. All mail-merge data must be selectable from one record (eg "Contribution #123"; eg no other inputs besides DB content).
* __Variants__: The UI for this can take on a few variations, eg
* In "Edit Msg Tpl", show a searchable tree. Filter by contact name and select the target record.
* In "Edit Msg Tpl", ask the user to enter a numeric ID.
* In "View Contact", add links/actions for generating previews.
* __Flagged records__: As an adminsitrator, find real records and save them in a list (e.g. "Record #$x is a sample for use in workflows $a, $b")
* __Comment__: Requires somewhere to store the list (e.g. custom-field or new-field or new-table)
* __Pro__: You can use real data, which can be subjectively attuned. The list of samples is curated/abbreviated.
* __Con__: The individual composing the template may not know how to flag/unflag sample records. Requires configuration (and extant samples) before you can use email preview.
* __Grain of ~~translation~~ localization__
* __Record level__: For each `(locale,workflow)`, create another `civicrm_msg_template` record.
* __Field level__: Each `(workflow)` only has one `civicrm_msg_template` record. Attached to that record are multiple translations (ex: `field=subject,locale=fr_FR,text="Bonjour"`).
* __String level__: Each `(workflow)` only has one `civicrm_msg_template` record, and that record has one `body_text`. However, the `body_text` uses translatable strings (ex: 'body_text={cts id=greeting}Greetings,{/cts}`)
* Trade-off
| Record-level | Field-level | String-level |
| -- | -- | -- |
| Each translation goes through a separate, well-delineated drafting workflow. | Translation workflow is batched (per template). All translations must be ready to activate together. | Orthogonal workflows for templates vs strings. |
| Message structure (paras/sections) is localized. | Message structure (paras/sections) is localized. | Message structure is uniform across locales. (Realistic? What abouts `<table>` LTR/RTL? |
| Cannot re-use translated snippets. | Cannot re-use translated snippets. | Re-use translated snippets. |
## Wireframes
There are a few different ways to approach this. To get a sense for the usability and work, we can post a few alternative wireframes. (*This section may be updated as more are added.*)
* __(0) Status quo__: [_0__Sys_Wf_Msgs](/uploads/ad96003d71133d298f95ce79902a7583/_0__Sys_Wf_Msgs.png), [_0__Edit_Msg_Tpl](/uploads/81bc234651182b65993bc28c8052b68b/_0__Edit_Msg_Tpl.png)
* __(1) Record-level translations w/previews and drafting workflow__: [_1Rec__Sys_Wf_Msgs](/uploads/532b842406c17c31be5ae9195ceb6aa8/_1Rec__Sys_Wf_Msgs.png), [_1Rec__Edit_Msg_Tpl](/uploads/13a1f717e829e3de480750c4893969a7/_1Rec__Edit_Msg_Tpl.png) (*Note: The blue-arcs indicate a couple variations on how to include the preview/sample functionality. This tracks close to status-quo, and incremental changes are highlighted with yellow-dots.*)
* __(2) Field-level translations__: [_2Fld__Sys_Wf_Msgs](/uploads/7e1f46e722d8afc706a92c56e295c55e/_2Fld__Sys_Wf_Msgs.png), [_2Fld__Edit_Msg_Tpl](/uploads/4bc932b7bb3f304853b0c107b98d41b5/_2Fld__Edit_Msg_Tpl.png) (*Note: This tracks close to status-quo, and incremental changes are highlighted with yellow-dots.*)
* __(3) String-level translations__: [_3Subfld__Sys_Wf_Msgs](/uploads/f52ffbb764917954790fbbd3ed807acf/_3Subfld__Sys_Wf_Msgs.png), [_3Subfld__Edit_Msg_Tpl](/uploads/b20bc83d302d55a485d975aff6ca70e1/_3Subfld__Edit_Msg_Tpl.png) (*Note: This tracks close to status-quo, and incremental changes are highlighted with yellow-dots.*)
* __(4) Record-level translations w/browse-edit tree + accordion-preview__: [_4Tree__Browse-Edit](/uploads/8cc47238a9d597f80950c1d98c0ea43a/_4Tree__Browse-Edit.png)
* __(5) Field-level translations w/browse-edit tree + accordion-preview__ [_5Tree__Browse-Edit](/uploads/aab231199d0b4217a2f44f9ad45aecb1/_5Tree__Browse-Edit.png)
* __(6) Field-level translations w/matrix, preview-dialog, and drafting workflow__: [_6Mat__Sys_Wf_Msgs](/uploads/3a32390f36f50d8e158e1155dc586969/_6Mat__Sys_Wf_Msgs.png), [_6Mat__Edit_Msg_Tpl](/uploads/353f97846fd9d2b1e037ec92e60101fc/_6Mat__Edit_Msg_Tpl.png) (*Note: This assumes that there is a separate string-storage service like [civi-data-translate](https://github.com/eileenmcnaughton/civi-data-translate), and it assumes that the `String` record has some workflow property like like `is_draft` or `revision`. So each string might have the properties `entity`,`entity_id`,`field`,`locale`,`text`,`is_draft`.*)
* __(7) As with 6, but with various refinements__: [_7Mat__Sys_Wf_Msgs](/uploads/73d37acab4307054b4fe4f3291ae7dcd/_7Mat__Sys_Wf_Msgs.png),
[_7Mat__Edit_Msg_Tpl](/uploads/4807652bd20bfb5bf0912bf24f05d444/_7Mat__Edit_Msg_Tpl.png)https://lab.civicrm.org/dev/core/-/issues/3587Unable to load the recipients group when sending mailing with Civimail2023-10-05T15:41:24ZmarzastUnable to load the recipients group when sending mailing with CivimailI am running CiviCRM 5.17.4 on Wordpress 5.2.3. The problem I am facing is that I am unable to send mailing to mailing lists (defined as groups) via CiviMail.
When I try to enter the group in the Recipients box I get Loading Failed mess...I am running CiviCRM 5.17.4 on Wordpress 5.2.3. The problem I am facing is that I am unable to send mailing to mailing lists (defined as groups) via CiviMail.
When I try to enter the group in the Recipients box I get Loading Failed message and none of the available groups is loaded (see image below). The problem appears for existing and new mailings.
Interestingly when I re-use the existing mailing I can still send it to a previously used group (though I cannot change the recipient lists). The groups load normally in manage group page.
![aE8nz](/uploads/dca6a15c237cbb7612df433a31b769d5/aE8nz.png)
*Issue already reported on CiviCRM Stack Exchange: https://civicrm.stackexchange.com/questions/33070/unable-to-load-the-recipients-group-when-sending-mailing-with-civimail*https://lab.civicrm.org/dev/core/-/issues/3586Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process ...2023-09-12T02:13:33Zjustinfreeman (Agileware)Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process any emails if a single email is sent to the bounce mailbox with an invalid returnPathCiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service ema...CiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service email" causing all other emails to remain unprocessed. CiviCRM mail reports are therefore incorrect and ultimately the CiviCRM Fetch Bounces scheduled job fails to perform the job it was designed to do.
Example invalid returnPaths as observed in production:
```
x2108-arfsfufwyfgkvni2x7ww5wvac3eifc2wx3vr1fcwbbysieqppis4ytnyip09xcohck9s1vrm77nbg4xh431tthdsmtmfvfpa@rspf.mxthunderstruck.net
```
```
SRS0=3Ypr33=QM=crm.nothappyjohn.org.au=bounce+b.25211.6039846.eb7e7e8ba282f9d0@airquotes.com.au
```
This problem has been highlighted by the introduction of "Scheduled Job Failures" feature which raises the visibility of this type of problem.
Bounce mailboxes have been observed with 10k to 50k unprocessed emails, depending on the time when the "Denial of service email" was received.
Agileware Ref: CIVICRM-1970https://lab.civicrm.org/dev/core/-/issues/3584Denial of service - CiviCRM Mailing Job, if there are 5 errors sequentially w...2023-11-09T13:25:31Zjustinfreeman (Agileware)Denial of service - CiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and mailing will not sendCiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and the mailing will not send.
An invalid email domain returns an SMTP error 451: _SMTP error 451 Unable to ...CiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and the mailing will not send.
An invalid email domain returns an SMTP error 451: _SMTP error 451 Unable to complete command, DNS not available or timed out 451 Domain of sender address does not resolve_
When the CiviCRM Mailing Job executes again, the process repeats, the same invalid email addresses are used, the same errors are returned and the mailing again will not send.
This process will repeat until someone intervenes, cancelling the mailing, locating and removing the contacts which have the invalid email addresses.
The CiviCRM Mailing Job does not currently check the actual SMTP error code and just treats all SMTP errors as "SMTP Connection Errors". Counting up to 5 and then aborting the job when the threshold is reached.
I think it's fair to call this a "denial of service" because a bad actor could sign up a bunch of fake Contacts to a CiviCRM mailing list with either invalid email domains or valid email domains which are then rendered invalid - and thus cause that mailing list to cease processing.
**Possible solutions**
Change the CiviCRM Mailing Job to check the return SMTP error code and only increment the count for valid "SMTP Connection Errors".
Change the CiviCRM Mailing Job to continue with the mailing, regardless of the 5 errors encountered, which will also skip sending the emails to those Contacts with an invalid email domain. As shown in the patch below.
```
Index: CRM/Mailing/BAO/MailingJob.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/CRM/Mailing/BAO/MailingJob.php b/CRM/Mailing/BAO/MailingJob.php
--- a/CRM/Mailing/BAO/MailingJob.php (revision 20a0376709a4616e32689a821a7e95ca255ed85f)
+++ b/CRM/Mailing/BAO/MailingJob.php (date 1620794367592)
@@ -672,20 +672,9 @@
if ($smtpConnectionErrors <= 5) {
$mailer->disconnect();
$retryGroup = TRUE;
+ CRM_Core_Error::debug_log_message("More than 5 consecutive SMTP Socket Errors. Re-starting mailer.");
continue;
}
-
- // seems like we have too many of them in a row, we should
- // write stuff to disk and abort the cron job
- $this->writeToDB(
- $deliveredParams,
- $targetParams,
- $mailing,
- $job_date
- );
-
- CRM_Core_Error::debug_log_message("Too many SMTP Socket Errors. Exiting");
- CRM_Utils_System::civiExit();
}
// Register the bounce event.
Index: ext/flexmailer/src/Listener/DefaultSender.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/ext/flexmailer/src/Listener/DefaultSender.php b/ext/flexmailer/src/Listener/DefaultSender.php
--- a/ext/flexmailer/src/Listener/DefaultSender.php (revision 20a0376709a4616e32689a821a7e95ca255ed85f)
+++ b/ext/flexmailer/src/Listener/DefaultSender.php (date 1620794390628)
@@ -76,15 +76,9 @@
if ($smtpConnectionErrors <= 5) {
$mailer->disconnect();
$retryBatch = TRUE;
+ \CRM_Core_Error::debug_log_message("More than 5 consecutive SMTP Socket Errors. Re-starting mailer.");
continue;
}
-
- // seems like we have too many of them in a row, we should
- // write stuff to disk and abort the cron job
- $job->writeToDB($deliveredParams, $targetParams, $mailing, $job_date);
-
- \CRM_Core_Error::debug_log_message("Too many SMTP Socket Errors. Exiting");
- \CRM_Utils_System::civiExit();
}
else {
$this->recordBounce($job, $task, $result->getMessage());
```
Agileware Ref: CIVICRM-1728https://lab.civicrm.org/dev/core/-/issues/3567CiviMail: Provide transparency about (non)recipient details2023-09-05T10:28:43ZtottenCiviMail: Provide transparency about (non)recipient detailsTwo stories which are not-uncommon:
## Story 1
Someone starts using CiviMail. They spend a bunch of time arranging for the list of contacts (e.g. migrating from a previous system) and then go to send a mailing. They start composing a m...Two stories which are not-uncommon:
## Story 1
Someone starts using CiviMail. They spend a bunch of time arranging for the list of contacts (e.g. migrating from a previous system) and then go to send a mailing. They start composing a message and realize that the number of recipients is way off. What's going on? They start searching Google, Stackexchange, etc trying to understand why. Eventually, they learn that there are a half-dozen rules about who gets included or excluded from a mailing. (*Does the contact have an email address? Are they deceased? Have they opted-out of the specific mailing list? Have they opted-out of email generally? Have they had bounces? Does an extension - like GDPR - add more constraints?*) But they haven't solved the issue yet: now they need to figure out which criteria are affecting them, to what extent, and how to adapt.
And of course people have this problem - the "New Mailing" UX only presents the affirmative list of active recipients. It doesn't give any hint or detail about the omitted recipients.
![Screen_Shot_2019-09-03_at_12.58.40_PM](/uploads/73d40846bf379562ed119a317163b4e9/Screen_Shot_2019-09-03_at_12.58.40_PM.png)
## Story 2
A new user composes an email blast and discovers this incredible drop-down button with a list of tokens. They start throwing in tokens like there's no tomorrow. The blast goes out, and recipients complain: "Why are you sending me these weird messages with skipped words?"
And, again, of course people have this problem - the "New Mailing" UX lets you choose tokens without knowing if your recipient-data is up to the job.
## Discussion
These two stories are thematically similar: in both cases, we have a long list of potential recipients, and we have obscure/hidden criteria which determines whether the mailing goes out successfully to the target audience.
In both cases, there needs to be more transparency - adding UI-elements/UX-mechanisms to communicate what's going on. For example, the data-table in the pop-up might be updated with more columns/icons/color-coding.
I've logged the issue because I think, at some point, there should be discussion around design/resourcing. I don't want to prejudge that by saying the UX mechanisms to resolve these two issues must or must-not be the same - they might be totally different, or they might be one data-table. Either way, I think it's good to have a record of the issue.
## Related Links
* https://chat.civicrm.org/civicrm/pl/nn5rjzwiz3fc78pgojmhubx89a
* https://civicrm.org/blog/simonparkervitiligosocietyorguk/solution-all-recipients-not-showing-up-for-sending-mass-email
* https://civicrm.stackexchange.com/questions/5086/civimail-does-not-send-to-a-whole-group
* https://civicrm.stackexchange.com/questions/6346/after-import-to-group-individuals-in-that-group-not-showing-up-as-recipients-to
* https://civicrm.stackexchange.com/questions/26320/contacts-visible-in-mailing-list-group-not-on-hold-or-otherwise-dnd-but-skippe
* https://civicrm.stackexchange.com/questions/18199/in-mailing-all-recipients-are-not-showing/18201#18201
* https://civicrm.org/extensions/zombie-check