Development issueshttps://lab.civicrm.org/groups/dev/-/issues2024-02-22T04:18:48Zhttps://lab.civicrm.org/dev/core/-/issues/5004Wrong Contribution ID passed to TokenValueEvent and Duplicate Invoice Numbers...2024-02-22T04:18:48Zginkgomzdginkgomzd@fastmail.comWrong Contribution ID passed to TokenValueEvent and Duplicate Invoice Numbers for Contributions created around same timeNew description after diagnosis:
Code was setting invoice number to a predicted next ContributionID instead of waiting for the Contribution ID to be set.
----
Overview
----------------------------------------
When two contributions are ...New description after diagnosis:
Code was setting invoice number to a predicted next ContributionID instead of waiting for the Contribution ID to be set.
----
Overview
----------------------------------------
When two contributions are made at the same time (same datetime to the second), one of the contributions is assigned a duplicate invoice_number. The invoice number is assigned based on a Contribution ID, therefore, we infer that the Contribution ID is corrected later in the request handling, but the invoice ID is not updated and is therefore a duplicate of another existing contribution.
There is no token for invoice number and we are not using the System Workflow messages that make it available as a variable in Smarty processing.
Our custom token sets it from the Contribution ID passed to TokenValueEvent. Perhaps this is a better framing of the bug over-all... the incorrect Contribution ID is passed to the TokenValueEvent. This is probably inconsequential for most cases since Contribution ID is not a likely token or token dependency.
Reproduction steps
----------------------------------------
It has been observed multiple times in our high-volume event registrations. Reproduction would require submitting simultaneous requests. This might involve some luck, but maybe an appropriately high load compared to the system configuration would trigger the condition reliably.
We are using the payflowpro payment processor extension.
Current behaviour
----------------------------------------
Two contributions, with different but sequential Contribution ID's, share the same invoice_number and are both returned in a search by invoice number.
Expected behaviour
----------------------------------------
The invoice_number should be unique and is expected to be derived from Contribution ID and invoice prefix.https://lab.civicrm.org/dev/core/-/issues/2963Export file empty when utilizing "Display results as" and searching for custo...2023-01-10T16:46:00Zfabian_SYSTOPIAExport file empty when utilizing "Display results as" and searching for custom dataOverview
----------------------------------------
When exporting data after an advanced search the exported file is empty if:
* "Display results as" was set to "related contacts" AND
* one search criterium was a custom field
The expor...Overview
----------------------------------------
When exporting data after an advanced search the exported file is empty if:
* "Display results as" was set to "related contacts" AND
* one search criterium was a custom field
The export preview shows all information correctly but the exported file does not contain any data.
Reproduction steps
----------------------------------------
1. Create a contact custom field (if you do not have one already, ex: Favorite Color: A/B/C)
1. Perform an advanced search, set "Display results as" to "related contacts" and any relationship type (ex: 'Employee Of')
1. Choose at least one custom filed as a search criterium (the search result must contain at least one contact (you may have to create/edit a test contact so that the Custom Field has a value)
1. Export the contact(s) from the search result
Current behaviour
----------------------------------------
The export file is empty although the export's preview shows all information correctly (in case you choose data to export and not export primary fields). Note that the export will work as intended if you do not set "Display results as" to "related contacts" or do not have at least one search criterium was a custom field.
Expected behaviour
----------------------------------------
Export file should contain data of selected contacts
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ all
* __CiviCRM:__ Master/5.35.x, probably also earlier versions
Comments
----------------------------------------
Possibly unrelated but referencing #2873 and assigning to @monish.deb as there is currently a fix for the export feature in progress.Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/3613System Workflow Messages - Improve localization experience2024-01-14T14:55:57ZtottenSystem Workflow Messages - Improve localization experience## Background
Several CiviCRM components - such as CiviContribute, CiviCase, and CiviEvent - send automated email notifications. Many organizations find it useful to customize/tune these emails, and the emails may be administered in the...## Background
Several CiviCRM components - such as CiviContribute, CiviCase, and CiviEvent - send automated email notifications. Many organizations find it useful to customize/tune these emails, and the emails may be administered in the web UI ("Administer: Message Templates: System Workflow Messages").
In principle, this is a powerful tool in which an administrator may use Smarty templating to produce highly tuned messages -- and it requires only the ability to understand template notations (e.g. `{$contact.first_name}` or `{$event.start_date}`). However, in practice, this is often difficult -- real-world templates are long and complex, and they may be used with data of varying quality/character, and the workflow to test a template can be tedious.
## Goals
This epic (*overarching issue*) aims to improve the experience of maintaining system workflow templates. We are specifically funded to work on it from the following perspective: You have an organization which maintains multiple templates for use in multiple countries/locales -- periodically, as campaigns/operations are revised, you may need to engage with a few people to update+retest a few templates. We have three main themes to improve upon:
* __Drafting/Workflow__: Currently, Civi has exactly one variant/revision of a template, and it is always live. There is no way to develop a draft (new revisions) or to keep a record of past revisions.
* __~~Translation~~ Localization__: For any given message, you may need to compose different versions for different locales.
* __Previews/Samples__: Many templates are sensitive to fine-grained details. This is true in multiple ways, e.g. (1) the template adjusts based on *available data* ("Do we have a mailing address for this person? Do we have a full-name, or just an email? Is this one-off contribution or a recurring contribution?"), and (2) the template includes *precise codes* that must be well-formed (e.g. matching tags, matching quotes, using an email-friendly dialect of HTML/CSS).
## Related work
* https://lab.civicrm.org/extensions/msgtpltools
* https://lab.civicrm.org/community/feature-request/-/issues/26
* https://github.com/eileenmcnaughton/civi-data-translate
* https://lab.civicrm.org/extensions/l10nx
## Subtopics / Brainstorming
(Some of the descriptions here are terse - but they may get more explanation in the wireframe.)
* __Sample data presentation__: While composing a message, how do you show a preview based on sample data? Some ideas:
* __Edit-Preview Split-Panes__
* __Pro__: very amenable to live-update
* __Con__: needs wide screen (prob dialog). agreeable with O(10) sample records - but disagreeable with O(1000) records
* __Edit-Preview Accordion__
* __Pro__: fairly amenable to live-update. full width.
* __Con__: agreeable with O(10) sample records - but disagreeable with O(1000) records
* __Explore-Preview Panes in dialog__
* __Pro__: agreeable with O(1000) records
* __Con__: requires more navigating.
* __Comment__: live-update might be possible with separate popup - but more difficult
* __Comment__: might mitigate navigation work with a hotkey
* __Sample data source__: When viewing a preview based on some example record(s), where does the example come from?
* __Basic/prepackaged__: Define some JSON files, which are linked to the workflows.
* __Comment__: Requires metadata to say, "workflows $a+$b can use sample data $c+$d"
* __Pro__: Some templates have very nuanced purposes and may have special vars. This allows you to mock data that's attuned to edge-cases, and you can use these samples in any environment (dev-site/public-demo-site/live-site). No mixing-up real records and sample records. Can simulate complex data (e.g. "tpl for use-case with 2 contribution records")
* __Con__: More upfront work in making sample data. User cannot test against real data.
* __Comment__: Should have unit-test to ensure sample-data-structure stays in sync with real-data-structure
* __Data search__: Search for a live record (eg `Contribution` or `Membership`) to use as a sample.
* __Comment__: Requires metadata to say, "workflows $a+$b need data from entity $c"
* __Pro__: You can use real data, which can be subjectively attuned.
* __Con__: You have to find real data that's *useful*. May be awkward if you're trying to test edge-cases, unless you're willing to add mock/synthetic records to the DB. All mail-merge data must be selectable from one record (eg "Contribution #123"; eg no other inputs besides DB content).
* __Variants__: The UI for this can take on a few variations, eg
* In "Edit Msg Tpl", show a searchable tree. Filter by contact name and select the target record.
* In "Edit Msg Tpl", ask the user to enter a numeric ID.
* In "View Contact", add links/actions for generating previews.
* __Flagged records__: As an adminsitrator, find real records and save them in a list (e.g. "Record #$x is a sample for use in workflows $a, $b")
* __Comment__: Requires somewhere to store the list (e.g. custom-field or new-field or new-table)
* __Pro__: You can use real data, which can be subjectively attuned. The list of samples is curated/abbreviated.
* __Con__: The individual composing the template may not know how to flag/unflag sample records. Requires configuration (and extant samples) before you can use email preview.
* __Grain of ~~translation~~ localization__
* __Record level__: For each `(locale,workflow)`, create another `civicrm_msg_template` record.
* __Field level__: Each `(workflow)` only has one `civicrm_msg_template` record. Attached to that record are multiple translations (ex: `field=subject,locale=fr_FR,text="Bonjour"`).
* __String level__: Each `(workflow)` only has one `civicrm_msg_template` record, and that record has one `body_text`. However, the `body_text` uses translatable strings (ex: 'body_text={cts id=greeting}Greetings,{/cts}`)
* Trade-off
| Record-level | Field-level | String-level |
| -- | -- | -- |
| Each translation goes through a separate, well-delineated drafting workflow. | Translation workflow is batched (per template). All translations must be ready to activate together. | Orthogonal workflows for templates vs strings. |
| Message structure (paras/sections) is localized. | Message structure (paras/sections) is localized. | Message structure is uniform across locales. (Realistic? What abouts `<table>` LTR/RTL? |
| Cannot re-use translated snippets. | Cannot re-use translated snippets. | Re-use translated snippets. |
## Wireframes
There are a few different ways to approach this. To get a sense for the usability and work, we can post a few alternative wireframes. (*This section may be updated as more are added.*)
* __(0) Status quo__: [_0__Sys_Wf_Msgs](/uploads/ad96003d71133d298f95ce79902a7583/_0__Sys_Wf_Msgs.png), [_0__Edit_Msg_Tpl](/uploads/81bc234651182b65993bc28c8052b68b/_0__Edit_Msg_Tpl.png)
* __(1) Record-level translations w/previews and drafting workflow__: [_1Rec__Sys_Wf_Msgs](/uploads/532b842406c17c31be5ae9195ceb6aa8/_1Rec__Sys_Wf_Msgs.png), [_1Rec__Edit_Msg_Tpl](/uploads/13a1f717e829e3de480750c4893969a7/_1Rec__Edit_Msg_Tpl.png) (*Note: The blue-arcs indicate a couple variations on how to include the preview/sample functionality. This tracks close to status-quo, and incremental changes are highlighted with yellow-dots.*)
* __(2) Field-level translations__: [_2Fld__Sys_Wf_Msgs](/uploads/7e1f46e722d8afc706a92c56e295c55e/_2Fld__Sys_Wf_Msgs.png), [_2Fld__Edit_Msg_Tpl](/uploads/4bc932b7bb3f304853b0c107b98d41b5/_2Fld__Edit_Msg_Tpl.png) (*Note: This tracks close to status-quo, and incremental changes are highlighted with yellow-dots.*)
* __(3) String-level translations__: [_3Subfld__Sys_Wf_Msgs](/uploads/f52ffbb764917954790fbbd3ed807acf/_3Subfld__Sys_Wf_Msgs.png), [_3Subfld__Edit_Msg_Tpl](/uploads/b20bc83d302d55a485d975aff6ca70e1/_3Subfld__Edit_Msg_Tpl.png) (*Note: This tracks close to status-quo, and incremental changes are highlighted with yellow-dots.*)
* __(4) Record-level translations w/browse-edit tree + accordion-preview__: [_4Tree__Browse-Edit](/uploads/8cc47238a9d597f80950c1d98c0ea43a/_4Tree__Browse-Edit.png)
* __(5) Field-level translations w/browse-edit tree + accordion-preview__ [_5Tree__Browse-Edit](/uploads/aab231199d0b4217a2f44f9ad45aecb1/_5Tree__Browse-Edit.png)
* __(6) Field-level translations w/matrix, preview-dialog, and drafting workflow__: [_6Mat__Sys_Wf_Msgs](/uploads/3a32390f36f50d8e158e1155dc586969/_6Mat__Sys_Wf_Msgs.png), [_6Mat__Edit_Msg_Tpl](/uploads/353f97846fd9d2b1e037ec92e60101fc/_6Mat__Edit_Msg_Tpl.png) (*Note: This assumes that there is a separate string-storage service like [civi-data-translate](https://github.com/eileenmcnaughton/civi-data-translate), and it assumes that the `String` record has some workflow property like like `is_draft` or `revision`. So each string might have the properties `entity`,`entity_id`,`field`,`locale`,`text`,`is_draft`.*)
* __(7) As with 6, but with various refinements__: [_7Mat__Sys_Wf_Msgs](/uploads/73d37acab4307054b4fe4f3291ae7dcd/_7Mat__Sys_Wf_Msgs.png),
[_7Mat__Edit_Msg_Tpl](/uploads/4807652bd20bfb5bf0912bf24f05d444/_7Mat__Edit_Msg_Tpl.png)https://lab.civicrm.org/dev/drupal/-/issues/81Drupal8: Large export results in logout2019-11-03T16:19:27ZJonGoldDrupal8: Large export results in logoutWhen you attempt a large export of CiviCRM data (~63K contacts, ~15 columns), Symfony kills the session. This is relatively recent behavior (didn't happen in April, for instance). Here are the watchdog logs:
Here are those stack traces,...When you attempt a large export of CiviCRM data (~63K contacts, ~15 columns), Symfony kills the session. This is relatively recent behavior (didn't happen in April, for instance). Here are the watchdog logs:
Here are those stack traces, more readable:
```
Warning: session_start(): Failed to decode session object. Session has been destroyed in Symfony\Component\HttpFoundation\Session\Storage\NativeSessionStorage->start() (line 149 of /var/www/crm.agbu.org/vendor/symfony/http-foundation/Session/Storage/NativeSessionStorage.php)
#0 /var/www/crm.agbu.org/web/core/includes/bootstrap.inc(587): _drupal_error_handler_real(2, 'session_start()...', '/var/www/crm.ag...', 149, Array)
#1 [internal function]: _drupal_error_handler(2, 'session_start()...', '/var/www/crm.ag...', 149, Array)
#2 /var/www/crm.agbu.org/vendor/symfony/http-foundation/Session/Storage/NativeSessionStorage.php(149): session_start()
#3 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/Session/SessionManager.php(164): Symfony\Component\HttpFoundation\Session\Storage\NativeSessionStorage->start()
#4 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/Session/SessionManager.php(118): Drupal\Core\Session\SessionManager->startNow()
#5 /var/www/crm.agbu.org/vendor/symfony/http-foundation/Session/Session.php(57): Drupal\Core\Session\SessionManager->start()
#6 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/StackMiddleware/Session.php(53): Symfony\Component\HttpFoundation\Session\Session->start()
#7 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/StackMiddleware/KernelPreHandle.php(47): Drupal\Core\StackMiddleware\Session->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#8 /var/www/crm.agbu.org/web/core/modules/page_cache/src/StackMiddleware/PageCache.php(106): Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#9 /var/www/crm.agbu.org/web/core/modules/page_cache/src/StackMiddleware/PageCache.php(85): Drupal\page_cache\StackMiddleware\PageCache->pass(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#10 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/StackMiddleware/ReverseProxyMiddleware.php(47): Drupal\page_cache\StackMiddleware\PageCache->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#11 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/StackMiddleware/NegotiationMiddleware.php(52): Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#12 /var/www/crm.agbu.org/vendor/stack/builder/src/Stack/StackedHttpKernel.php(23): Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#13 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/DrupalKernel.php(693): Stack\StackedHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#14 /var/www/crm.agbu.org/web/index.php(19): Drupal\Core\DrupalKernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#15 {main}.
```
```
RuntimeException: Failed to start the session in Symfony\Component\HttpFoundation\Session\Storage\NativeSessionStorage->start() (line 150 of /var/www/crm.agbu.org/vendor/symfony/http-foundation/Session/Storage/NativeSessionStorage.php)
#0 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/Session/SessionManager.php(164): Symfony\Component\HttpFoundation\Session\Storage\NativeSessionStorage->start()
#1 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/Session/SessionManager.php(118): Drupal\Core\Session\SessionManager->startNow()
#2 /var/www/crm.agbu.org/vendor/symfony/http-foundation/Session/Session.php(57): Drupal\Core\Session\SessionManager->start()
#3 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/StackMiddleware/Session.php(53): Symfony\Component\HttpFoundation\Session\Session->start()
#4 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/StackMiddleware/KernelPreHandle.php(47): Drupal\Core\StackMiddleware\Session->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#5 /var/www/crm.agbu.org/web/core/modules/page_cache/src/StackMiddleware/PageCache.php(106): Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#6 /var/www/crm.agbu.org/web/core/modules/page_cache/src/StackMiddleware/PageCache.php(85): Drupal\page_cache\StackMiddleware\PageCache->pass(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#7 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/StackMiddleware/ReverseProxyMiddleware.php(47): Drupal\page_cache\StackMiddleware\PageCache->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#8 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/StackMiddleware/NegotiationMiddleware.php(52): Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#9 /var/www/crm.agbu.org/vendor/stack/builder/src/Stack/StackedHttpKernel.php(23): Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#10 /var/www/crm.agbu.org/web/core/lib/Drupal/Core/DrupalKernel.php(693): Stack\StackedHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#11 /var/www/crm.agbu.org/web/index.php(19): Drupal\Core\DrupalKernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#12 {main}.
```
The first issue is a warning that the session is too large, and so it destroys the session (which causes a logout). The second issue is of "error" severity saying the session can't be found.
Here's someone else having a similar issue in Symfony (not Drupal8): https://github.com/symfony/symfony/issues/26623tottentotten