CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-11-05T14:13:53Zhttps://lab.civicrm.org/dev/core/-/issues/2122Account for time zone on event registration pages2023-11-05T14:13:53ZwmortadaAccount for time zone on event registration pagesOverview
----------------------------------------
There doesn't appear to be a way to account for the time zone on an event registration page.
This is less of an issue for a local event, but is quite a problem for a global online event....Overview
----------------------------------------
There doesn't appear to be a way to account for the time zone on an event registration page.
This is less of an issue for a local event, but is quite a problem for a global online event.
At a minimum I would expect there to be an option to set the time zone for the event and for the time zone to display on the event registration page. (Ideally you'd want the page to show the time in the user's time zone.)
Current behaviour
----------------------------------------
There is no option to set the time zone for an event. (I presume that the system time zone is used but I'm not sure.)
![image](/uploads/6727ccb71e4b3129120a3275240c56cf/image.png)
The time zone isn't displayed on the event registration page.
![image](/uploads/a16d1c595ab9020e2d9e9c2d668e114c/image.png)
Proposed behaviour
----------------------------------------
There is an option to set the time zone when creating an event. This should default to the user's time zone.
The time zone is displayed on the event registration page.
Comments
----------------------------------------
I became aware of this when trying to register for this event: https://civicrm.org/civicrm/event/info?reset=1&id=1553
Probably relates to https://lab.civicrm.org/dev/core/-/issues/441
Agileware Ref: CIVICRM-404 (feature not found)https://lab.civicrm.org/dev/core/-/issues/1896Add "All Contacts" link to the Contacts menu on new installs.2023-04-10T05:03:15ZhomotechsualAdd "All Contacts" link to the Contacts menu on new installs.Overview
----------------------------------------
_Currently viewing a list of your contacts is a two or three step process - load a search page, then click search. We can do this with link to `civicrm/contact/search?force=true`. For a b...Overview
----------------------------------------
_Currently viewing a list of your contacts is a two or three step process - load a search page, then click search. We can do this with link to `civicrm/contact/search?force=true`. For a better out-of-the-box experience it would be good to have this as a default link to `All Contacts` in the `Contacts` menu._
Example use-case
----------------------------------------
1. Click on **Contacts -> All Contacts**.
1. A pre-populated list of all contacts is available without needing to create a custom Navigation Menu entry.
Comments
----------------------------------------
_Discussion on MatterMost: https://chat.civicrm.org/civicrm/pl/zb5e6g8neibotqbp3g34iz3esa. If the issue here gets traction I'm happy to raise the PR myself._https://lab.civicrm.org/dev/core/-/issues/1442Use "Active Membership" to describe a "current member" to avoid confusion wi...2022-10-25T21:30:07Zjustinfreeman (Agileware)Use "Active Membership" to describe a "current member" to avoid confusion with the "current" membership statusOverview
----------------------------------------
Use "**Active Membership**" to describe a "current member" to avoid confusion with the "current" membership status. Using the same word for two different meanings is confusing to users a...Overview
----------------------------------------
Use "**Active Membership**" to describe a "current member" to avoid confusion with the "current" membership status. Using the same word for two different meanings is confusing to users and problematic on the screens as the purpose is not explicit.
As being discussed here, https://civicrm.stackexchange.com/questions/33946/search-find-memberships-difference-between-membership-status-current-and-curr/33948
Example use-case
----------------------------------------
1. User wants to find all contacts with membership.
2. Goes to the Find Memberships screen.
3. Unable to decide if they select "membership status = current" or "Current Member? Yes"
4. User selects both options and is shown an **incorrect** list of contacts.
5. The selection has excluded "New" and "Grace" statuses.
Having "**Active Membership**" is a clearer choice in this case.
Current behaviour
----------------------------------------
Same terminology is being used to refer to the membership status and a grouping of multiple membership statuses.
Proposed behaviour
----------------------------------------
Use "**Active Membership**" to describe a "current member" to avoid confusion with the "current" membership status
Screenshots of current situation
----------------------------------------
![chrome_XmRm57E4cR](/uploads/ae20f559b5f222e87a6148b4a3936b76/chrome_XmRm57E4cR.png)
![chrome_8XPF55uH7x](/uploads/dafd5cbb0fa31067cd2b1e6d18ddc82f/chrome_8XPF55uH7x.png)
![chrome_6ieI5PVZOZ](/uploads/8222442650219bedebfeb0ca6eb39f4c/chrome_6ieI5PVZOZ.png)
Agileware Ref: CIVICRM-1388https://lab.civicrm.org/dev/core/-/issues/1231Administrator UX: Identify pending extension DB updates2022-12-12T05:03:29ZtottenAdministrator UX: Identify pending extension DB updatesAs described in MM by @mattwire:
> The existing extension upgrades prompt is next to useless because it doesn't tell you which extensions have database upgrades waiting so you have to just click and cross your fingers! If, instead it g...As described in MM by @mattwire:
> The existing extension upgrades prompt is next to useless because it doesn't tell you which extensions have database upgrades waiting so you have to just click and cross your fingers! If, instead it gave a list of extensions with database upgrades it would at least give the user something to "decide".
>
> I do see a use-case for not automatically running them as it gives you the opportunity to decide to rollback the extension instead of upgrade at that point. But that may not actually be useful in the real world (ie. outside of my developer world).
>
>So, either run them automatically and tell the user they were run. Or don't run them and tell the user what is actually in need of upgrade.https://lab.civicrm.org/dev/core/-/issues/732Parsing of addresses in the format of different countries2023-03-05T05:03:27ZjaapjansmaParsing of addresses in the format of different countriesCiviCRM has the ability to parse addresses into a street name, house number and street unit.
This parsing is based on I assume US address formats.
E.g. an address like _799 E DRAGRAM SUITE 5A_ is parsed into house number: 799, street n...CiviCRM has the ability to parse addresses into a street name, house number and street unit.
This parsing is based on I assume US address formats.
E.g. an address like _799 E DRAGRAM SUITE 5A_ is parsed into house number: 799, street name: E Dragram, and Street unit Suite 5a.
In the Netherlands and Belgium the adresses are formatted in a different way, first it is the street name, then the house number and then the street unit.
Some examples:
* _Grote straat 1a_, which should be parsed to Street name: Grote straat, house number 1, and street unit a
* _Grote straat 2-3_, which should be parsed to Street name: Grote straat, House number 2, and street unit 3
* _Plein 40-45 60-II_, which should be parsed to Street name: Plein 40-45, house number 60 and street unit II
**What I think CiviCRM should be capable of (either with an extension or in core):**
1. Parse addresses based on the country of the address
2. Format addresses based on the country of the address, meaning glue the individual address (Street name, house number, street unit) parts into one address
**How does it work currently?**
When I enable Address Parsing I can enther a full address into CiviCRM.
If I enter _Grote straat 1a_ it gets parsed into Address name: Grote straat 1a, House number: _empty_ and street unit _empty_
If I enter _Plein 40-45 60-II_ it gets parsed into Address name: Plein 40-45 60-II, House number: _empty_ and street unit _empty_
If I enter the individual parts, street name: Grote straat, house number: 1 and street unit a, it gets formatted to 1 Grote straat a
As you can see the formatting and parsing of Dutch and Belgium addresses does not work.
**Proposed solution**
I do think it will be good when the address parsing functionality could be made in such way that extensions could hook into it.
I am thinking of a hook which provides callback functions for address parsing based on the country ID of the address.
Example code
```php
$callbacks[1020] = array('CRM_Address_BAO_Address', 'parseBelgiumAddress');
$callbacks['default'] = array('CRM_Address_BAO_Address', 'parseAddress');
/**
* Alter the callbacks for address parsing
*
* @param array $callbacks
*/
hook_civicrm_alter_address_parser_callbacks(&$callbacks) {
// Add a custom address parser
// For example the dutch one:
$callbakcs[1152] = array('CRM_MyExtension_DutchAddressParser', 'parseDutchAddress');
}
```
This way CiviCRM core could provide a default address parser, and a set of address parser for certain countries and if one is missing an extension developer could write the missing parser.
What do you think of this? If everyone is happy I am happy to implement this.jaapjansmajaapjansmahttps://lab.civicrm.org/dev/core/-/issues/655Soft Credits Multiply GIft Amount in Contribution Detail Report2022-04-22T15:48:39ZguyiacSoft Credits Multiply GIft Amount in Contribution Detail ReportThe Amount column in the Contribution Detail report is multiplied by the number of soft credits there are for a given contribution. So a $1000 gift will show up as:
* $2000 if there are 2 contacts credited with soft credits for that con...The Amount column in the Contribution Detail report is multiplied by the number of soft credits there are for a given contribution. So a $1000 gift will show up as:
* $2000 if there are 2 contacts credited with soft credits for that contribution,
* $3000 if there are 3 contacts credited with soft credits for that contribution,
* $4000 if there are 4 contacts credited with soft credits for that contribution, etc.
It happened on a client's site using Civi 5.7.0 under WordPress, and I tested it on 5.11.alpha1 on wpmaster.https://lab.civicrm.org/dev/core/-/issues/533Update recaptcha to v32023-06-30T05:03:20ZMartinUpdate recaptcha to v3Google recaptcha v3 has now been officially released:
https://webmasters.googleblog.com/2018/10/introducing-recaptcha-v3-new-way-to.html
Obviously v2 is still working for the time being but it would be beneficial to update core to use v...Google recaptcha v3 has now been officially released:
https://webmasters.googleblog.com/2018/10/introducing-recaptcha-v3-new-way-to.html
Obviously v2 is still working for the time being but it would be beneficial to update core to use v3. For example one of our clients is currently seeing this issue (significant usability bug in iOS):
https://github.com/google/recaptcha/issues/130
Thanks!https://lab.civicrm.org/dev/core/-/issues/416Schedule job time shifting every day2022-08-30T13:42:45ZsamuelsovSchedule job time shifting every dayWhen scheduling a cron job daily, the starting hour is shifting a little bit every day.
It means that eventually, we'll get a day skipped from 23:59 to 00:01, 2 days later.
It can be a problem e.g. for membership schedule reminders wher...When scheduling a cron job daily, the starting hour is shifting a little bit every day.
It means that eventually, we'll get a day skipped from 23:59 to 00:01, 2 days later.
It can be a problem e.g. for membership schedule reminders where we check for all membership that correspond to the today's date. Some schedule reminders won't be sent.https://lab.civicrm.org/dev/core/-/issues/392Support MySQL 8.0 now that it is GA2021-04-01T01:52:03ZJoeMurraySupport MySQL 8.0 now that it is GA# Overview
MySQL 8.0 went General Availability on April 19, 2018. CiviCRM's aim is to support the latest GA for required infrastructure (MySQL, PHP) as soon as feasible.
In reviewing removed features that cause incompatibilities (htt...# Overview
MySQL 8.0 went General Availability on April 19, 2018. CiviCRM's aim is to support the latest GA for required infrastructure (MySQL, PHP) as soon as feasible.
In reviewing removed features that cause incompatibilities (https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html) I noticed a couple of issues that definitely require changes, one that might cause syntax errors that are easily fixed when found, and a fourth that we may want to change to improve our functionality and get current.
This is a meta issue for tracking what's needed for MySQL8. I have added a MySQL8 flag to lab for the moment to help in tracking. We can delete that flag after this issue is deemed complete.
# Related issue: Upgrade test infrastructure to enable testing against a version on the edge of being officially supported: https://lab.civicrm.org/dev/core/issues/1142
# Issue: Field Names now Reserved Words https://lab.civicrm.org/dev/core/issues/1143
# Testing issue: dependence on PHP version for MySQL authentication ~~https://lab.civicrm.org/dev/core/issues/1144~~
# Document dealing with changes to MySQL user password library https://lab.civicrm.org/dev/core/issues/1145
# Possible issue: Sort order on GROUP BY clauses
~~I don't think we have any code that tries to specify sort order on GROUP BY fields, but if so we will need to change to add an ORDER BY clause with the sort order. I think we should just try running tests on MySQL 8.0 and using it manually to determine locations in code with this issue, and play whack-a-mole on anything missed.~~
# Nice to have: Upgrade our text fields to support all utf8 characters https://lab.civicrm.org/dev/core/issues/339https://lab.civicrm.org/dev/core/-/issues/4899Modelling many to many relationships with Entity reference, SearchKit, FormBu...2024-01-17T18:18:18ZMichael McAndrewModelling many to many relationships with Entity reference, SearchKit, FormBuilder, ECK, etc.Creating this issue as a way to keep track of different bits of functionality that can be used together to model many to many relationships in CiviCRM as when you put this all together, I think it is quite a game changer for data modelli...Creating this issue as a way to keep track of different bits of functionality that can be used together to model many to many relationships in CiviCRM as when you put this all together, I think it is quite a game changer for data modelling in CiviCRM :grinning:
Might make sense to document this at some point soon, and it would be good to collect feedback on how people are finding this functionality, ideas for improvement, etc.
Also, if you have any budget that you would like to put towards this work: to improve it or build out more features, etc. please get in contact with @colemanw or me :heart:.
* Entity Reference fields - allows you to reference other entities in custom data fields effectively creating **one to many** relationships.
* Multivalue custom data sets - [now available to all entities](https://github.com/civicrm/civicrm-core/pull/27549) allowing you to model **many to many** relationships _with additional meta data_ about the relationship
* Searchkit support for [joining via EntityRef fields in multivalue custom data](https://github.com/civicrm/civicrm-core/pull/28721) - allows you to create searches that span many to many joins
* FormBuilder support - allowing for editing of many to many relationships in entities (could probably do with some improvement)
* [ECK](https://github.com/systopia/de.systopia.eck) which allows people to make arbitrary new entities that can be joined via these relationshipshttps://lab.civicrm.org/dev/core/-/issues/4693Admin UI: Maturation model2023-10-20T07:49:58ZtottenAdmin UI: Maturation model(**This issue is `type:exploration`. It is intended to discuss/explain/invite comment. It is not a singular action-item for implementation. The discussion may lead to other more concrete things. We can close the discussion when the appro...(**This issue is `type:exploration`. It is intended to discuss/explain/invite comment. It is not a singular action-item for implementation. The discussion may lead to other more concrete things. We can close the discussion when the approach is agreed; it doesn't need every possible follow-up task.**)
Context
-------
* Much of CiviCRM UI was originally built with one particular architecture. (Key concepts: `HTML_Quickform`, `Smarty`, `Selector`, `StateMachine`.) (Short-hand: "Civi-Quickform" or "CQF")
* We're in the middle of a multiyear project to convert large swaths of the CiviCRM UI to another architecture. (Key concepts: APIv4, Form Builder, SearchKit, Angular, Managed Entities.) (Short-hand: "SearchKit/FormBuilder" or "SKFB")
* The "Admin UI" extension (`civicrm_admin_ui`) is a bundle of SKFB screens (which replace CQF screens). The bundle currently provides ~20 screens.
Question
--------
As the screens in "Admin UI" mature, how do we roll them out to the user-base?
Opinions/Thoughts
--------
* The great thing about "Admin UI" extension is this: we can "move fast and break things" while keeping the ship "pointed in one direction" and also allowing "real-world testing".
* For site-builders, it's an opt-in that doesn't interfere with their existing screens. They can try new-screens and easily switch back to the familiar screens.
* For developers, it lowers the effort to get started -- you don't need perfection. You can do a first-pass in February -- it's a "good/decent" screen with a couple quirks or missing-features. You can take some time to get feedback or think-through the quirks. Then in May, you come back and make a better version.
* `civicrm_admin_ui` is accumulating a set of viable replacements.
* But they're not *all* equally viable.
* There are still many screens that don't have replacements -- screens which will need their own maturation time.
* There was some discussion about whether we're ready to enable `civicrm_admin_ui` by default. Some options are:
* __No, keep it disabled by default__ (until the entire UI has been re-done). But then a lot users don't get the benefit of perfectly good (*improved*) screens. And developers still have to maintain both the old+new variant of each.
* __Yes, enable it by default__ (while conversions are on-going). But then it puts more pressure for new screens to be perfect. You can't merge a replacement unless you've addressed all the features of the original.
Proposal
--------
Replacement-screens go through process of incubation/graduation, mediated by `civicrm_admin_ui`.
1. __Incubation__: When a new replacement is written in SKFB, put it in `civicrm_admin_ui`. Allow 3-9 months for additional development, real-world usage, etc.
2. __Graduation__: When the replacement is sufficiently mature, move it to a standard/universal folder. (Ex: A new CiviContribute SKFB screen will go to `ext/civi_contribute/`.)
3. __Cleanup__: Three months after graduation, delete the old screen.https://lab.civicrm.org/dev/core/-/issues/4571Proposal: Add possibilty to assign custom CSS classes to containers in Form B...2024-03-12T13:14:12ZTobias Voigttobias.voigt@civiservice.deProposal: Add possibilty to assign custom CSS classes to containers in Form BuilderOverview
----------------------------------------
This issue intends to achieve more control and flexibility in styling the frontend appearance of form builder forms. To achieve this we suggest to add the possibility of assigning custom ...Overview
----------------------------------------
This issue intends to achieve more control and flexibility in styling the frontend appearance of form builder forms. To achieve this we suggest to add the possibility of assigning custom CSS classes to containers in the configuration backend / GUI of Form Builder.
This is meant to be an addition to the [Form Builder Roadmap](https://lab.civicrm.org/dev/core/-/issues/3761).
Current behaviour
----------------------------------------
Currently it is possible to select a predefined CSS class from the 'style' dropdown menu ("Panel Pane"). When this style is selected, an additional class is added to the div container ("af-container-style-pane").
![Screenshot](/uploads/306fe60b466b20d838972ff125ebb988/Bildschirmfoto_vom_2023-09-13_16-48-18.png)
Proposed behaviour
----------------------------------------
We propose to give users the possibilty to add custom CSS classes to form builder containers in a new input (text) field to allow more control and flexibility in styling the frontend appearance of form builder forms.
![Mockup](/uploads/047b3bc36039cf5f138dc9342834e25c/mockup-custom-css-classes.png)
Comments
----------------------------------------
Some major parts of the relevant code seems to be in these files:
https://github.com/civicrm/civicrm-core/blob/master/ext/afform/admin/ang/afGuiEditor/afGuiMenuItemStyle.component.js
https://github.com/civicrm/civicrm-core/blob/master/ext/afform/admin/ang/afGuiEditor/afGuiMenuItemStyle.htmlhttps://lab.civicrm.org/dev/core/-/issues/4539Very slow query when sending mailings via CiviMail2023-09-12T05:08:13ZwmortadaVery slow query when sending mailings via CiviMailOverview
----------------------------------------
We have come across an issue where the database connection was timing out due to slow, long-running queries when the Send Scheduled Mailings job was running. We became aware of this when...Overview
----------------------------------------
We have come across an issue where the database connection was timing out due to slow, long-running queries when the Send Scheduled Mailings job was running. We became aware of this when we set a maximum execution time limit of 3 minutes for MySQL queries. Mailings for one of our clients were not being sent because the database connection was timing out.
As a workaround, we have increased the maximum execution time to 10 minutes and reduced the Mailer Batch Limit and Mailer Job Size to 1,000. However, we'd like to fix the underlying issue that the queries are running so slowly.
When sending a mailing CiviCRM runs a query of the form:
```sql
SELECT r.contact_id, r.email_id, r.phone_id
FROM civicrm_mailing_recipients r
INNER JOIN civicrm_contact c on
(c.id = r.contact_id
AND c.is_deleted = 0
AND c.is_deceased = 0
AND c.do_not_email = 0
AND c.is_opt_out = 0
)
INNER JOIN civicrm_email e ON (r.email_id = e.id AND e.on_hold = 0)
WHERE r.mailing_id = 3271
LIMIT 18000, 1000
```
There are no indices for `is_deceased`, `do_not_email` or `is_opt_out` so this query is very slow. (There is an index for `is_deleted`. There should be an index for `is_deceased` but this appears to be missing from this site.)
We have tested adding these indices in a development environment to see what difference it would make and this increase the speed of the query by a factor of one thousand:
- 20,000 rows with no indices = 80 seconds
- 20,000 rows with indices = 0.085 seconds
Reproduction steps
----------------------------------------
This is likely to only affect sites with a large number of contacts and large number of rows in the `civicrm_mailing_recipients` table (in this case 46,000 contacts and 9 million rows respectively).
1. Set a maximum execution time for MySQL of 3 minutes (say)
2. Create a mailing to a large group of contacts (>20,000)
3. Schedule the mailing to send
4. Send Scheduled Mailings job fails with error: "Finished execution of Send Scheduled Mailings with result: Failure, Error message: DB Error: unknown error"
Current behaviour
----------------------------------------
No indices ~~`is_deceased`~~, `do_not_email` or `is_opt_out` in `civicrm_contact` so queries run very slowly.
Expected behaviour
----------------------------------------
Indices for the above fields so that the query runs quickly.
Environment information
----------------------------------------
* __CiviCRM:__ _5.62_
* __PHP:__ _8.1__
* __CMS:__ _N/A_
* __Database:__ _MySQL 5.7.43_
Comments
----------------------------------------
There are other speed improvements in #4045 but these relate to the speed of the user interface rather than the speed of sending emails.
Perhaps this can also help to [reduce our collective CO2 emissions](https://civicrm.org/blog/systopia/beginning-green-how-can-we-make-civicrm-more-sustainable)?https://lab.civicrm.org/dev/core/-/issues/4516Rethinking 20 minutes QF session timeouts2023-10-05T15:27:38ZJonGoldRethinking 20 minutes QF session timeoutsThe 20 minute timeout has been a part of Civi for as long as I can remember. It's also caused lost donations, frustrated applicants, and so on. Reading an [article about why short session timeouts aren't helpful](https://www.sjoerdlang...The 20 minute timeout has been a part of Civi for as long as I can remember. It's also caused lost donations, frustrated applicants, and so on. Reading an [article about why short session timeouts aren't helpful](https://www.sjoerdlangkemper.nl/2023/08/16/session-timeout/) made me think about this for Civi.
* What is the original reason? Is it still valid?
* If we lengthen the timeout, do we need to make special arrangements around event registration when there's a maximum number of seats available?https://lab.civicrm.org/dev/core/-/issues/4370Group opt in and subscribe/unsubscribe in form builder2023-09-01T19:44:08ZMichael McAndrewGroup opt in and subscribe/unsubscribe in form builder# Group subscribe and unsubscribe in form builder
This issue is an attempt to add functionality to handle group subscribe and unsubscribe in form builder as per the roadmap https://lab.civicrm.org/dev/core/-/issues/3761.
## Scenarios
...# Group subscribe and unsubscribe in form builder
This issue is an attempt to add functionality to handle group subscribe and unsubscribe in form builder as per the roadmap https://lab.civicrm.org/dev/core/-/issues/3761.
## Scenarios
Here are a few scenarios that we'd like to cater for...
### Opt in only flow
This flow is useful to give people the option to opt in to a group as part of another flow, e.g. when I register for an event, I might want to opt into an events newsletter. Note: I don't typically want to opt *out* of such a newsletter in these situations, and an opt out is likey unexpected and undesirable.
Opt in flows are also useful to create newsletter sign ups that contain multiple groups/mailing lists that people can sign up for.
We render a checkbox and group name. When the checkbox is ticked, we add a contact to the group (this might be a no-op if they are already in the group). If the checkbox is not ticked, we do nothing (i.e. we do not unsubscribe anyone that is already subscribed to the group).
We render a separate component for each group that the contact wants to opt into (rather than a single component that contains all groups).
By default checkboxes are not ticked when the form is loaded. This aligns nicely for workflows (e.g. GDPR compliant workflows) where opting in should be a conscious decision. However, it should be possible to set the checkbox to be ticked by default if desired. This may be appropriate in certain situations, e.g. it would be reasonable to pre-check a 'Please send me members newsletter' checkbox in a membership form if you wanted most people to receive the members newsletter but also wanted to give the option to opt out.
## Subscribe and unsubscribe flow
This allows people to both add themselves and remove themselves from groups.
This is typically used in 'Manage my notifications/communication preferences' screens and similar. It allows people to see what groups they are currently subscribed to, and subscribe and unsubscribe from them as appropriate.
We render a separate UI element for each group that the contact wants to opt into. When the checkbox is ticked, we add a contact to the group (if they are not already in it). If the checkbox is not ticked, we remove them from the group (if they are already in it).
## Automatic subscribe / unsubscribe flow
Sometimes we might want to automatically sign someone up to a mailing list without presenting a checkbox to tick. For example...
![image](/uploads/e5774b8f0d319fa33cee79f471d7049b/image.png)
One solution in this situation would be to add a hidden and checked subscribe or opt in element. In the situation that submitting the form should unsubscribe you, adding an unchecked subscribe/unsubscribe component should do the trick.
## The data model and form processing
I'm not 100% sure what the mark up for this should look like. An email join looks like this:
```html
<fieldset af-fieldset="Individual1" class="af-container" af-title="Individual 1">
<afblock-name-individual></afblock-name-individual>
<div af-join="Email">
<af-field name="email" />
</div>
</fieldset>
```
But group subscriptions and un-subscriptions don't work quite the same way.
In terms of the API, I think that what we want to do can probably be achieved with something like the save api:
```php
$results = \Civi\Api4\GroupContact::save()
->addRecord([
'group_id' => 2,
'contact_id' => 203,
'status' => 'Added',
])
->setMatch([
'group_id',
'contact_id',
])
->execute();
```
But I think there are a few tweaks that we'd want to do, e.g. we ideally do not want to record a status=removed when submitting a form with an unsubscribe/subscribe if they were not in the group in the first place.
And I am not sure how the save api gets called in formbuilder.
In terms of what the html might look like, I am wondering if it makes sense to create some components that encapsulate some of the logic, so that it ends up looking like this.
```html
<fieldset af-fieldset="Individual1" class="af-container" af-title="Individual 1">
<afblock-name-individual></afblock-name-individual>
<af-group group_id="2" />
</fieldset>
```
It might make sense to add a mode property that could be set to control the display and behaviour, e.g. `<af-group group_id="2" mode='opt-in'/>`
- normal - opts in and out
- opt-in - only allows opting in
- auto-add - hidden and adds people on submission
- auto-remove - hidden and removes people on submission
And a default property for opt in workflows, e.g. <af-group group_id="2" mode='opt-in' default=true/>
The above component would be responsible for generating the checkbox and text and for generating appropriate values when the form is submitted.
I can't see a way of doing it with the existing components. I might be missing something conceptual but here is my attempt. It made me think that a new component would probably make things easier.
```html
<fieldset af-fieldset="Individual1" class="af-container" af-title="Individual 1">
<afblock-name-individual></afblock-name-individual>
<div af-join="GroupContact">
<af-field name="group_id" /> <?-- do we also need to join on group to get the name? -->
<af-field name="status" /> <?-- Would need to translate a checkbox to value?'Added':'Removed'? -->
</div>
</fieldset>
```
## Email confirmation
At the moment, double opt in is closely linked to subscribing to an email group. IIRC in the past we have used something like
```php
<?php
civicrm_api3('MailingEventSubscribe', 'create', ...);
```
But this has a few drawbacks, including having to confirm multiple email subscriptions for a single form. Since we are also working on https://lab.civicrm.org/dev/core/-/issues/4232, we think it probably makes sense to leverage that for any logged out forms which will require that an email needs to be confirmed before any form processing occurs.https://lab.civicrm.org/dev/core/-/issues/4192Improvement: Required? when creating Custom Field2023-03-23T08:14:19ZStoobImprovement: Required? when creating Custom FieldAfter many years in my position working with clients, and even explicit training on this issue, clients continue to flummox themselves by not reading the 'help text' and clicking **Required? [ ]** in a Custom Data field ... when that in ...After many years in my position working with clients, and even explicit training on this issue, clients continue to flummox themselves by not reading the 'help text' and clicking **Required? [ ]** in a Custom Data field ... when that in fact isn't their intent.
Proposal Pt 1: Expanding the caution to the label itself like so: **[ ] Always require a value?**
Proposal Pt 2: Clarifying the help text like so. **This forces a value in this field to create or edit this type of record. To make a field required on a specific form, use a Profile.**
In a comparison other software:
Salesforce:
`[ ] Always require a value in this field in order to save a record`
Network for Good:
`[ ] Required?` same as Civi but I like their help text a bit better than Civi's `If "Yes" this custom field will be required when manually adding/editing contacts.`https://lab.civicrm.org/dev/core/-/issues/4045Ideas for (funded) speed improvements in mailings2023-08-28T14:44:15ZAndrew WestIdeas for (funded) speed improvements in mailingsWe'd be interested in funding some speed improvements when building mailing recipients in CiviMail. They'd inherently feed through to Mosaico, I think.
We find the mailings screens pretty snappy apart from selecting recipients. There a...We'd be interested in funding some speed improvements when building mailing recipients in CiviMail. They'd inherently feed through to Mosaico, I think.
We find the mailings screens pretty snappy apart from selecting recipients. There are few bottlenecks that it'd be nice to get rid of:
**Improve the dropdown**
When you have lots of groups and mailings the dropdown doesn't work very well. It stalls, with multiple spinners appearing, or wipes out your text while typing, or says 'none found' for a bit before updating, or alternates between 'Searching' and 'Loading', and generally takes an achingly long time to find the groups in its own dropdown.
![image](/uploads/91dcd7a0a2c2e7d3230c4016d1d49db0/image.png)
Compare this to the smart group dropdown in Search Kit, which is blindingly fast.
Maybe this could be replaced with the new API4-based entityref? Possibly split into two (or even four?): Groups to include / Groups to Exclude / Mailings to include / Mailings to exclude?
**Improve generating the list of recipients**
A complicated selection of groups can easily take 30 seconds to update and save, for us. This is very frustrating when adding one smart group at a time in the dropdown. Possible fixes:
- Refactor CRM_Mailing_BAO_Mailing::getRecipients(). At the moment this function seems to create a lot of temp tables, and uses a bunch of heavy queries to compare them against each other. Again, Search Kit can do complicated includes/excludes way faster. So I figure this could be modernised.
- Possibly the slow part here is emptying/filling civicrm_mailing_recipients each time the criteria are changed. Maybe add a 'getRecipientCountPreview' function to get a count of recipients without actually populating the table? This would let people experiment in the mailing screen quickly. Then only generate the actual recipient list when the mailing is scheduled or saved as a draft?
Any other ideas? Or thoughts on whether the above makes sense?
We could fund maybe 15hrs on this atm. If anyone wants to join us that'd be great too :-)https://lab.civicrm.org/dev/core/-/issues/3878SearchKit: Add ability to search and filter by active periods2023-02-27T22:02:08ZfrancescbassasSearchKit: Add ability to search and filter by active periodsOverview
----------------------------------------
Some entities (Relationships, Memberships, Cases, Events, Recurring Contributions) may be defined by a start date and end date that indirectly define a period in which the entity is activ...Overview
----------------------------------------
Some entities (Relationships, Memberships, Cases, Events, Recurring Contributions) may be defined by a start date and end date that indirectly define a period in which the entity is active or not.
This is a proposal to enable the possibility to search for entities active during a specific period via SearchKit and also filter results through related SearchKit Afforms.
Example use-case
----------------------------------------
1. **Relationships:** Search for active volunteers on a specific year
1. **Cases:** Get a pivot report with active cases per week
1. **Events:** Count number of active events on last year
1. **Memberships:** List active members per month
1. **Recurring contributions:** Get a pivot report with the active recurring contributions per month
Current behaviour
----------------------------------------
Currently, search for active relationships it’s only available through Advanced Search and Relationships Report, but it’s not possible to do this search via SearchKit.
![advanced-search](/uploads/dbd57f4abd494fe4225494c7f2faa5ef/advanced-search.png)
![relationship-report](/uploads/1d37472120e4f2cde4b055a0c60b147b/relationship-report.png)
@ayduns, points in [this SE answer](https://civicrm.stackexchange.com/a/41871/104) a way to search for a specific active period.
![searchkit-approximation](/uploads/5d151234bd3093ab422ffd6b057d936c/searchkit-approximation.png)
This it’s good, but not enough. It’s very tricky to build this clause intuitively. And if you create an Afform with this SearchKit you can’t expose start dates and end dates to reproduce this clause.
Proposed behavior
----------------------------------------
Create an “Active period” search field selector available for entities with start and end dates: Relationships, Events, Memberships, Recurring contributions and Cases.
This would allow for example:
1. Click on **Search -> Search Kit**.
1. Search for **Relationships** or **Events**
1. Select **Active Period** in **where** clause
1. Choose options for filter **=**/**is Between** and **Pick Date**/**Date Range**...
Active period selector, like in the image below:
![active-period-field-selector](/uploads/f7bff145d568e35ad01ce87dd0f211f1/active-period-field-selector.png)
Comments
----------------------------------------
Would answer questions like:
[How to search active relationships (e.g) during past year?](https://civicrm.stackexchange.com/q/8315/104)
[Is it possible to get point-in-time membership data?](https://civicrm.stackexchange.com/q/2253/104) [duplicate](How do I determine how many active members there were on a date in the past? [duplicate])
[Create a report or graph of the number of campaigns active in any particular month](https://civicrm.stackexchange.com/q/34465/104)
Will cover the use of this extension [Historic Membership Data](https://github.com/fuzionnz/nz.co.fuzion.historicmembershipdata)https://lab.civicrm.org/dev/core/-/issues/3721Add Entity Reference custom field type (implementing EntityRef QuickForm elem...2023-07-28T15:04:53ZjensschuppeAdd Entity Reference custom field type (implementing EntityRef QuickForm element type)Overview
----------------------------------------
`EntityRef` ist a QuickForm element type being used to reference CiviCRM entities in several forms. But there is no custom field type implementing this, apart from the *ContactReference* ...Overview
----------------------------------------
`EntityRef` ist a QuickForm element type being used to reference CiviCRM entities in several forms. But there is no custom field type implementing this, apart from the *ContactReference* custom field type, which is (surprise!) restricted to referencing contacts.
Allowing users to model their "real world" use cases in CiviCRM using the UI without mis-using entity types for things they have not been intended for, just because you need some kind of relation between two things, would be a great improvement.
At this time, we are evaluating requirements for a current project where this might be needed, but wanted to discuss our thoughts with some Core people, especially @colemanw who already offered help with reviewing code, before we start, because this will have to be a PR to Core, probably with some refactoring involved.
Example use-case
----------------------------------------
With either Core or custom entities (such as multi-value custom field groups or [ECK entities](https://github.com/systopia/de.systopia.eck)), you might want to model references to different "things" in your business logic, not just contacts.
Imagine a *Project* entity that you would like to add references to contacts, but also to specific activities, contributions, etc. - without a *Project* having to be a contact type or a *Case*, or a *Campaign* because that might not be suitable because of reasons.
Current behaviour
----------------------------------------
Only contact entites can be referenced in custom fields.
Proposed behaviour
----------------------------------------
* Add a custom field type *Entity Reference* with configurable
* entity type
* label property (a property of the selected entity type that will be used for displaying the referenced entity when rendering the field value)
* entity filters (API parameters for the entity)
* Migrate the *Contact Reference* custom field type to be an *Entity Reference* field type with *Contact* as the referenceable entity type and the `display_name` as the label property
* Maybe allow custom field types be extendable, i.e. allow extensions to define additional custom field types
* Maybe add support for SearchKit (e.g. for defining filters or providing widget displays)
Technical Details
----------------------------------------
* *Contact Reference* fields currently store their API filters in the `filter` column of the `civicrm_custom_field` table. The entity itself might be just added there
* The label/title property might be stored in the `attributes` column, which is used for different stuff depending on the custom field type. Alternatively, a new column could be added to the `civicrm_custom_field` table (just as there are columns for other field types), although this is not that good of a design IMO
* Alternatively, all field-type-specific configuration could be migrated into a generic JSON-formatted `configuration` or `properties` column, at least for this new field type for now. There are many columns only needed for specific field types:
* `mask`
* `options_per_line`
* `text_length`
* `start_date_years`
* `end_date_years`
* `date_format`
* `time_format`
* `note_columns`
* `note_rows`
* `option_group_id`
* `filter`
I think adding a new custom field type that stores its configuration in existing columns would be a good first step. Any thoughts on that?
After storing information about a custom field in civicrm_custom_field, a table is created for each group or set of custom fields to store the value of each field for each instance of the set of custom fields. The name of this table is stored in custom_field_group.table_name. This table correlates each record of custom field values with the entity they extend using meta-data about the custom field set to identify the table extended (I think there is a translation of value in custom_field_group.extends to the table name of the entity extended) and the id of the instance extended.
It would be good to support regular CiviCRM entities that are implemented as tables as well as the higher order entities of Order and Payment that have id fields and APIs but not table implementations.
Funding
---------------------
Required: 40
Pledged:
- Third Sector Design - 8
- Systopia - 18
- Megaphone tech - 4 (depending on timeframe)
- JMA - 10
- Humanists UK - 55.60.0https://lab.civicrm.org/dev/core/-/issues/3096Marking historical contact data - add on hold, hold date, reset date columns ...2023-01-06T22:02:03ZandyburnsMarking historical contact data - add on hold, hold date, reset date columns to civicrm_address and civicrm_phone to match civicrm_emailCurrently only civicrm_email has a mechanism to mark a bad email. This data model should be mirrored over to civicrm_address and civicrm_phone. We currently use the lightweight "location type" approach where we have an Invalid location t...Currently only civicrm_email has a mechanism to mark a bad email. This data model should be mirrored over to civicrm_address and civicrm_phone. We currently use the lightweight "location type" approach where we have an Invalid location type to designate a bad email/phone/address. However, it is limiting as you can only have 1 address per location type for a contact.
This was previously brought up here but was not pursued: https://issues.civicrm.org/jira/browse/CRM-13777.
The rationale is storing older data on these 3 entities helps prevent the creation of duplicates when bringing external sources of data (that may be older as well) upon import and therefore need to remain in the respective db table and not only sent to a log table.
The limitation of 1 address per location type would not apply to on hold addresses, therefore creating a history of addresses.
Setting to on hold would automatically set the hold date.
On hold entities would be shown in a "Former Communications Data" tab much like this extension (https://civicrm.org/extensions/former-communication-data) to prevent cluttering the Contact Summary Screen by only querying the entities that are on hold (and conversely, the contact's summary screen would only show those not on hold) and showing them in descending On Hold date order. Allow filtering by email/phone/address.
On hold entities would not be able to used throughout CiviCRM. E.g. a on hold mobile phone would not show Outbound SMS action, an on hold address would be added as a criteria to this checkbox upon export:
![image](/uploads/d481ef9470febc1620b895af2fb1baff/image.png)
There is obviously overlap with what detailed logging does and this so I'd greatly appreciate anyone's thoughts on how to approach / refine this feature request. We are willing to fund a solution.
Ref:
- https://civicrm.stackexchange.com/questions/681/what-are-best-practices-for-bad-postal-addresses
- https://civicrm.stackexchange.com/questions/20584/need-historical-address-information