CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2019-11-04T04:08:32Zhttps://lab.civicrm.org/dev/core/-/issues/449Bug: log_civicrm_group table grows insanely quickly2019-11-04T04:08:32ZAgilewareBug: log_civicrm_group table grows insanely quicklyBest demonstrated by example:
```
select count(id) `entries`, count(distinct id) `ids`, min(log_date) `since`, now() `now` from log_civicrm_group;
```
| entries | ids | since | now |
|---------|-----|-----...Best demonstrated by example:
```
select count(id) `entries`, count(distinct id) `ids`, min(log_date) `since`, now() `now` from log_civicrm_group;
```
| entries | ids | since | now |
|---------|-----|---------------------|---------------------|
| 31813 | 223 | 2018-10-16 14:14:51 | 2018-10-17 11:03:18 |
This is for a < 22hr period after I cleared this table to try and solve a data migration issue; before, this particular log table had grown to nearly 7 gigabytes (for the same 223 distinct group ids).
What appears to be happening is that every time the group_rebuild job is called, the cache_date is being NULLed and filled back in, creating two additional pointless log entries.
Per group id, I counted up to 250 log entries where only these fields had changed.
I think it should be possible to fix this by one of:
1. Disabling logging the the group_rebuild API call runs; or
2. excluding changes to just the cache_date and refresh_date fields from the civicrm_group triggers
Agileware ref CIVICRM-10015.12.0https://lab.civicrm.org/dev/core/-/issues/441Proposal - Add Timezone support for all Date/Time fields in CiviCRM2023-07-02T19:22:33Zjustinfreeman (Agileware)Proposal - Add Timezone support for all Date/Time fields in CiviCRMAdd Timezone support for all Date/Time fields in CiviCRM. Currently, CiviCRM does not have any timezone handling for date/time fields - timezone is not stored with the value.
The first change is to start storing a timezone with the date...Add Timezone support for all Date/Time fields in CiviCRM. Currently, CiviCRM does not have any timezone handling for date/time fields - timezone is not stored with the value.
The first change is to start storing a timezone with the date/time field. After this is implemented then other changes can follow which deal with the display of the date/time field in the local time zone or showing the timezone for the value.
**Proposed Change**
Have ability to set the timezone for a date/time field and store that value with the field.
Default timezone should set using the CMS timezone default and/or a new setting on the Localisation Settings page.
Agileware Ref: CIVICRM-995https://lab.civicrm.org/dev/core/-/issues/302End of life plans for 5.x php versions & planning for 7.0 EOL2020-01-30T02:01:26ZeileenEnd of life plans for 5.x php versions & planning for 7.0 EOLWe have had some discussion about EOL on php 5.x versions. The text below is a proposed blog post:
**End of life plans for 5.x php versions & planning for 7.0 EOL **
This blog serves as advance notice of our intention to stop supportin...We have had some discussion about EOL on php 5.x versions. The text below is a proposed blog post:
**End of life plans for 5.x php versions & planning for 7.0 EOL **
This blog serves as advance notice of our intention to stop supporting php versions 5.5, 5.6 and our ongoing evaluation of 7.0.
For php 5.5 we intend to end support in January 2019. This is already unsupported by php and we strongly recommend you upgrade off it as soon as possible. The release in February 2019 will be the first release that does not support php 5.5
For php 5.6 our TARGET is to end support in September 2019 (Oct release would support php 7.0+). 5.6 and 7.0 will unsupported by php from the end of this year. Usage of these versions is still pretty high amongst CiviCRM users http://stats.civicrm.org/?tab=sites so we will review this target in the first quarter of next year & extend it if we feel it will cause undue pain. Supporting 5.6 has downsides in that it restricts the external packages and versions of those packages we can use. As time goes on the risk of it leaving us without a secure option increases, as well as reducing opportunities to use more modern packages and to improve our code and product.
For php 7.0 our TARGET is the end of 2019. We will be reviewing this in March as well to see how reasonable that is. There is less to gain from dropping 7.0 than 5.6 so we will take that into account.
**What version of php should you be using?**
In general you should aim to be on php 7.1 or 7.2. For drupal 7 users you may find that there are extensions that you rely on that do not yet support php 7.1 - although this will be less and less likely over time. See https://www.drupal.org/docs/7/system-requirements/drupal-7-php-requirements#php_required for more information.
Note that as of writing there are only a small number of sites on 7.2 http://stats.civicrm.org/?tab=technology so you may wish to check php 7.2 adoption stats before choosing php 7.1 over php 7.2 in case there are any remaining issues.https://lab.civicrm.org/dev/core/-/issues/4842Allow bulk-enabling extensions through UI2023-12-08T13:17:56ZnoahAllow bulk-enabling extensions through UICurrent behaviour
----------------------------------------
On Administer > System Settings > Extensions, only one extension can be acted on at a time.
Proposed behaviour
----------------------------------------
It should be possible to...Current behaviour
----------------------------------------
On Administer > System Settings > Extensions, only one extension can be acted on at a time.
Proposed behaviour
----------------------------------------
It should be possible to select multiple extensions and perform the same action (Install, Disable, Uninstall) on all of them.https://lab.civicrm.org/dev/core/-/issues/4721Get rid of `civicrm_option_value.domain_id`2023-10-26T11:41:06ZcolemanwGet rid of `civicrm_option_value.domain_id`Background/Motivation
------
Most tables with a `domain_id` column have that field *required* for every record.
`civicrm_option_value` is an outlier; most option groups don't care about domains. In fact there are only two that do:
```p...Background/Motivation
------
Most tables with a `domain_id` column have that field *required* for every record.
`civicrm_option_value` is an outlier; most option groups don't care about domains. In fact there are only two that do:
```php
/**
* $_domainIDGroups array maintains the list of option groups for whom
* domainID is to be considered.
*
* FIXME: Hardcoded list = bad. It would be better to make this a column in the civicrm_option_group table
* @var array
*/
public static $_domainIDGroups = [
'from_email_address',
'grant_type',
];
```
The `civicrm_option_value.domain_id` column and that lovely accompanying hardcoded list were added [in this commit](https://github.com/civicrm/civicrm-svn/commit/406091dc969907753a56fd7fae73320b1b012e67) as part of [this issue](https://issues.civicrm.org/jira/browse/CRM-5546). The assumption at the time seemed to be that the list might grow in the future, but that's proven to not be the case. Instead, we have this extra column making the already overloaded `civicrm_option_value` table even more complex to deal with. It would have been better at the time to move those two option groups into their own tables. It may not be too late to do so.
Proposal
-----------
1. Create a new table `civicrm_grant_type` - manage that table from the `civigrant` extension.
2. Create a new table `civicrm_from_email_address`.
3. Migrate all options into the new tables. Update pseudoconstant metadata in the schema.
4. Add some legacy handling at a few key points like `civicrm_api()` & `CRM_Core_OptionGroup::values()` to fetch options from the new table, for backward compatibility.
5. Deprecate the `civicrm_option_value.domain_id` column and eventually drop it.https://lab.civicrm.org/dev/core/-/issues/4631CiviGrant labels are...not good2024-02-15T15:50:15ZJonGoldCiviGrant labels are...not goodThe CiviGrant fields are strange.
The "Amount Requested" field has this description: *Requested grant amount, in original currency (optional)*.
"Total Amount" has this description: *Requested grant amount, in default currency*.
If you...The CiviGrant fields are strange.
The "Amount Requested" field has this description: *Requested grant amount, in original currency (optional)*.
"Total Amount" has this description: *Requested grant amount, in default currency*.
If you're creating a FormBuilder form for grant requests, it makes sense that you'd want a "Requested Amount" field - but you actually need to use "Total Amount". This is unintuitive, and there's nothing in the field labels to suggest anything about original vs. default currency.
I don't think "Total Amount" makes any real sense here as a label - if anything, I'd expect that to be the amount granted (except that's the "Amount Granted" field).
**Proposal**
"Total Amount" gets relabeled as "Total Amount Requested".
"Amount Requested" gets relabeled as "Amount Requested in Original Currency".https://lab.civicrm.org/dev/core/-/issues/4251Track Contact `image_URL` files in the `civicrm_file` table2024-01-02T21:02:33ZcolemanwTrack Contact `image_URL` files in the `civicrm_file` table# History
**Note: CiviCRM is a large, old project with many contributors, which makes "big picture" perspectives cumbersome to gather. Often a contributor simply wants to fix a bug or add a small feature, not dive into decades-long hist...# History
**Note: CiviCRM is a large, old project with many contributors, which makes "big picture" perspectives cumbersome to gather. Often a contributor simply wants to fix a bug or add a small feature, not dive into decades-long history and contemplate massive refactoring. The story of the `civicrm_contact.image_URL` field is a microcosm of the complicated world of CiviCRM.**
- The `civicrm_contact.image_URL` field (added v1.1) predates the `civicrm_file` table (added v1.5), which may explain why it was originally designed as a simple textfield with no file-management. It is a simple varchar and can store the url to any image file on the web.
- Originally, the UI allowed contact image files to be uploaded to a publicly-readable directory on the webserver, and `image_URL` stored an absolute url to that file.
- In 2014, [security hardening](https://issues.civicrm.org/jira/browse/CRM-14499) led to the addition of an `.htaccess` rule which blocked the contents of that directory from public visibility.
- This accidentally broke contact images, leading to [a rushed fix](https://github.com/civicrm/civicrm-core/pull/3058) which created `CRM_Contact_Page_ImageFile` at `civicrm/contact/imagefile`. This path allows open access to all contact images, but not other files, by querying `civicrm_contact.image_URL` for a matching filename before outputting the contents. An upgrade script rewrote all local `image_URL` fields to point to this path, using an absolute URL. This solution works but is very slow on big databases due to the unindexed query.
- During this rushed fix, it doesn't appear that consideration was given to html escaping of `&` characters. The default behavior of `CRM_Utils_System::url` is to escape `&` to `&` which is (IMHO) a bad default and certainly a poor choice for storing url strings in the database, but it was the default and no one changed it, and then Tim inadvertently cemented it with 065034510d48b722844b2007f8016ea644bd0cbd so now in order to safely read the urls you must pass them through the aptly-named `CRM_Utils_String::unstupifyUrl()` function.
- In 2016 there was an [attempt to fix the absolute URL and performance issues](https://github.com/civicrm/civicrm-core/pull/9237) which updated all the `image_URL` fields to relative paths.
- But this would have broken Drupal Views and other tools that rely on the convenience of querying the field from the db and outputting the url directly (in the future, SearchKit would rely on this too).
- A [compromise solution was reached](https://github.com/civicrm/civicrm-core/pull/9241) which rewrites the url at runtime on Civi pages. A `image_URL` like `http://wp.demo/civicrm/contact/imagefile/?photo=abc.jpeg` would get rewritten to `http://wp.demo/civicrm/file?reset=1&filename=abc.jpeg&mime-type=image/jpeg`. For reasons not entirely clear, this goes through a different internal path (`civicrm/file` instead of `civicrm/contact/imagefile`).
- In 2019, thinking it was unused, [Eileen removed support](https://github.com/civicrm/civicrm-core/commit/2762985c11e01ede473d86a33af279bc341f9a46) for passing `filename=` into the `civicrm/file` path, since that endpoint is typically supplied with a file id from the `civicrm_file` table plus a security hash.
- This accidentally broke contact images, leading to [a rushed fix](https://github.com/civicrm/civicrm-core/pull/13663) which added back the ability to get files by name from the `civicrm/file` path (since contact images are not tracked in `civicrm_file` and don't have an `id`). With the benefit of the history laid out above, a better fix might have been to switch to using the `civicrm/contact/imagefile` path and keep the `civicrm/file` path secure.
- In 2021 [I proposed adding](https://lab.civicrm.org/dev/core/-/issues/2755) an `image_file_id` FK field to the `civicrm_contact` table to track uploaded files. This proposal was met with approval, but when I recently tried to implement it I realized that the `civicrm_file` table already has an FK to `civicrm_contact` and circular references are not allowed.
- In 2023 [an option group was added](https://github.com/civicrm/civicrm-core/pull/25904) for the previously unused column `civicrm_file.file_type_id`. One possible use for that field would be to designate a file type of `"contact_image"`.
# Current Situation
The `civicrm_contact.image_URL` field can still store any url string pointing to a file on or off the server. It could point to any image on the internet, and would work fine. But if it's a file uploaded via the Civi UI, it will be an absolute link pointing to the `civicrm/contact/imagefile` path with a `photo=filename` argument. If CiviCRM recognizes this pattern it will rewrite it on core pages to the other path at `civicrm/file`, otherwise it will leave it alone.
For the confused, yes contact images are accessible at two paths, and neither is a direct link to the file on disk:
| Path | `civicrm/contact/imagefile` | `civicrm/file` |
| ------ | ------ | ------ |
| Class | `CRM_Contact_Page_ImageFile` | `CRM_Core_Page_File` |
| Permission | _none_ | "access uploaded files" |
| Args | `photo` | `filename`, `mime-type` |
| Uses | Stored in `image_URL` field as absolute URL. Output by Views & SearchKit | `image_URL` rewritten to this path on Civi pages for logged-in users |
**This situation leads to the following quirks and problems**
1. The absolute url to `civicrm/contact/imagefile` works great in Views and SearchKit... as long as the site name never changes! Otherwise, absolute URLs are a pain.
2. The url is still stored with html-escaped `&` characters that must be unstupified.
3. Anyone can access a contact image via the 1st path if they know the filename, however, the 2nd has a permission check which means logged-in users without "access uploaded files" cannot see contact images even though anonymous users can!
4. The security hash usually required by `civicrm/file` can be circumvented if you know the filename and mime-type. But the risk is mitigated by that path requiring "access uploaded files" permission.
5. There is still no file-management of contact images. Deleting a contact does not delete their image file. Deleting or changing a contact image also doesn't delete the old one.
# Proposal for File Management
1. Stop using `civicrm/file` for all contact images and [restore the patch to remove support for `filename`](https://github.com/civicrm/civicrm-core/commit/2762985c11e01ede473d86a33af279bc341f9a46).
2. Include `cid` as an argument to `civicrm/contact/imagefile` (and update stored paths accordingly) to fix the unindexed query. Also add `is_deleted = 0` to the query.
3. Add an option_value `"contact_image"` to the option group for `civicrm_file.file_type_id`.
4. When uploading a new contact image, create a record in `civicrm_file` table, and designate it `file_type_id` = `"contact_image"`.
5. Also create a record in `civicrm_entity_file` for contact images.
6. Add a virtual APIv4 field for the contact entity `image_file_id` which would allow getting/setting the file id. When setting a new file id, regenerate the `image_URL` with a post hook.
# Thoughts on Absolute URLs
All of these changes would result in better file management, but still doesn't solve the absolute url issue. This is tricky to solve because Views and SearchKit still rely on being able to output the image url directly from a query. Here are a few ideas for that one:
1. Bite the bullet and update all `image_URL` fields pointing to a local file to use a relative URL. SearchKit will still work. Views and other SQL-based tools will still work unless embedded on a remote site. Random offsite images will be unaffected.
2. Keep `image_URL` absolute but add an APIv4 virtual field like `Contact.image` which calculates the url at runtime. This satisfies moree use-cases but at the expense of adding complexity to an already overcomplicated situation.https://lab.civicrm.org/dev/core/-/issues/3920Update backend footer2023-05-27T17:14:31ZlarsssandergreenUpdate backend footerThe footer shown to backend users seems like it could be improved a bit:
![image](/uploads/3c4361fd529cf3ec27872a3a10660f81/image.png)
I think that bottom line could just be:
[Support](https://civicrm.org/help) [Documentation](http...The footer shown to backend users seems like it could be improved a bit:
![image](/uploads/3c4361fd529cf3ec27872a3a10660f81/image.png)
I think that bottom line could just be:
[Support](https://civicrm.org/help) [Documentation](https://docs.civicrm.org/)
Download is covered in the version number, which is also a download link. Support is a much more useful link that leads to options: Stack Exchange, chat, Gitlab, etc. rather than just Gitlab, which is probably not the right first stop for someone with a problem, in most cases.
Alternatively, support and documentation are covered in the top menu. Maybe there is no need for a second line at all?
Minor detail: Add a space between the status box and the word CiviCRM.5.63.0https://lab.civicrm.org/dev/core/-/issues/3841Core (etc.) extensions should be categorized and filterable2022-10-11T10:11:26ZJonGoldCore (etc.) extensions should be categorized and filterableThe extensions page is getting harder to navigate, and isn't alphabetized by description. Finding an extension can take a while.
We should take a cue from Drupal and add `category` to the `info.xml`, then put extensions into accordions ...The extensions page is getting harder to navigate, and isn't alphabetized by description. Finding an extension can take a while.
We should take a cue from Drupal and add `category` to the `info.xml`, then put extensions into accordions by category.
Like Drupal, I don't think we need a set taxonomy, and even if only core extensions had this, it would be a good UX improvement.
I also realize that we're probably near the point where the Extensions page is built in Form Builder, but the first step is buy-in on the concept, implementation can wait a bit.JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/3648SMS limit is hardcoded at 460 but should be changeable somewhere2024-01-29T09:44:22ZjasonobrownSMS limit is hardcoded at 460 but should be changeable somewhereTwillio (and probably other sms providers) allows SMS messages up to 1600 characters, but Civi has the limit hardcoded at 460. I've verified that by simply going to CRM/SMS/Provider.php and changing MAX_SMS_CHAR from 460 to 1600 the sys...Twillio (and probably other sms providers) allows SMS messages up to 1600 characters, but Civi has the limit hardcoded at 460. I've verified that by simply going to CRM/SMS/Provider.php and changing MAX_SMS_CHAR from 460 to 1600 the system now accepts (and sends) messages up to 1600 characters. It seems logical that the character limit should able to be adjusted and stored somewhere rather than set as a static value in the code. We are currently using the larger limit, and would really like to see this implemented asap so that we can stop manually updating the code each time we update civi.https://lab.civicrm.org/dev/core/-/issues/3533Rewrite the standalone bootstrap protocol. Deprecate civicrm.settings.php.2024-02-09T05:03:20ZtottenRewrite the standalone bootstrap protocol. Deprecate civicrm.settings.php.## tldr
Rewrite `Bootstrap.php` to support CMS-first -- and make `civicrm.settings.php` unnecessary.
## Background: Problem
CiviCRM has many entry-points -- specific examples would be `bin/cli.php`, `extern/open.php`, `extern/url.php`...## tldr
Rewrite `Bootstrap.php` to support CMS-first -- and make `civicrm.settings.php` unnecessary.
## Background: Problem
CiviCRM has many entry-points -- specific examples would be `bin/cli.php`, `extern/open.php`, `extern/url.php`, `extern/rest.php`, or `extern/cxn.php`. More broadly, *all test suites and CLI processes* use a standalone bootstrap script. (These entry-points are distinct from the front-end entry-points typically associated with the UI.)
There are a number of technical details and happenstance we can go into, but there are two fundamental nubs in writing and executing a standalone script:
* *It doesn't know which CMS is active.* There's usually some *hard-coding* to the find the CMS.
* *Bootstrapping the CMS adds overhead.* Many admins have written-off this consideration as cost of doing business - but for someone, somewhere, at some point it mattered.
## Background: `civicrm.config.php` protocol
The original protocol for standalone bootstrap originated many years ago (circa 1.x or 2.x?). It's still used in several cases. It's key design elements:
* `civicrm.config.php` is *generated* at build-time to match the CMS. The tarball/zipball for each CMS includes a different build of this file. It's main purpose is to locate `civicrm.settings.php`.
* `civicrm.settings.php` includes a `define` for the `CIVICRM_UF` (among many others).
* `CRM_Utils_System::*` includes functions like `loadBootStrap()`.
* __Original protocol__: A standalone script would include `civicrm.config.php`, which would locate the `civicrm.settings.php` using a CMS-specific search. It then boots Civi by loading `civicrm.settings.php` and calling `CRM_Core_Config::singleton()`. Optionally, it also boots the CMS by calling `CRM_Utils_System_*::loadBootStrap()`.
The original boot protocol has some major consequences -- for example:
1. It requires creating an extra settings file (`civicrm.settings.php`) on every build.
2. Depending on how the page-load starts, the Civi settings file and CMS settings may load in different orders. It's not valid to infer Civi settings based on CMS settings (and vice-versa).
3. You need to figure the *absolute or relative* path of `civicrm.config.php`. This is *possible* in `civicrm-core` -- but it's basically impossible with any other publishable deliverable.
4. Reading/understanding the boot protocol requires having multiple builds of Civi (one for each CMS) -- *that's the only way to see each version of `civicrm.config.php`.
## Background: `cv` bootstrap protocol
A few years ago, I combined the various `civicrm.config.php` files with the aim of producing a universal boot script that solved `#3` and `#4`. This produced https://github.com/civicrm/cv/blob/master/src/Bootstrap.php which eventually became the heart of the `cv` command. This endeavor preserved a bunch of assumptions from the original protocol... but it made a key change: *dynamically discovering* the boot info.
The dynamic discovery works a bit like this:
* Do a file-system search (from PWD) to figure out the CMS.
* Do a file-system search (from CMS dir) to figure out where `civicrm.settings.php` is.
* Unless... you really want a faster bootstrap. Then you can set an environment variable with your boot info (and bypass the searches).
IMHO, dynamic CMS discovery (with optional env-var for optimization) is a pretty good way to make the code more robust to deploying in a variety of environments and file-structures.
There's just one problem: the *implementation* grew out of `civicrm.config.php` and kept some of the assumptions, so the implementation still shares issues `#1` and `#2`.
## Proposal: Provide a universal, CMS-first bootstrap script -- and knock over some dominoes
The basic idea is to write a CLI helper which bootstraps the CMS first -- and then bootstraps Civi. The logic is roughly:
```php
function boot($startDir) {
$parentDirectories = array($startDir, parent($startDir), grandparent($startDir), ...);
foreach ($parentDirectories as $dir)
if ($dir has Drupal) { bootDrupal(); bootCivi(); return; }
elseif ($dir has Joomla) { bootJoomla(); bootCivi(); return; }
elseif ($dir has WordPress) { bootWordPress(); bootCivi(); return; }
elseif ($dir has Backdrop) { bootBackdrop(); bootCivi(); return; }
}
}
```
This helper still provides standalone bootstrap, so we can still have all our standalone scripts/unit-tests/commands/etc. But it makes a fundamental change:
* **You can always rely on having the CMS bootstrapped first**.
* Therefore, you can detect things like `CIVICRM_BASEURL` using CMS API's -- because they're always available.
* Therefore, you don't need `civicrm.settings.php` to store those settings.
* Therefore, you don't need `install/index.php` to create `civicrm.settings.php`.
* Therefore, you don't need write-access to create `civicrm.settings.php` (in the typical case).
* Therefore, you don't need to migrate or preserve `civicrm.settings.php` (in the typical case).
*Note: Of course, the entire CiviCRM install-base currently uses `civicrm.settings.php`, and some of them use it to accomplish special ends (like custom multidomain/multisite/multitenant arrangements). Therefore, I wouldn't argue for complete removal of the file -- but, instead, it should be an optional/advanced thing.*
## General Tasks
* Write the new bootstrap script.
* Phase-in the new bootstrap script for use by `cv` and `civicrm-core`.
* Update `civicrm-core` to compute values for all those constants (`define('CIVICRM_FOO',...)`) at runtime using CMS functions.
* Update the installer to omit `civicrm.settings.php`
* Update docs to describe the advanced/optional use of `civicrm.settings.php`.https://lab.civicrm.org/dev/core/-/issues/3529Does CiviCRM work well with database replication and clustering?2022-06-11T14:44:13ZdavejDoes CiviCRM work well with database replication and clustering?There have been a few discussions on Mattermost in recent months in which people have described problems running Civi in replicated / cluster environments. In particular, Civi's use of temporary tables has caused issues. E.g.
10 Oct 20...There have been a few discussions on Mattermost in recent months in which people have described problems running Civi in replicated / cluster environments. In particular, Civi's use of temporary tables has caused issues. E.g.
10 Oct 2017 eileen:
> We have been having some replication lag issues & one metric we are investigating is the slaves (only) seem to show a large number of open temp tables on one metric. The odd thing about the replication lag is it seems to get worse the longer since a server restart (presumably just the mysql part needs to be restarted).
10 Oct 2017 jackrabbithanna:
> Ok we're dealing with Percona DB
>
> Database Error Code: Statement violates GTID consistency: CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed outside transactional context. These statements are also not allowed in a function or trigger because functions and triggers are also considered to be multi-statement transactions., 1787
>
> followed by a php error: Fatal error: Uncaught CRM_Core_Exception: [0: Transaction integrity error: Expected to find active frame thrown
>
> Per https://dev.mysql.com/doc/refman/5.7/en/replication-gtids-restrictions.html the multi-statements transactions are not allowed as they could potentially run using the same global transaction ID (GTID)
>
> this error happens when trying to create a Drupal user, with CiviCRM profile on the form
Further comments from Eileen:
> OK - I'll have a read of that - the CiviCRM usage of temp tables is pretty common & in particular ties in with the way mailing recipients are calculated & rolled back. However, I'm also reading concerns that replicated DBs don't like temp tables (although it's not blocked in the version we have)
>
> (we use mixed mode replication)
>
> I think it's kind of a big picture question isn't it? I had thought to move all temp table creations to a single function (so if we want to do something different we can do it in there) - the thing is a lot of them are used to calculate group_contact_cache & those will cause rollback to fail
> "A recommended practice when using statement-based or mixed-format replication is to designate a prefix for exclusive use in naming temporary tables that you do not want replicated, then employ a --replicate-wild-ignore-table option to match that prefix. For example, you might give all such tables names beginning with norep (such as norepmytable, norepyourtable, and so on), then use --replicate-wild-ignore-table=norep% to prevent them from being replicated." - we could at least do that for reports etc
> if you are using statement based replication or mixed replication then any temp tables used to build real tables have to be replicated
>
> or else the query will fail on the slave
> it's an improvement basically 'make civicrm support replication'
> it does work with replication but I think that's more because various people have just configured it to do so
>
> but I don't think there has ever been any official support for that or effort to understand the design implications
13 Oct eileen:
> ... our replication issue DOES seem to be related to the temp tables created for civimail at this stage. ... What we are seeing is the DROP statement is written to the binlog BEFORE the create statement is - somewhat in line with https://www.xaprb.com/blog/2007/01/20/how-to-make-mysql-replication-reliable/
>
> (not really clear from there but it says "As I mentioned, each transaction goes into the binlog in the order in which it commits, not in the order in which it is started, so transactions may execute in a different order on the replica. ")
>
> the binlog is parsed on the slaves & they temp tables mount up
>
> not 100% sure if this will solve all our woes - but not letting civicrm use temp tables to create recipient lists seems to be important
We ran a bunch of Civi sites with MySQL replication several years ago and had a lot of problems with it: replication would fall over periodically and then everything would grind to a near standstill as it was re-sync'd.
As a starting point, it would be useful to gather some data on how many people use replication / clustering with Civi, how frequently problems are encountered, what the most common problems are and whether they have been solved.https://lab.civicrm.org/dev/core/-/issues/3003Preserve previous tab when navigating to and from contact page2022-05-05T14:57:50ZBradley TaylorPreserve previous tab when navigating to and from contact pageThe contact page is structured around a series of tabs: Summary, Contributions, Membership etc. There are often times when you click through from a contact to a related entity (say a related contact from the relationships tab). It would ...The contact page is structured around a series of tabs: Summary, Contributions, Membership etc. There are often times when you click through from a contact to a related entity (say a related contact from the relationships tab). It would be nice if using the browser's back button returned you to the tab you were previously looking at rather than returning to the summary tab each time.
This is a bigger issue for users with the "Enable Popup Forms" option unticked, but all users are affected by this.
This shouldn't be a massive technical change, but I do think this is one of those little annoyances, where fixing should improve the overall usability of the CRM for those who need to use it on a daily basis.
**Implementation details**
It is already possible to link to a specific tab with the `selectedChild` query parameter. I propose that whenever the active tab is changed, the [replaceState](https://developer.mozilla.org/en-US/docs/Web/API/History/replaceState) API is used to amend the `selectedChild` parameter in the URL.5.49.0https://lab.civicrm.org/dev/core/-/issues/2755Proposal: add `civicrm_contact.image_file_id` column2023-04-20T00:24:59ZcolemanwProposal: add `civicrm_contact.image_file_id` columnProblem
-----------
Handling of contact images is nonstandard and difficult to work with.
Current Behavior
--------------
**Uploading a contact image will:**
1. Store the file in (by default) `civicrm/custom`.
2. Update the `civicrm_...Problem
-----------
Handling of contact images is nonstandard and difficult to work with.
Current Behavior
--------------
**Uploading a contact image will:**
1. Store the file in (by default) `civicrm/custom`.
2. Update the `civicrm_contact.image_URL` column with a link to an ajax callback for accessing the image.
3. Not create a record in the `civicrm_file` table.
4. Not create a record in the `civicrm_entity_file` table.
**Deleting a contact image in the UI will:**
1. Not delete the file.
2. Set `civicrm_contact.image_URL` to `null`.
Proposed New Behavior
-------------------
*Add a `civicrm_contact.image_file_id` column, FK to `civicrm_file.id`.*
**Uploading a contact image will:**
1. Store the file as before.
2. Update the `civicrm_contact.image_URL` column as before.
3. Create a record in the `civicrm_file` table.
4. Store the file id in the `civicrm_contact.image_file_id` column
5. *Undecided:* should an entry also be created in `civicrm_entity_file`? See #2760
**Deleting a contact image in the UI will:**
1. Delete the file if an id is present in `image_file_id` column.
2. Set `civicrm_contact.image_URL` and `civicrm_contact.image_file_id` to `null`.
See PR https://github.com/civicrm/civicrm-core/pull/21141https://lab.civicrm.org/dev/core/-/issues/2155Remove 'onlinePendingContribution' payment support from membership edit form2020-11-16T20:05:33ZeileenRemove 'onlinePendingContribution' payment support from membership edit formI want to proposed removing support for 'onlinePendingContribution' from the Membership Edit form.
**What is onlinePendingContribution support you ask?**
It's this box on the membership edit form
![Screen_Shot_2020-10-31_at_6.30.02_PM]...I want to proposed removing support for 'onlinePendingContribution' from the Membership Edit form.
**What is onlinePendingContribution support you ask?**
It's this box on the membership edit form
![Screen_Shot_2020-10-31_at_6.30.02_PM](/uploads/10211382f41b9d0fe92362f620e68d6e/Screen_Shot_2020-10-31_at_6.30.02_PM.png)
**Hey but I've never seen that box!**
Well you only see if if someone has created a pending pay later membership through an online form which you later complete through the membership edit form
**And it allows you to send a customised receipt?**
No, it says you can, you can't
**How else would someone complete that contribution?**
Well - they could do it the same way they would for a pending contribution created in any other way - use the link a moment further down the page
![Screen_Shot_2020-10-31_at_6.27.20_PM](/uploads/3e29866a816528a9265a26a852aa73de/Screen_Shot_2020-10-31_at_6.27.20_PM.png)
**And why do you want to remove it**
There is considerable code complexity needed to support it - about 85 lines of code. Oh, and it's broken in master
**But how do we know no-one is using it**
Well I tested on 5.21 & it's broken there too so my guess is is has broken for a long time....5.33.0https://lab.civicrm.org/dev/core/-/issues/1933Failed scheduled reminders should log message somewhere more useful2021-12-16T14:06:16ZAndy ClarkFailed scheduled reminders should log message somewhere more usefulOverview
----------------------------------------
A failed schedule reminder should log the error in a place that the end user can see why it failed.
Example use-case
----------------------------------------
A schedule reminder is set u...Overview
----------------------------------------
A failed schedule reminder should log the error in a place that the end user can see why it failed.
Example use-case
----------------------------------------
A schedule reminder is set up to send an SMS to a contact with a bad phone number. The reminder will fail, and a message will be added to table civicrm_action_log. The user who set up the reminder will not know whether it failed, or why, since they cannot view database tables.
Proposed behaviour
----------------------------------------
I propose a couple of alternatives:
1. The error message should be logged as an activity, either against the creator of the schedule reminder or the target. Then the user can check for this.
Or
2. The error message should be written to the system log, so at least someone with the 'Log Viewer' extension could see it.
Comments
----------------------------------------
My preference would be the first of the alternatives. There are a number of questions on SE about analysing failed Schedule Reminders - this would be a help with that.5.46.0https://lab.civicrm.org/dev/core/-/issues/1413Search the contacts modified in a period of time by a user2023-01-09T05:03:22ZdmunioSearch the contacts modified in a period of time by a userOverview
----------------------------------------
Possibility to search the contacts added or modified in a period of time by a particular user.
Example use-case
----------------------------------------
1. Click on **Search -> Advanced ...Overview
----------------------------------------
Possibility to search the contacts added or modified in a period of time by a particular user.
Example use-case
----------------------------------------
1. Click on **Search -> Advanced Search**.
2. In set **Change log**, enter time period and sort name of the user who added or modified the contacts sought.
3. Click on **Search button**.
Current behaviour
----------------------------------------
Currently it is only possible to search for contacts added or modified in a certain period. And separately, you can filter the contacts added and modified by a specific user. In the current version of civicrm (5.20) a note is displayed that clarifies this functionality in the **Modified By** field: **Note this field just filters on who made a change no matter when that change happened, It doesn't have any link to the modified date field.**
Proposed behaviour
----------------------------------------
Having the possibility to search for contacts added or modified in a period of time and by a specific user, constitutes a fundamental functionality in the monitoring of change logs.
For this improvement we have made the following pull request: https://github.com/civicrm/civicrm-core/pull/15913
Comments
----------------------------------------https://lab.civicrm.org/dev/core/-/issues/1156Do not delete associated contributions when membership is deleted2022-11-29T05:03:30ZStoobDo not delete associated contributions when membership is deletedBack in 2011 https://issues.civicrm.org/jira/browse/CRM-7188 it appears it was the intent to change the behavior of membership deletion to NOT delete associated contributions. This warning message was instituted as a work around but the...Back in 2011 https://issues.civicrm.org/jira/browse/CRM-7188 it appears it was the intent to change the behavior of membership deletion to NOT delete associated contributions. This warning message was instituted as a work around but the behavior change was apparently never implemented. I'd like to get some discussion and/or traction on finally implementing this change. Having discussed with @eileen @andrewhunt @JoeMurray @KarinG and BoT, I haven't found yet an opinion or a use case where the current behavior is desirable, but I don't want to assume that the community is unanimous in this reagard.
Input from others is welcome, I'm asking @colemanw @cividesk and any others who have run into this behavior for their thoughts and/or historical memory.
![delete-member](/uploads/2528146de59e3210bf7015eb726d036e/delete-member.jpg)https://lab.civicrm.org/dev/core/-/issues/285Scheduled Reminders for Membership not being sent2019-07-12T03:35:58ZMickCScheduled Reminders for Membership not being sentMembership reminders are not being sent - 2 probable causes found:
1. civicrm_action_log records for the same reminder 1 year ago
2. Status Override is checked
When either of the above conditions exist, the reminder is not sent.
Whe...Membership reminders are not being sent - 2 probable causes found:
1. civicrm_action_log records for the same reminder 1 year ago
2. Status Override is checked
When either of the above conditions exist, the reminder is not sent.
When neither of the above conditions exist, the reminder is sent.
When corrective action is taken to remove past civicrm_action_log records and remove any override, the reminder is sent.
Environment:
* CiviCRM 4.7.29
* Extension 'Transactional Email' is installed5.17.0https://lab.civicrm.org/dev/core/-/issues/5061New structure for getFields array in `.entityType.php`2024-03-25T20:55:29ZcolemanwNew structure for getFields array in `.entityType.php`Currently getFields reads from a `schema/xml` file with some wonky formatting. The plan in #4999 is to migrate those `.xml` files to `.entityType.php` files. So now we have a clean slate to decide how those files should be formatted.
No...Currently getFields reads from a `schema/xml` file with some wonky formatting. The plan in #4999 is to migrate those `.xml` files to `.entityType.php` files. So now we have a clean slate to decide how those files should be formatted.
Note: This will not affect the output of `DAO::fields()` or `APIv(3|4).getFields`, as we'll add a compatibility layer and tests to ensure those remain unchanged. But since we're replacing the underlying source of the data with something new, we get to clean that up.
Here are all the current keys in the schema/xml files, and what we plan to do with it in the new file. Many of the proposed changes would improve consistency with APIv4.getFields.
| Old key from `xml` | What to do | New key in `php` | Notes |
| ------ | ------ | ------ | ------ |
| `<name>` | ⚠️ index by | | Use name as array key, omit from array values |
| `<title>` | ✅ keep | `title` | |
| `<required>` | ✅ keep | `required` | |
| `<add>` | ✅ keep | `add` | |
| `<pseudoconstant>` | ✅ keep | `pseudoconstant` | snake_case all values in array |
| `<default>` | ✅ keep | `default` | |
| `<contactType>` | ✅ keep | `contact_type` | |
| `<deprecated>` | ✅ keep | `deprecated` | |
| `<readonly>` | ✅ keep | `readonly` | |
| `<serialize>` | ✅ keep | `serialize` | |
| `<localizable>` | ✅ keep | `localizable` | |
| `<usage>` | ✅ keep | `usage` | |
| `<component>` | ✅ keep | `component` | Ok, but it smells bad to have columns belonging to one component in another component's table :nose: (fixme for another day) |
| `<localize_context>` | ✅ keep | `localize_context` | Obscure property only used by 3 fields, but it's easier to keep it. |
| `<type>` | ⚠️ rename | `sql_type` | |
| `<crmType>` | ⚠️ rename | `data_type` | Historically this was inferred by `<type>` but a few fields explicitly declare `<crmType>` |
| `<uniqueName>` | ⚠️ deprecate | `unique_name` | We can't get rid of it yet but we can discourage it & stop indexing by it at every layer except `DAO::fields()` |
| `<uniqueTitle>` | ⚠️ deprecate | `unique_title` | Are title & attributes[label] not enough? |
| `<comment>` | ⚠️ rename | `description` | |
| `<html>` | ⚠️ rename/reorganize | `attributes` | `type` gets moved out (see `<html><type>` below), while `length` gets moved in as `maxlength` |
| `<html><type>` | ⚠️ move/rename | `input_type` | |
| `<length>` | ⚠️ rename | `attributes[maxlength]` | That seems better. |
| `<import>` | ⚠️ move | `usage[import]` | |
| `<export>` | ⚠️ move | `usage[export]` | |
| `<foreignKey>` | ⚠️ move+reformat | `entity_reference` | This tag is outside of `<fields>` in the xml but doesn't need to be. Let's move it into the getFields array. |
| `<dynamicForeignKey>` | ⚠️ move+reformat | `entity_reference` | Conceptually the same as above so can share the same name. |
| `<permission>` | ⚠️ reformat | `permission` | Convert `<or>` tags to nested array format used by `CRM_Core_Permisison::check()`. |
| `<drop>` | ❌ remove | | `<drop>` designates a field has been dropped, but in that case let's just [remove it](https://github.com/civicrm/civicrm-core/pull/18244) from the array as nothing else was being done with it. |
| `<rule>` | ❌ remove | | Unused as of [PR#29611](https://github.com/civicrm/civicrm-core/pull/29611). |
| `<headerPattern>` | ❌ remove | | Unused as of [PR#29612](https://github.com/civicrm/civicrm-core/pull/29612). |
| `<dataPattern>` | ❌ remove | | Appears to be unused in `universe`. |
| `<fulltext/>` | ❌ remove | | Apparently unused. Ignored by GenCode when writing DAOs. |