Thanks for the heads up @JoeMurray
The invoice date should be the accounting recognition date, which according to CiviCRM's accounting entries is the contribution received date (which we did some work to rename to contribution date here - #1403 - https://github.com/civicrm/civicrm-core/pull/26145) as the contribution received date was ambiguous is a number of situations.
I believe it was the contribution received date when we first implemented invoices and not sure why someone changed this.
So I think my vote is for either 4 or 5:
{contribution.receive_date|crmDate:"Full"}{* Code comment: - You can replace the domain.now token with this to show the current date instead with {domain.now|crmDate:"Full"} *}
Administrators can only choose either to enable or disable sending email notifications (including a copy of the activity) when an activity is assigned to a contact OR limit it to specific types of activity.
https://dmaster.demo.civicrm.org/civicrm/admin/setting/preferences/display?reset=1
We would like to allow administrators to specify which groups of users would receive an email so that certain users can be assigned an activity, but they would not receive CiviCRM's default activity assignee email.
Note, there is already some similar (but subtly different!) functionality on CiviCase for case activities:
The above restricts who can be selected to be assigned an activity, whereas we are specifying who would be sent a notification for an activity.
We propose adding a new functionality that allows administrators to specify which CiviCRM groups would receive an email notifications. This can be achieved by introducing a new dropdown field.
We would create 2 new settings for this under the setting for "Do not notify assignees for" on the Display Preferences page: https://dmaster.demo.civicrm.org/civicrm/admin/setting/preferences/display?reset=1 which states:
If the field is blank there will be no restriction. If a group is selected, emails will be restricted to only go to that group.
Emails will be suppressed in each case.
We are prepared to submit a pull request (PR) for this feature. If the concept is approved, we will promptly submit the core PR.
Thank you.
Hey @mattwire @ayduns Thanks for the feedback. Sounds like it's going to be better to create an extension. I thought core might be better as it's quite a small improvement that might be welcomed to avoid the risk of emailing the wrong people. We'll take this back and just see if we can make an extension and will link to it here once done. Lets close this ticket for now!
@JoeMurray Hi Joe! Would it be possible for your team to take a look at this one if you get the chance? Do let us know if you have any comments, thanks!
Hey @jaapjansma , you were very kind to look at the previous ticket we submitted. Do you think you could take a look here and see if this could get a concept approved also?
@jaapjansma we have a PR for core here: https://github.com/civicrm/civicrm-core/pull/26180 do you know how we get this reviewed?
Thanks @jaapjansma! Can we get the magic concept approved tag or do we not do that anymore? @olayiwola_compucorp should submit a PR shortly.
How it works currently:
Currently CiviCRM forces the double opt in email to an email address which is:
the words "do not reply" + the domain of the site from your default mail account configured here: https://dmaster.demo.civicrm.org/civicrm/admin/mailSettings?reset=1
What is the issue?
The problem with this is that the default mail account email domain (used for bounce handling), may not be the same domain that you want to use as the from address for your double opt in emails (or for any other email that you don't want a reply from).
Proposed solution
We already have a place to configure the available from addresses in the system. This is the option group here: https://dmaster.demo.civicrm.org/civicrm/admin/options/from_email_address?reset=1
As such I would suggest that:
Next steps
We're happy to submit a PR for this so if we can get this concept approved we will submit a core PR asap.
Thanks
I think part of the reason the start date is ignored is that you can end up in a logical paradox where if you put in a start date (lets say it's in the past); if we had an automated job running with the logic "start date is in the past and end date is in the future or null" the status would be set to current.
But then you might want to disable the relationship manually, but often you might not have a known end date. As such the next time the rule runs you have a scheduled job to refer to the start date checking if the start date is in the past with no end date of the relationship, and so the scheduled job would in that situation automatically set the relationship to active again, which might not be correct as you have told the system that the relationship is disabled. This is probably why start date is ignored.
(nb. Probably we have a similar issue for end dates; but it's less of a problem as you are less likely to put in an end date and then "enable" the relationship again afterwards).
As a solution to all this, perhaps a better data structure here would be to have something implicit to state that the relationship status is being overridden (similar to how membership status rules are managed). We could have more complete schedule jobs/rules for the start and end date of the relationship if we also allow the user to implicitly state they are overriding those rules by having a checkbox to say "I want to set this manually" and then allow them to set the status.
Obviously a significant schema change so not expecting anyone to take this on immediately, but I thought I'd just jot this down here for a record.
Hi @jaapjansma thanks for taking a read.
You've jumped the gun a little as it's all still WIP but I've been working with @totten and a few members of the Compuco team on this and one of the guiding principles would be that this would be a UX change only and none of the underlying API's, data structures or option values would change, and as such extensions, reports and integrations can continue to work as they do currently.
It would only be the user experience from an end user perspective in Civi that would be modified, and hopefully to make things far clearer and simpler (think big buttons rather than confusing array of status options).
I hope to have finalised this ticket in the next few weeks then we will be sharing it wider for comment and then organising a call to discuss.
Best
Background:
Contribution status’s are ambiguous in CiviCRM and can lead to confusion over the current state of an invoice or contribution and whether there are amounts owed or owing to the contributor. This leads to a wealth of problems about what the intention of the user really is in a number of scenarios when they change the contribution status.
Examples:
Example | Narrative | Video |
---|---|---|
Example 1a: Overpayment / pending refund | A contribution which is created for $100 donation, but pending. We then record an overpayment of $120. Hypothetically the contribution is overpaid by $20 and should be pending refund. This is not shown in CiviCRM. There is nothing showing the user that there is an overpayment of $20. | 1-Overpayment |
Example 1b: Change contribution status to refunded | We can further demonstrate this by taking the same contribution and then changing the status of the contribution to refunded. This raises the question of what this means? Does this reflect the users intention to: Cancel the original debt of $100 (i.e. to credit note the original amounts owed and clear the balance Or reflect the fact a refund was made for the payment of $120, but that they are still owed the original $100 Or both? CiviCRM has refunded the original value of the invoice, i.e. $100, even though the payment of $120 was received. As such we still owe $20 but this is not shown in CiviCRM. | 2-Change_status_to_refunded |
Example 2: A typo entered in the wrong contribution amount | Add a contribution for $100 and set as completed. Realised that actually the contribution was $120. System will automatically add the additional $20 payment. Note: If this was a change in the line items of an invoice, it would not be safe to assume we received the additional payment. The aim of the user might be to state the obligation to pay was $120. Also it probably wasn’t 2 separate payments and hence we should reverse out the first payment (-100) and then make a second payment for the full amount (+120) | Updating_a_contribution_amount_error |
Much of this stems from the contribution status field being both be the driver of changes to the contribution (i.e. by changing the status to completed or refunded) or changed itself (i.e. the result of changes to the position of the contribution) by recording payments or refunds.
As such this is a proposal to simplify this workflow so that the the contribution status is only the result of the status of the contribution, and can no longer be used to change the status of the contribution.
We would retain the ability to make changes to the contribution status via the API for backwards compatibility until extensions can adopt the new model.
Further ramblings (feel free to skip this part if you are not too interested in accounting principles)
Accounting has simple and sounds principles for dealing with all of this.
“Obligations” are managed by 2 documents: Invoices and Credit notes. An invoice would stipulate what is obligated. This can be reduced by creating a credit note, which would then reduce the obligated amount.
It’s worth noting that for sales tax (for example VAT) purposes invoices should be sequentially numbered and should not be edited once they are generated. Hence usually you would not just simply delete an invoice to reduce the obligated amount, you would create a credit note for some or all of the amount, apply it to the invoice and send that to the customer to show the reduce in their balance owed to you.
As such I think in CiviCRM we need to fully enforce the split of concepts of the “obligation” and the actual payments (or refunded) amounts. Currently we have started on the journey towards that where Contributions (and line items on them) represent the obligations in most cases, and payments (or financial transactions of type payment) are the payments.
The contribution status should be a reflection of the net of that position. Does the customer owe us money (= pending or partially paid*) or is the balance settled (= completed) or do we owe them money (= pending refund)?
Proposed changes:
We should recalculate the contribution status whenever a contribution is created or updated based on the following rules:
Ref | Calculation | Status |
---|---|---|
1 | If there are no payments or refunds | Pending (arguably there would be a better label for this but in order to avoid making actual changes to the contribution status options this would seem to be the best fit) |
2 | If the sum of the line items (including any tax line items) > (sum of payments received - sum of refunds paid out) | Partially paid |
3 | If the sum of the line items (including any tax line items) < (sum of payments received - sum of refunds paid out) | Pending refund |
4 | If the sum of the line items (including any tax line items) = (sum of payments received - sum of refunds paid out) | Completed |
Screen | How it looks currently | How it will work | Changes required |
---|---|---|---|
New contribution screen | We split the "obligation" with the "payment". As such we remove the contribution status field from this screen and instead allow people to decide whether they want to record the "payment" at the same time as recording the "obligation" by checking the "Record payment" checkbox. The contribution status would then be calculated using the logic above. i. | ||
View contribution screen (one line item) | No Changes | ||
View contribution screen (multiple line items) | |||
Edit contribution screen (one line item) | Status field becomes read only and we introduce a button to cancel the contribution. See below for new screen for cancelling a contribution | ||
Edit contribution screen (multiple line items) | |||
New membership screen | cell | ------ | |
View membership screen (one line item) | |||
View membership screen (multiple line items) | |||
Edit membership screen (one line item) | |||
Edit membership screen (multiple line items) | |||
New event participant screen | Change the checkbox "Record Payment" to "Record Contribution" OR have a toggle to allow the admin to decide whether they are recording a "Paid" or "Free ticket". We only need the financial type and contribution date if we are only recording the contribution (note in core#1403 we have agreed to change label of contribution received date to contribution date). The amounts / lines should come from above. | ||
View event participant screen (one line item) | |||
View event participant screen (multiple line items) | |||
Edit event participant screen (one line item) | |||
Edit event participant screen (multiple line items) |
@JoeMurray Sorry for slow reply. We are in general pushing more and more clients down this route and expect to have a few larger organisations using this in more detail this year. So would be good to get more eyes on this and see if we can make it more robust and easier to use.
Agree that showing a filter which says "contribution status" and a column which says "status" and having them being different fields is confusing.
I think this is relatively benign as long as we clearly label this. Would it be worth exposing all the financial transaction fields if we are doing this? I can see it being useful to be able to filter on other aspects of the financial transaction.
I'd stick with Financial Transaction Status. Let's say what it really is, I think if we try and say Payment Status, and given that some transactions do not involve payments it would be confusing.
If anything this field is more to explain the "Transition type", i.e. the reason the transaction was created as it mirrors the status of the contribution when it was created.
I'm a little on the fence about this. My reasoning being that this would (I assume) show the status of the contribution currently which isn't that relevant to the transaction. On the other hand, if I filtered by contribution status it would be helpful to get the visual feedback that the results are correct (nb. I can't think of a relevant use case for this as organisations shouldn't be selectively exporting by status of contribution - if they were I'd probably wonder if they were using it correctly!). Probably fine to add this column.
Basically:
If we agree that the objective of this screen is to allow accountants to export a list of the journal entries that CiviCRM has created in the background and that we equate civicrm_financial_transactions to journals then this seems like it would help move us a little in the correct direction.
Hi @magnolia61 , yes we are. We are planning to do quite a big phase of accounting improvements around Feb time including some iterations from this ticket also: financial#186 so if you can leave this with us and we'll get an update to you as soon as this is planned in.
Only took 2 years (and 5 days)
I'll see if we have capacity to look at this.
Before we do, can I get the magic concept approved tag @eileen ?
Ta!
@sluc23 so sorry for disappearing, thank you so much for working on this - will test shortly - just a manic 2 weeks!
See config here: https://nimb.ws/CLAS74
This would be super useful to support start + end dates + custom fields :)
Hey @sluc23 !
Loving report plus, thank you for all the work that has gone in.
I've been having a play and ran into an error on the membership matrix report:
$Fatal Error Details = Array ( [callback] => Array ( [0] => CRM_Core_Error [1] => handle ) [code] => -19 [message] => DB Error: no such field [mode] => 16 [debug_info] => SELECT SQL_CALC_FOUND_ROWS COUNT(*) as 'value', DATE_FORMAT(membership_civireport.membership_start_date, '%Y-%m') as 'row', membership_civireport.membership_type_id as 'col' FROM civicrm_contact contact_civireport INNER JOIN civicrm_membership membership_civireport ON contact_civireport.id = membership_civireport.contact_id WHERE ( contact_civireport.is_deceased = 0 ) AND ( contact_civireport.is_deleted = 0 ) GROUP BY col, row ORDER BY row, col LIMIT 0, 999999 [nativecode=1054 ** Unknown column 'membership_civireport.membership_start_date' in 'field list'] [type] => DB_Error [user_info] => SELECT SQL_CALC_FOUND_ROWS COUNT(*) as 'value', DATE_FORMAT(membership_civireport.membership_start_date, '%Y-%m') as 'row', membership_civireport.membership_type_id as 'col' FROM civicrm_contact contact_civireport INNER JOIN civicrm_membership membership_civireport ON contact_civireport.id = membership_civireport.contact_id WHERE ( contact_civireport.is_deceased = 0 ) AND ( contact_civireport.is_deleted = 0 ) GROUP BY col, row ORDER BY row, col LIMIT 0, 999999 [nativecode=1054 ** Unknown column 'membership_civireport.membership_start_date' in 'field list'] [to_string] => [db_error: message="DB Error: no such field" code=-19 mode=callback callback=CRM_Core_Error::handle prefix="" info="SELECT SQL_CALC_FOUND_ROWS COUNT(*) as 'value', DATE_FORMAT(membership_civireport.membership_start_date, '%Y-%m') as 'row', membership_civireport.membership_type_id as 'col' FROM civicrm_contact contact_civireport INNER JOIN civicrm_membership membership_civireport ON contact_civireport.id = membership_civireport.contact_id WHERE ( contact_civireport.is_deceased = 0 ) AND ( contact_civireport.is_deleted = 0 ) GROUP BY col, row ORDER BY row, col LIMIT 0, 999999 [nativecode=1054 ** Unknown column 'membership_civireport.membership_start_date' in 'field list']"] )
It's been installed on a clean Civi 5.33.2 with only some sample data. Not sure why it can't find the start date field... end date seems to work fine!
Hey both, thanks for reading. I know it's a long one, and certainly would welcome getting Dev list on it. I would suggest thought that we focus on the getting agreement on the "action" points so we don't drift off topic. Ideally action 1: Can we agree those ground rules so we know what we are aiming for?
I'd actually suggest once we get some eyes on this we have a call to go through it as this is complex stuff and I worry that we all drift a little in terms of our understanding.
Just to reply to a few points:
But on edit more statuses become available & these are inappropriate. I didn't expect them to but I traced it back to here https://github.com/civicrm/civicrm-core/pull/10718 and I think the issue is that the status that if you filter out 'Partially paid' then you can't edit a partially paid contribution.
If you check table 1, the column "Pending/Pay later" will show what is shown, and what a user can do. These are not quite the same thing due to validation rules on the form on save. As such in my "Actions" - action 2 suggests that we should hide anything they actually cannot do.
I would be OK with completely removing status from the edit form and instead ensuring the following actions are easily available from the form (most of those are already available from the action links beside the contribution - the cancel/fail being the exception)
- cancel contribution
- add payment
- add refund (which is the same form as add payment)
- fail contribution (yeah this feels a bit odd - but I think history has it being different to cancelling - it could be the same popup form as cancel)
If we remove the ability to change the status, we will need to make sure we can facilitate the scenarios in table 5 which at the moment probably is done in an incorrect way through marking the contribution as cancelled (or I guess failed). I wonder if instead we could just use the actual accounting terms: Void (where nothing has yet been received) / Write off (where we are underpaid) / Allocate overpayment (where we have an overpayment). If we agree on Actions 1-3 I can propose a UX for this.
Also note - as long as the Payment api (or add payment form) are used to add a payment the contribution status should be updated based on calculations.
I see, but perhaps this doesn't cover all the scenarios? For example if you change the "selections" for an event you can end up with the wrong contribution status (and of course there is the line item editor).
Note that the line item editor is not in core so we either have to bring a variant of that into core or support changing amounts as we currently do in core.
If we agree to the business logic in table 3, I think we should plan to bring the line item editor into core, but test it and ensure both it, and the "edit selections" and adding payments / making refunds all together aligns to the proposed business logic for the contribution status field.
Another thought - it feels like you want a status like 'finalized' added - which would mean 'the payments don't match the amount requested but we are administratively 'done' on this' - ie they have underpaid & we are not gonna chase them or they have overpaid & we are not gonna refund them
Actually... I really want to avoid any more status's
JMA is supporting the Line Item Editor as though it is a core extension, and we'd encourage shipping it with core. If more automated tests are in order we're happy to add them.
I think the only problem I see with the LIE is the interaction with the status field, and this is more CiviCRM doing things wrong. As I said above, if we can get agreement on table 3 then we can test all of this together and make it work in all scenarios robustly.
Although I am not confident in the correct terminology, I would encourage there to be two buttons for refunds, one recording a refund made outside of CiviCRM, and another to cause CiviCRM to make a refund.
Partially agree. We currently have a "record payment" and "record credit card payment" and the same would be true for a "refund payment" that we should have 2 different buttons.
There is a complication to this however that most credit card payment processors require the ID of the original payment in order to make a refund (as they will only allow you to refund up to the value of the amount paid to you). As such the "record credit card refund" might need to be a "refund" button alongside the original payment financial transaction instead. It should of course make a new record for the financial transaction payment outgoing in Civi however, but from a UX perspective that would make more sense in case there are multiple payments against the contribution. I'll share some wireframes on this shortly as this is how we will be implementing this.
The 'Finalized' notion of a partially paid or unpaid contribution is tantamount to the accounting action of marking the debt as uncollectible or bad debt, and reducing the accounts receivable by that amount.
Yes, this is the bit that doesn't work in CiviCRM currently. Marking a partially paid contribution as cancelled or completed (Table 2 scenarios 3f/4e/4f) causes a bit of havoc with your financial entries.
In the context of a donation which is not an obligation to pay, I believe the accounting approach can be different, just reducing the revenue amount and the accounts receivable amount.
I think if we have the line item editor you can simply change the line item for the line expected, so we wouldn't need a different workflow here. i.e. I originally had an order for £120 and was paid only £100, and I'm now no longer expecting the additional £20, I can either do a write off or change the original order lines so I only expect £100. Either would be reasonable but the second way would be more correct. (Although, I might argue that you maybe shouldn't be making a pending contribution if there is no obligation to pay!).
@eileen I've passed this onto the team to look into.