Stripe Authorises payment before form validation causing problems

I think this is a problem; Matt and I have discussed it previously.

But it's not super easy to fix within the confines of the way CiviCRM does its forms. In fact it's pretty much impossible.

I've fixed this with a different flow on custom forms (validate non stripe form data first before confirming funds), but I don't know how to work this into Civi's flow.

I think the issue should stand here, although it's really blocked by a bug/limitation on Civi's forms implementation.

@artfulrobot Thanks for explaining, this actually causing a lot of trouble. Specially when a large amount is charged/uncaptured but not captured, its taken from customer's bank and they may not have enough fund to submit another payment. I am saying this from experience, we had numerous transactions was uncaptured and customer didnt have enough fund to submit another transaction until next few days when that money would have been refunded.

Thats a real issue and for long term its gonna affect us financially. Isn't there any fix at all?

Perhaps you should raise an issue in Core on contribution forms; I don't think it's something that can be handled in an extension.

Is this a duplicate of #112 (closed)?

added Needs funding confirmed labels

We need to raise £500 to resolve this issue - Help fund a fix now!

This is a funding campaign by MJW Consulting who are the maintainers of Stripe and will be developing the fix for this issue.

changed the description

@JonGold @artfulrobot This will need to be a CiviCRM core fix. It's going wrong down at the core form level.

@tapash I think your client offered some funding towards this? Funding campaign now available - see link in this PR!

mentioned in issue #112 (closed)

changed the description

@mattwire agree: this is a core patch.

It would be really great to document the flow as it could get pretty complex and make it hard for future devs to reason about.

added Funding Campaign Active label

changed the description

mentioned in commit 825c2629

mentioned in commit 1dd0d23c

There is a workaround in 6.3.1: https://lab.civicrm.org/extensions/stripe/tree/6.3.1 - please test and feedback so I can officially release.

@artfulrobot This is the workaround: 1dd0d23c and this is probably the core fix: https://github.com/mattwire/civicrm-core/commit/cbaa401edc27776ab07905508c611e708d6fe0e9 - basically the "required" flag is getting dropped in CRM_Core_Payment_Form which means the required fields don't get the class=required and jquery validate doesn't check them!

added Waiting feedback label

Well done! Interesting. Do you think there will be side effects, e.g. on a form that switches between 2+ pay processors, the other of which needs/expects those as not required?

If this is a fix, it's much lighter and less involved than I imagined. Is there any other Civi PHP logic that can cause a fail after the funds have been confirmed?

@artfulrobot It's certainly not perfect. The jquery runs each time the billingblock is loaded and specifically looks for the span.crm-marker element which means the field is required - so it will only fix the issue for stripe because the code is loaded via our stripe js. But it won't break the others any further.

Also, jquery validate is only checking if fields are required or not. Which in most cases is all that is required, however, if there is more complex formrule logic then it will not be picked up until submission - that's the more complex bit and probably needs some way of auto-converting formRule into a jquery equivalent (you can add validate functions I think). I intend to progress the core fix above so it's fixed for all processors but the additional formrule logic is probably out of scope!

@mattwire Couple of payment submitted in last hour after upgrading to 6.3.1 And uncaptured issue still persists. At least happened for 1 transaction. And another transaction didn’t have “updated” entry (not sure if that makes any difference though as payment was successful). Please see screenshot attached

Here’s few more

@tapash Are you able to get in touch with any of the users who had the uncaptured payments to find out what happened? The fix in 6.3.1 means that empty required fields will not cause an uncaptured payment to be created, but there could still be other reasons.

@mattwire One user said, " I was using my iPhone and I did not see check box for the Captcha, but when I tried to go to the next page, it said I needed to complete the Captcha! "

Could this be fixed with some validation check?

As a user of the web, I frequently come up against captcha issues like this: you can't see/use them but then your form submission fails.

I've had this with firefox which tends to have privacy settings (and I use some privacy extensions) that disallow evil, such as google's captcha.

@artfulrobot Ever done anything with the captcha stuff - do you think we can detect whether there is meant to be on on the form that needs to pass?

@mattwire no I have avoided captcha for years TBH; I hate it so much as a user ("find your own bloody traffic lights!") that I always try to talk clients out of it!

I prefer my own solutions, both invisible, both v effective.

send the data, send back a signature (hash of data + some unique salt) and expect that all to be sent again, check the hash. While this is easy to hack programmatically, most spammers are looking for easy ways and a double programmed submission costs them more. The other thing I've done is:
in the background request a token from the server. That token has a future validity date which is 10-15s in the future (this assumes it's going to take at least that long for the user to submit the form), and an end date, which is an hour or such. This means it's impossible to load the form and submit the form quickly. While it does leave tokens that are valid for an hour, it means the spammer has to do a lot of co-ordination to make that work.

@mattwire @artfulrobot I have turned off ReCaptcha for the entire site. But uncaptured payment still persistent. I have tried to contact the users , will update if I get any reply.

Good to know it's not that at least! (though I realise it's cold comfort to you)

Hi, I have tested it and purposely left out my surname and clicked continue, the page came back with an error below Please correct the following errors in the form fields below:

Billing Last Name is a required field.

however my card was then charged but uncaptured in Stripe.

@tapash @alexd I can't reproduce - do you have links to pages that are doing this? And have you cleared caches etc. to make sure you're loading the latest version of the extension resources?

@mattwire I have cleared cache many times before this happened.

Ok, I need to know why it is working on most sites and not yours? Do you have a public URL or one you can email to me?

@mattwire today we had a case where the Giftaid option was not selected, and the payment was uncaptured. Hope this might help to resolve the problem. That’s for a contribution page issue. Not sure what can cause a payment to be uncaptured on an event registration though, because we do not have gift aid on event reg pages.

changed milestone to %6.3

mentioned in commit afdd88a2

@tapash @alexd Please test the 6.3.2 branch for a potential fix: https://lab.civicrm.org/extensions/stripe/tree/6.3.2 (thanks Tapash for finding a way to reproduce - the branch is already on your test site ).

hi @mattwire

It flags up the required field without authorising the payment until the next page. So it is working for us now.

Thanks Matt.

Hi @mattwire

I spoke too soon, currently it is flagging up the nationality field as required despite it being filled in and won’t allow the purchaser to continue.

Hi @alexd please upgrade again (same 6.3.2 branch). I made a few more changes yesterday

Thank you for the swift response @mattwire . I’ll have a go at upgrading it again but have not done it myself before and if not will have to wait for our website maintainers on Monday to do it. Will let you know the results once it’s done.

Hi @mattwire ,

Just a quick update to let you know that that issue with the nationality required field has been resolved. What I have noticed is that after filling in the form, our site takes you to the next page where it asks to confirm your details. When this happens an uncaptured charge is sent to Stripe. IF the customer then clicks continue, it will confirm the registration and civi sends out a ticket and Stripe then captures the payment. Assuming if the customer, goes back to amend any details on the first page, then another uncaptured charge is sent again.

On contribution pages, when an email is entered without a proper format, ie without ‘@‘ or ‘.' On submit the page reloads with “incorrect email” error. Payment was uncaptured.

I was also able to get an input from a donor,

"My donation failed the first time but the message that flashed on the screen said that I was timed out of my session. I therefore had to attempt a second time when I was successful. But money was taken both times.

I have had difficulty in the past when making a contribution on my mobile device. This is because for some reason the card number, the 3 digit security number and the post code don’t fit in the same line when one starts to fill in this part. “

mentioned in commit 976a6845

@tapash The email validation should be fixed in aac41d56 which is in the 6.4 branch

@mattwire Thanks for this. However the email field validation still not working. For example, I wrote abcdefgh.com in email field, and I was able to proceed although there’s no ‘@‘ present the field. It came back saying there’s an error after the page reloaded and as a result an uncaptured payment was created.

I think I’ve found another case where the JS isn’t stopping the form submission for a required field.

We have a radio field in a profile I use on our membership and donation forms and it’s set in the profile as required. It has no default value (basically I want the user to click it to say they accept our terms).

If the user doesn’t make a selection the form is submitted to the server which responds by flagging the required field but taking an uncaptured payment in Stripe.

Whoops looks like you've already got this one in 976a6845

Hi @mattwire thanks for your work On this issue. But seems uncaptured payment still being created. I had more than 3 yesterday. There’s still something that’s causing the issue.

changed the description

@mattwire I was able to spot one of the cause of "Uncaptured" payment by following this steps..

fill in info on a contribution page
Click "Confirm Contribution"
Click "Go Back" to correct some info on the page
Click "Confirm Contribution" again on the initial page

and you would find 1/2 uncaptured payment. Hope this helps.

mentioned in issue #192 (closed)

mentioned in commit dac094cc

Thanks for working through this tedious bug!

I'm working with the latest 6.4.1 beta code and found that I can't submit a contribution form if I have the option to contribute on nehalf of an organization. After some debugging, I found that Stripe is marking the on behalf of state / province field as required, even if I don't select the option to donate on behalf of an organization.

This is particularliy confusing since I get the error: Please fill in all fields. But, since the state field is not even visible, I can't tell what field I'm supposed to fill in.

If I disable the option to donate on behalf of an organization it works as expected.

I'm still working through this one. It seems to be reporting that the State/Province field in the on behalf of profile is required. If I configure the site with a default state/province, then I don't have this problem (since a value is entered). But if no default state/province is configured for the site, then this field is emtpy and an error is triggered. I'm trying to understand why that state/province field would be affected but none of the others. In the on behalf of profile, all the fields are marked as required, not just the state/province one.

I haven't dug into this, but given that I've experienced idiosyncrasies with that field in the past - state/province is the only field whose values are loaded via AJAX after page load. If I had to guess why the others don't trigger this, it's because a JS event fires on this field after the page is rendered, but not the other fields.

assigned to @jamie

@JonGold @jamie There are two outstanding core PRs relating to this - the first one https://github.com/civicrm/civicrm-core/pull/16495 is being shipped by the stripe extension as an override.

The second https://github.com/civicrm/civicrm-core/pull/16488 builds on that and fixes the root cause of many of these validation issues. We also do some stuff in stripe javascript to work around this. See setBillingFieldsRequiredForJqueryValidate() here: https://lab.civicrm.org/extensions/stripe/-/blob/6.4.1/js/civicrm_stripe.js#L758

Hi all - thanks @JonGold and @mattwire for your thoughts. I found that if I unset the default country, then I get an error on the country. I suspect the country also does some special ajax business too.

@mattwire - how is the first PR shipeed as an override? I looked for templates/CRM/common/l10n.js.tpl and Common.js but don't see it in the extension. Maybe you are overriding a different way?

I've been focusing my debugging on the setBillingFieldsRequiredForJqueryValidate method - if I return rather then execute that function, I don't have this problem.

@mattwire - If I port both of those PRs to core - does that make setBillingFieldsRequiredForJqueryValidate unnecessary? I'm wondering if I should just apply your core fixes and remove that function so I can use the latest version (we really need to fix for sending a 200 response on contributions not found!).

@jamie Yes it should do. This is the override for 16495: https://lab.civicrm.org/extensions/stripe/-/blob/master/js/civicrm_stripe.js#L690-692

If you apply patch to core you don't need 16495 because it is already in 16488

I'll try to have Seamus review the two core PRs:

There are two outstanding core PRs relating to this - the first one https://github.com/civicrm/civicrm-core/pull/16495 is being shipped by the stripe extension as an override. The second https://github.com/civicrm/civicrm-core/pull/16488 builds on that and fixes the root cause of many of these validation issues. We also do some stuff in stripe javascript to work around this.

Getting the core PRs backported to 5.21 was a bit much. So I returned to the overrides in the stripe extension and finally narrowed it down to:

validator.settings.ignore = '.select2-offscreen, [readonly], :hidden:not(.crm-select2)';

By default jquery validation ignores hidden fields. So, if the "On behalf of" checkbox is not checked, the On behalf of fields are hiddnen, and jquery will properly ignore them. All is good.

Except... select2 seems to screw things up since it hides the actual select field and displays a friendlier field to the user instead. If we allow jquery to ignore hidden fields, it won't validate select2 fields, even if they are visible and required.

So, the code above cleverly tells jquery validator to ignore hidden fields as long as they are not .crm-select2 fields.

And here comes the problem: that means if an entire set of fields is hidden, the select2 fields will be required.

I'm a bit flummoxed by this one - any suggestions??

It seems this problem will affect the core PR as well.

@jamie Please see ff44521b (and https://github.com/civicrm/civicrm-core/pull/17672) now that 16488 is merged to core.

You don't need the core patches with the above commit for stripe (but it will work ok whether core has the patches or not (ie. will work before/after CiviCRM 5.28).

Thanks @mattwire! This looks great.

changed milestone to %6.4

I am still having issue with uncaptured payment, though not as much as it used to be initially. But still a few daily.

For logged in user, despite all the required fields are populated before submission, uncaptured payment was created with a latest version of stripe and civicrm installed.

@mattwire Just wondering, is this issue going to be addressed in future release? Thanks

There are still things that can go wrong (eg. user failing 3DS check) or not clicking through the confirmation page. So you will always have cases of uncaptured payments. They will be reduced some more when we implement card on file functionality.

Thanks @mattwire. That sounds great! Here are couple of things to consider on my contribution pages settings:

Confirmation screen is turned off entirely
All my inputs are validated prior to stripe payment box can be filled in
Payment is submitted within 3-4min of opening the contrib page
No mistakes were made during entering personal/card information
Recaptcha is validated correctly

In my view it couldn't get any perfect than this, but still uncaptured payment was created. Just wanted to point it out to you, might be helpful to troubleshoot.

Anyways, thanks for all the hard work :)

mentioned in issue #300 (closed)

Closing this as most issues were resolved. Separate issues should be opened for specific problems now.

closed

changed the description

Stripe Authorises payment before form validation causing problems

MJW need funding to work on this issue - Please contribute now to help us continue working on this!

Designs

Child items ...

Activity