Thousands of failed/spam transactions

I haven't seen this problem before on sites using the Stripe integration, but on two sites that have been upgraded to v6.1.6 of this extension, I see thousands of spam attempts in the Stripe account:

Could this be related to 6.1.x? Has anyone else seen this in relation to newer versions of this extension?

@laryn You're the only one that I've had a report like that from. Do you have reCaptcha or similar enabled on the contribution page - as that should help prevent spam submission? It may also be worth contacting Stripe directly as their "algorithms" are supposed to help stop this kind of thing.

@mattwire Thanks, I'll follow up with Stripe to see what they say and if necessary will try a ReCaptcha (although I prefer not to make people do that if possible). The strange thing is I never noticed it except on these sites after upgrading, but I know it's not always cause and effect... Will let you know what I find or what they say.

@laryn Before upgrading you'd have had no way of knowing I think, because nothing would be recorded at Stripe until payment was actually taken and, previously the civicrm_stripe.js would have returned a card declined or other error and never submitted the form. So it probably was happening before, but now you have visibility of it!

added discussion label

Interesting... So far I've chatted with Stripe, who suggested ReCaptcha and forwarded it along up the chain (haven't heard anything more yet), then I implemented ReCaptcha which didn't seem to make a difference. I spot checked these entries in Stripe and they all seemed to be coming from the same IP address, so I've banned that IP for now...

I'm experiencing this, too. Thousands of repeat payment submissions that I can see in our logs. Resulting in hundreds of thousands of failed POSTs from the webhook, where Civi returns 400 bad request.

I blocked an IP range in Brazil, and it started up again from Cyprus! I've now blocked that one as well.

Very alarming to hear that ReCaptcha did not stop the submissions, for @laryn! Could it just be the resubmission of failed webhook POSTs that Stripe does? How long do those last?

Also, even if it is difficult to stop the SPAM, shouldn't the plugin be able to return something other than a 400 in this situation, so that it is more Stripe's problem and hopefully doesn't endanger the function of our Stripe webhooks? Stripe is treating it as a malfunction on our server, when it seems everything is working just as they designed it.

Also, even if it is difficult to stop the SPAM, shouldn't the plugin be able to return something other than a 400 in this situation, so that it is more Stripe's problem and hopefully doesn't endanger the function of our Stripe webhooks? Stripe is treating it as a malfunction on our server, when it seems everything is working just as they designed it.

I'd be happy to look at putting something in place, however I don't know how you would go about identifying the spam ones from the genuine ones in a reliable way? Thoughts @johnk If CiviCRM is returning a 400 bad request it is probably also logging something in the CiviCRM log files - what is the cause of the bad request?

Aha! I didn't find anything in the CiviCRM log, initially, but I had checked the wrong file. (Recently when I go to the log in the web interface, there is an error, "unable to read entries from logfile." I have to look manually in that directory now, for some reason.)

The error is:

Nov 01 17:39:51  [debug] Stripe Exception: Event: charge.failed Error: Missing customer_id!

Nov 01 17:39:52  [debug] Stripe Exception: Event: charge.failed Error: Missing customer_id!

Nov 01 17:39:54  [debug] Stripe Exception: Event: charge.failed Error: Missing customer_id!
...

I get the same Error: Missing customer_id! debug notifications.

At this point the fraudulent attempts are still coming in, but seem to have slowed down somewhat. Not sure if that's the ReCaptcha or not. There are already 8 for today, but that's way down from earlier (if you see the screenshot, they were coming in about 10 per minute!)

I found the answer to my question about the resubmissions in the Stripe docs. It's 3 days, with an exponential back-off. So these will all expire before the service cutoff date that was in my email. It's close, though! I'm still getting about 1 per second.

@laryn @johnk We might be able to match on the missing customer IDs and return a 200. It would be helpful if one of you could send me a dump (privately) of one of the charge.failed events so I can see what parameters are/are not being sent over - and if you have a "valid" charge.failed a dump of that would also be helpful.

Okay, I sent it to you by email.

mentioned in commit b56bf8ab

changed milestone to %6.3

added Waiting feedback bug labels

@johnk @laryn Fixed in 6.3 - please confirm (you'll need to download the 6.3 branch of stripe + 0.6 branch of mjwshared).

closed

For me, the webhook events have stopped now. So I can't easily test it. But thanks for making this change! I had a look at Stripe.php and StripeIPN.php and the logic seems correct: doPayment always creates and then specifies a customer, so it must be that Stripe is dropping it (presumably due to the fraud detection kicking in).

I know this is closed, but I'd like to understand it more.

Can somebody who experienced it spell out which URL/path were the spam submissions being sent to/where in the process was the spam happening? e.g. was it on the custom confirm payment ajax endpoint or the main civi contribute form submission (after having generated a payment intent via confirm)?

It seems the fix has simply focussed on ignoring the results rather than preventing the spam at source; I'd like to understand more to see if there's a way to prevent it earlier.

@artfulrobot My understanding of this is that it was targetting the frontend contribution page - which I understand is a common target for spammers to test credit cards and needs protecting in some way. Because the pre-elements version of Stripe did not validate the card details or create any payments until the form was submitted and validated you would never know it was happening. But now as soon as you enter card details and click submit you have a record (paymentIntent) that gets created. So I've fixed the symptom because we already ignore lot's of webhooks from Stripe that are not relevant / invalid. But the correct fix for each site is to make sure that you've got adequate protection (such as recaptcha etc) on the actual form.

OK, so spammers are either humans or specifically designed Selenium-type bots which load the page with all the JS, wait for Stripe's JS to initialise, feed in data to Stripe's iframed elements, wait for paymentMethod creation, then trigger form submission?

Most spam is much simpler (e.g. stateless POSTs to particular URLs), are we sure that the above is what's happening?

@artfulrobot Well, when you put it like that :-) It's more a question for @laryn and @johnk I think as I've not seen the setup of their sites.

Likewise for me, I don't see this anymore on the site where this was happening. I suppose the Captcha may have done it -- although the webhooks continued for some time after I implemented it, perhaps those were the retries that Stripe does automatically, as was suggested earlier.

As far as code/implementation, I don't have the knowledge to form an opinion on the best way to handle this, I'm afraid!

Their script was POSTing directly to /civicrm/stripe/confirm-payment.

I get the impression that it's not card testing, but rather a DoS attempt. They did succeed in briefly shutting down our Stripe payments (I just had to log in to the dashboard and reenable our webhook, after it automatically deactivated).

Most of the organizations that are similar to us don't have captchas on their online donation pages. It is something we'd like to avoid, if possible, because of the effect on conversion rate. I am hoping that, if it happens again, it will be sufficient to come up with a way to do IP-based rate limiting.

@johnk as I suspected!

And if so, a captcha is not going to help.

Possibly the spammers thought it might do something it didn't; thought they could use their own PaymentMethod/Intent IDs against that end point - could be why it stopped. Or it could be testing it as a (D)DOS vector as you suggest; since it's going to take a while to process it's going to clog the server. Were they all from the same IP?

There's a ticket to close off that endpoint and move it to an API but I guess we're still left with the issue that we need a public endpoint for that function, but the new public endpoint (which I guess might be the same URL, but become a wrapper for the API) could check the form key is valid?

Mentioning @jamie on this.

It's hard to understand why this attack happened.

I don't think submitting a paymentIntent or paymentMethod generated from one account would work if it was later retreived from a different account, right? If that's is possible, then I could believe this as a elaborate card checking scheme.

And as an intentional DDOS attack... seems like there are much easier ways to take down a CiviCRM site (I do it to myself on accident all the time) and attacking this end point seems pretty random.

Based on real experiences, we implement some very generic fraud settings across all Powerbase contribution pages (we block you if you submit an anonymous contribution more than 4 times from the same IP in one hour and also if you make a contribution with that is not an even dollar amount). These are hideously simple and we have to expose a button that allows groups to turn them off when necessary (e.g. a fundraiser with a terminal for people to make contributions). Also, the code is buried in one of our legacy Drupal modules.

Having said all that... I wonder if the best long term answer would be to write an extension that attempts to provide some abuse defenses in a more generic and configurable way. If there is a way to intervene at the API level, for example, to handle floods of traffic from anonymous users. I know we have IDS system in place already - maybe that could be tweaked?

@artfulrobot it was from various IPs, but across only about 3 different class B blocks.

mentioned in commit 269e9371

@jamie @artfulrobot @johnk We are seeing this on a site where we've just installed the Stripe plugin. Card testing fraud on a significant scale.

Did we find a good way to protect against this? We are considering Cloudflare rate limiting but this is going to create friction on the payment process for end users.

I've seen this issue may protect against it: #131 (closed)