Extension does not work with CiviCRM Webforms with Contribution page

Created by: artfulrobot

description is passed into the "redirect flow" (in GC you set one up, get a special URL to send the user to, then they come back to you and you have to refetch the redirect flow object from GC to complete the processing) and its existence is checked for on completion of the redirect flow.

(You should have been killed by an exception not the notice, patched that)

The description is used for the subscription's 'name' parameter, which is later used by GC as the public-facing 'description' for each payment. GC docs

This extension does require it. However I don't think it's your issue; I think manually setting it should work.

The session is handled by CiviCRM; I don't think the description is used. The quickform key, however, is used as a GC session token, which may be related. I've not actually tried this with webform, so I'm not familiar with the issues.

Could that be something else? When do you see that error - when presenting the webform, or when submitting the webform, or after having completed the off-site GC page?

Created by: PJKerrigan

Thanks for your quick response!

The error occurs after I've entered all of my details into GoCardless. It redirects me back to my site and displays that CiviCRM error message. White screen and the "We can't load the requested web page. This page requires cookies..." message.

Without setting the description manually, it just attempting to go to GoCardless and redirects to /civicrm.

I'll keep trying as it looks like I'm in the right area. Cheers!

Created by: artfulrobot

Hmmm.

The redirect page is set at https://github.com/artfulrobot/uk.artfulrobot.civicrm.gocardless/blob/master/CRM/Core/Payment/GoCardless.php#L111

So if it's not for an event, it's the generic contribute/transact. But note the qfKey is here - is that in use in a webform context?

Created by: reswild

I'm using this extension with civicrm_webform + a bit of custom code to set up the payment instead of using a CiviCRM Contribution page, and I had to hack the GoCardless.php page referred to above in order to insert a custom redirect to my own page instead of to the CiviCRM contribute/transact page.

So if you get really stuck, I could share my code with you, but it's a bit of an ugly hack, so you are probably better off if you can figure out how to do it the regular way.

The qfKey should be there also in a webform context, so it might only be a question of how to pass it onto the GoCardless extension.

Created by: artfulrobot

I wrote the extension so that it could be driven without contribution pages (which I never use, preferring a more custom and streamlined UX). So the methods to create a flow are available without using quickform (I am on a personal mission to rid CiviCRM of quickform!). @reswild I'd be interested in seeing your hack.

Created by: PJKerrigan

Would it be possible to get a look at your hack, please, @reswild? It could help a lot!

I'm not very familiar with CiviCRM yet, enough to solve this. I'm more Drupal oriented, so I'm not entirely sure whether this is the problem. Looking at the code, I'm inclined to agree that the description changing shouldn't be causing the problem with the session.

I've checked the resource URLs, base URLs in settings.php files, directories, cleared caches, etc, after other people with similar problems. But I can login to the site fine, so it doesn't seem like all sessions are broken. Just in this case.

Created by: reswild

I can upload my custom module to my GitHub account later this weekend - I just need to clean up the code a bit first.

When you use contribution pages with webform_civicrm, I believe they are just loaded through javascript, so your session isn't set the same way as when you use the contribution page from within CiviCRM. So you might need to go through a couple of extra steps in order for the qfKey to work correctly.

Created by: artfulrobot

@reswild ah, that makes perfect sense.

Created by: PJKerrigan

In the mean time we're just going to go with a modified Contribution page instead of the webform.

It would be good to look at your code, though. Cheers.

Created by: PJKerrigan

I'll be looking at this again over the weekend, as I'd love to finally solve it! It's been eating at me.

Created by: artfulrobot

This might help.

The process is

create a 'GC redirect flow'.
- This requires a session token of some sort which is passed into the create call and stored by GoCardless. It must also be kept by the CiviCRM server in session -see point 4.
- It also requires a redirect URL.
send user off to the GC site to set up the mandate. User submits GC form.
GC tells user to redirect to the redirect URL you supplied in (1), appending the redirect flow ID as GET data.
The CiviCRM server handles the redirect URL, extracts the redirect flow ID and calls GC to ask for the redirect flow object. It must then compare the token stored in the retrieved object with the token stored in session data in (1)
The CiviCRM server then completes the redirect flow and sets up the subscriptions etc. and finally gives the user some thank you page.

Sounds like the problem is that the session token stored in (1) is not available when using webform in (4). Over to you to find out how webform and CiviCRM work together with session storage there!

As an alternative, you don't have to use CiviContribute's contribution pages. You can call the methods on GoCardlessUtils directly yourself, call getRedirectFlow() and store what's needed on your session, then make your own callback (or hook into a webform page) to call completeRedirectFlowWithGoCardless.

hope this helps, Rich

Created by: TomCrawshaw

Hi @reswild, did you get round to uploading your hack?.. I've got the same issue, and would love to be able to get webform working with GoCardless.

Created by: reswild

Yeah, sorry, I did look through my custom GoCardless module, but there was a lot of unrelated code there that I wanted to clean up first, and then I never got around to it. I'll see if I can set aside some time for this next week.

Created by: TomCrawshaw

Thanks for this. Looking forward to seeing your fix, but if you just want to let me have the "messy" version that's ok!

Created by: TomCrawshaw

Hi @reswild, did you get a chance to look at this? Although my coding isn't up to creating a webform/GoCardless solution from scratch, I might be able to use yours as a basis if I could see it please. Best wishes.

Created by: reswild

Yeah, I'm actually doing some work on my GoCardless integration this summer - my code is a bit of a mess at the moment, but I'll try to get something added to a public repo so you can have a look at it. I will also try to rewrite some of the code so it can be committed here (like support for Euro payments and pre-filling of customer data in gocardless.com).

Created by: Upperholme

Would love to see this working with webform_civicrm - just to add my vocal support.

Created by: artfulrobot

If you have a client who needs this I would be happy to do an estimate.

Created by: adshill

@artfulrobot Rich how much development would we be looking at to get this working with webform module? We need to migrate a site from the Veda extension but it uses webforms.

Created by: artfulrobot

@adshill I'm not sure. It's potentially a small change (to do with storing form keys in the session, and identifying a different redirect URL after completion of the redirect flow), but I need to find out exactly what's going on wrt webform compared to the normal civi proceses. It's at least 4-6 hours unless I get lucky ;-) If interested, drop me an email.

mentioned in issue #37

Interested in picking this up again if it's not going to be costly. Wondering if @reswild got any further?

Also interested in this / wondering if anyone was able to share progress?

Hello, we're just assessing whether we should use Webforms for GC contribution/membership pages in our Drupal 9 rebuild, so will be testing out what happens in the next few weeks. Can report any findings here if that's useful or start a new Drupal 9 related ticket. Looking at the webforms module page it says "Webform for Drupal 9+ is a completely new code base" so this may be new territory... if anyone else has explored yet that would be good to know. EDIT: I didn't actually get round to testing this but probably just as well!

Just wanted to note not working for me with D10 webform. I'd love to get it working but not sure I'll find the time/money for it any time soon

@ufundo yeah, unfortunately these things don't get magically fixed over time :-)

I don't have clients using D10 or webforms, nor anyone here offering funding or merge requests, so I have no plans to work on it.

Can you give me a ball park figure on how much this would cost to fix?

changed the description

added time estimate of

assigned to @artfulrobot

Investigation work write-up

Thanks to @jonty17 for funding this.

This will be hard to implement and will require changes to webform as well as additional custom code in either the GoCardless extension or a new GoCardless-Webform extension.

The main problem is that webform does not do things the CiviContribute way; it does its own custom contribution processing with its own logic that I expect was once copied from core a long while back and is now neither the recommended way nor extensible.

Ideally, webform's payment processing code would be caught-up with core; it should be using a ContributionRecur.create » Contribution.create » Payment.create flow not Contribution.transact. It should set up all the data for a contribution before passing to the payment processor and it should not duplicate/replace core logic. It should be able to use test processors.

The flow GoCardless needs

GoCardless' Billing Request API requires:

A Billing Request and Billing Request Flow be created.
The Billing Request to be attempted: this involves data collection by GC and results in success/failure
On success, a subscription must then be created.

However, it also requires that step 3 be asynchronous; i.e. it might not happen during one user's push of a button.

This is similar to several other payment processing platforms.

The flow the GoCardless extension supports

Pending ContributionRecur (hereafter: just recur) and a Pending Contribution be created with all the data needed to describe the intended subscription.
Create Billing Request and flow IDs. These are written to the recur + Contribution records but will later be overwritten with different IDs.
A process to interactively (attempt to) complete the flow.
A process to detect successful completion of a flow and update the recur + contribution accordingly.

There are currently two known implementations:

For Civi core, this is done by the GoCardless extension by sending people off-site to GoCardless-hosted pages on the final submission of the Contribution Page, and using the thank you page (part of the CiviContribute) as the return page, intercepting the processing of that using the build form hook to do step 4.

For InlayPay, it is achieved by custom calls via Inlay's endpoint + using GoCardless provided client-side code for step 3. Then a final custom call passes the outcome back to the server for step 4. InlayPay's method is much nicer - no whole-page-reloads - and allows more features (e.g. it can support instant payment for first payment, then continuing with direct debit). By not sending people off-site, it also avoids trouble with session loss on return; some browsers are aggressive about deleting temporary sessions.

The payment flow of webform

Webform expects to:

present a form that collects all the required data
be able to find/create the contact and do the payment stuff in a single validation call in which it also creates the recur + contribution.
continue on after validation to do more processing, including payment related work.

Possible webform flow

Re-engineer the webform's contribution handling to

intercept the form build for the contribution page and give processors chance to inject some Javascript. There's probably a way to do this already.
Include a way for client-server communication of machine data. This could be a hidden field on the form, for example, whose value can be set by Javascript and passed along in regular form submission, and set on the back end and passed back in the form.
find/create contact (as now)
if any validation errors: return them now.
Check if we have pending recur + contribution IDs stored in form state already; if not create recur + contribution in pending state and store these IDs in the form.
Offer a new hook: civicrm.webform.paymentattempt. The Event object passed will contain a reference to the webform (to make its data available to other code) and a reference to a status variable.
After running the hook, the status is inspected:
1. the status is untouched. webform continues to do its thing in the way it has before, but using the recur + contribution IDs already set up.
  In all other cases below, webform's existing payment stuff is skipped.
2. the status is "needs action". webform marks the form invalid and also sends data from the event back to the browser, e.g. JSON added to a hidden form field, for any payment-processor-custom-javascript to pick up on.
3. the status is "payment failed". As previous arm, but the data may contain information such as the reason the mandate setup failed.
4. the status is "payment ok". Webform may consider the form valid, and can move onto the next page/finish.
JS on the webform contribution screen looks at any data it received and takes action accordingly.

Then, if this was in place, the GoCardless integration could:

on initial submission ➊, we implement a listener on the new hook, wherein we recognise that there's no Billing Request yet, so we create one, and store its IDs on the recur + contribution. We return status == 'needs action' and also pass the billing request flow ID back.
the form page reloads but this time our javascript spots the billing request flow ID in the hidden data, and kick-starts GoCardless' JS SDK process to obtain a mandate from the user in a modal overlay, leading to one of two situations:
1. the mandate set up failed/cancelled.
2. the mandate set up ended without error.
The javascript now submits the form.
on this next form submission ➋ our hook listener notices the billing request IDs in the custom data and looks them up on GoCardless to see what happened. If unsuccessful, we cycle back to ➊, creating a new request, setting status to "failed" and a suitable error message ("that didn't work. You're welcome to try again."). If it's a non-error (rarely will it be a "success, completed" because of the async nature), then we now do our processing of setting up the subscription, altering IDs of the recur + contribution, and return status "payment ok".
Webform can continue to the next page/thanks/etc.

Hurdles

webform doesn't use Test mode processors! If you disable live payments, it just skips the payment stuff. This really needs to be fixed in my opinion because otherwise the only way to test is with a live real money processor, or by hacking your live-processor to actually use test credentials, but this may be non-trivial. Webform in test mode should use test processors so the full process can be tested. If webform wants a checkbox to skip payment taking, that's fine, but it's a separate feature.
Civi's payment processing has not traditionally been API-first, instead focussing on serving the needs of its Contribution Pages. Contribution Pages were built to support the logic of available processors of old and things have moved on a lot. As a result, webform duplicates/reimplements a lot of the financial/core contribution logic, but this looks like it was taken from a point a while back in history. e.g. it does not use Contribution.create » Payment.create APIs, nor doPayment(). As long as Civi has it's current restrictive logic it won't encourage better APIs, and a lot of work will go into hacking processors required flows into Civi's old system, requiring yet more dev on the payment processor side if it also wants to support a more reusable API. It also means that webform (or other forms) have to recreate a lot of logic, and this compounds the problem.

Costs

There's a lot of work to be done here. The webform changes: these could maybe be done in a way that means: if the new hook is used, we cut out all the old payment processing; if not, we keep the current way - not because it's good, but just to avoid the costs of poking a hornet's nest (presumably there's lots of people+processors for whom the current thing works). In theory it might be ~4 hours' work if things work out cleanly, but I would not be surprised if there were some gnarly gremlins waiting.

Implementing proper test mode support in webform: 5 hours? Without this, then it would mean changing/hacking the GoCardless extension to support the idea that test credentials can be put in the 'live' payment processor. Which is a recipe for problems; may need to be restricted to dev environments; but is probably cheaper in hours. But webform should support test processors.

Implementing the GoCardless specific billing request flows after the above changes: I learnt a lot doing this in InlayPay. Although here we would not be targeting instant payments, which add a lot of complexity. I think this is 12 hours.

Other ideas?

I can't see a reliable way to leave the webform (during "validation"), use the offsite GoCardless pages, and come back to continue on the next page, but perhaps there is a way, so if you have ideas, please do share.

Were one to want/need to continue with minimal changes, acknowledging that this means continuing to build more code upon code that has been deprecated since 2018 or 2019 (contribution.transact) then a hacky fix might be:

Add a hook in CRM_Core_Payment_GoCardless::getBillingRequestFlowURL() to configure the return URL - currently it assumes event or contribute - to point back to the webform with some success identifier.
have js identify the identifer, and add in some data to the form and re-submit.
have webform spot the new data, and disable its payment processing - like it does in 'test' mode, allowing the form to proceed.
Rely on the async webhooks to complete the subscription.

Adding hook: 1hr. Adding hidden field to contain identifier data; adding JS to identify this from the URL and re-submit form: 2hrs. Making webform look for this data and skip payment: 1hr. Realising that it needs some extra data to continue, handling failures: ?hrs. Total: 4 hours + the ?s + ? testing - note that there are still costs here due to the lack of test mode, unless someone wants to supply their bank details for who-knows how many test mandates...

Thank you for writing all of this Rich, hugely helpful!

OK, so that plan did not work!

But I found another way, here's proof:

webform-gocardless-working

Ingredients

patched webform_civicrm Drupal module. https://github.com/artfulrobot/webform_civicrm/tree/support-gocardless
latest 2.0.2 GoCardless extension (not yet released). https://lab.civicrm.org/extensions/gocardless/-/tree/2.0.2?ref_type=heads
a contribution page that uses GoCardless for recurs
a webform that uses the contribution page.

Method

The patched webform_civicrm Drupal module was branched from the 7.x-5.x (dev) branch. So if you're on some older version I'd upgrade to 5.8 first, then try replacing with the patched module (zip download link)

Then upgrade the GoCardless extension to 2.0.2. To do this, you'll need to move your current gocardless ext somewhere away and safe (in case you need to go back to it); download the source code (e.g. with git clone https://lab.civicrm.org/extensions/gocardless.git -b2.0.2 or unpacking a tar.gz and then run composer install from within the downloaded directory.

Set up the contribution page. Of course you'll need to recurring, and select GoCardless as the processor.

Set up the webform. Select the contribution page and remember to set the frequency to Monthly (Annual should work too).

This should then work in a live context.

If you want to test it instead of just going live (and who wouldn't?!)...

...there's a bit more of a fiddle. It seems to me that webform might not support test payment processors (if someone knows webform well, I'd love to be wrong about this). But it looks to me that if you don't enable live payments then you don't get any payments.

So for the sake of testing only you'll need to do the following. Please note that doing this on a live site will break your actual real money GoCardless integration.

Add this to your civicrm.settings.php file:

define('CIVI_GOCARDLESS_FORCE_SANDBOX_ENVIRONMENT', 1);

Then edit your GoCardless Payment Processor and (a) copy the existing API keys + webhook secrets somewhere safe, then (b) put the test/sandbox versions in civi's "live" fields.

You should then be able to cook up the above video.

Proof is in the tasting

If you get as far as this, then you go look at a contact who completed the webform, you should expect to see a Pending contribution linked to an In Progress recur. The recur will probably have a "processor ID" that begins BRQ... (If it already has one beginninng SB... you're all good.) This means no subscription has been set up yet (just the mandate). However when the webhook comes in, the subscription should become set up (at GoCardless) and you should see the processor ID has changed to one beginning with SB... The Payment will remain Pending until a week later - unless you force it via the developer panel in the gocardless admin.

On a live site

On a live site you'd just install the new webform + gocardless as above, but you wouldn't do any of the civicrm.settings or swapping API keys bits - all that is just for testing.

Note: the D7 branch of webform-civicrm is kinda EOL, so this change is not going to make it into webform-civicrm: https://github.com/colemanw/webform_civicrm/pull/967

Closing this after updating docs. See 2.0.2 (released v soon)

closed

Extension does not work with CiviCRM Webforms with Contribution page

Designs

Child items ...

Activity

Investigation work write-up

The flow GoCardless needs

The flow the GoCardless extension supports

The payment flow of webform

Possible webform flow

Hurdles

Costs

Other ideas?

Ingredients

Method

If you want to test it instead of just going live (and who wouldn't?!)...

Proof is in the tasting

On a live site