disabled optionvalues are not shown in participant export

a participant field has a custom field containing option values. Participants choose one of these option values. The option value that the participants chose gets disabled after they chose it. In a participant export containing this custom field the participants who chose for the disabled option will have a blanc space in the table on that location. Expected is having the option they chose over there, all-tough it has been disabled, and isn't available to choose any more.

to recreate this on a demo site:

1. create a participant to the fall fundraiser dinner with a soup preference

2. go to custom fields and edit the food preference custom field

3. deactivate the soup selection you chose in step 1

4. find participants to the fall fundraiser dinner

5. select all participants and export them using 'select fields to export'

6. select the fields to export:

   • individual - display name to find the person you gave the food selection to
   • participant - food preference: soup selection the field we are interested in

7. open the exported file. The soup selection for the disabled field isn't in there. When there are participants who chose for a soup that is still active the soup selection is in there for them

changed the description

mentioned in issue #36 (closed)

mentioned in issue #65 (closed)

added Concept approved triaged type:proposal labels

Agree it makes sense to export disabled options & a PR on this would likely be accepted (with test)

I have allready made a pull request for this: https://github.com/civicrm/civicrm-core/pull/11817. Apperently it still needed some attention, but colemanw had some different opinions on when a disabled option value should be shown and when not.

added needs:concept approval label

removed Concept approved label

OK - from github

You bring up an important question. What does disabled mean? Doesn't it mean "hidden from all parts of the CiviCRM interface"? And isn't export part of the CiviCRM interface?

Thinking through this some more, I'm inclined to reject this PR. I don't think you've made a case for why disabled options should be expected to show up in exports, but nowhere else in CiviCRM. Why would that be the expected behavior?

@JO0st Contributor JO0st commented on Mar 22 That indeed is an interesting question. For me a disabled option value is unavailable for input, but once shosen they should still be visible in the interface. Disbled custom fields on the other hand can disappear complettly from the interface.

---

@colemanw I'm not sure who to ping for thoughts on this - @lcdweb might have thoughts @JonGold perhaps?

I've been part of several conversations over the years on what "disabled" should and shouldn't mean. I think this should probably be a bigger conversation (civicrm-dev?) and a spec assembled. This would make an excellent sprint project - I estimate it would take multiple days with 3-4 folks to comprehensively fix/UT disabled items.

Two quick examples of behavior I think isn't ideal: Financial types are often more ephemeral for groups with accounting integration - they may have 100 financial types, 80 of which will never be used again. However, you can't delete them without data loss. Disabling them means you can't run a report on them. More importantly, you can't edit a contribution with a disabled financial type, because the financial type will be deleted on save.

So let's figure out how to distinguish between, "This option can't be written to the db (unless it's the value that already exists in the db)" and "This option shouldn't appear as a search/filter criterium". I really think these need to be separate.

I agree with @JonGold that this should be part of a bigger discussion. Another example that a client recently requested was to distinguish admin options from public options. For a given field, they want to limit what is available on a public facing page, but allow all options available to the backend. So I think there are a few different use cases out there that we may want to consider.

For this ticket/PR, I'm inclined to agree with @colemanw that the current handling/intent should be retained -- if for no other reason than it's the way we've historically handled it and how most orgs will assume the disabling function works. But I would love to see a broader discussion to expand the options available.

We'd love to have disabled option values be included in exports (and the admin UI, but exports are more important). We disable options in order to hide them from forms, but very often we'll have an option that still needs to be in an export (for example, a show that's no longer on the network, but we still need to pay them for the current billing cycle).

We noticed this behavior recently when trying to do an export, and we were so surprised that we thought it must have come in recently in our 4.6->4.7 upgrade. But it sounds like it was like this the whole time, and somehow we didn't notice.

If there were a separate attribute on options for "hide in forms," that would work just as well for us.

I could get behind the general idea of disabled options showing up in output but not available for input. I think that would be good for the majority of use-cases, however there may be some orgs relying on the current behavior.

@eileen suggested adding an option to the settings (presumably defaulting to the old behavior). I'd be happy to make a PR for that.

@a_lieb slightly off-base from the original ticket, but because you mentioned it in a recent comment I'll note it here --

I have an extension I'll be releasing this week that extends the "visibility" concept to the option value level. It will allow you to create option values that are admin or publicly visible (public by default, which is the current fixed behavior). It sounds like this may actually be a bit more in line with what you need -- the ability to have option values that are visible to the public, and some that are only visible on the admin side. I'll do a blog post when that is released.

@lcdweb extending visibility field makes a lot of sense!

I'm very interested in the visibility options for option values too! I've had it in the back of my mind that setting visibility of custom fields is also worth pursuing at some point.

mentioned in commit a35c96dd

mentioned in commit d7a1586b

closed via commit d7a1586b

mentioned in issue #339 (closed)

Hi @JonGold and @lcdweb and @eileen so my and a client's strong recollection is that in 4.6 and prior disabled option values for custom data fields were visible, both in the administrative area and upon export. However disabled option values were not select-able, neither publicly nor in administrative area. Example:

Red, Yellow, Blue are options for a field. I've selected Blue. Blue is later disabled as an option. But if you view my Contact you still view Blue. And if you export the CSV contains Blue. But if you were to edit this custom field, the only option values to select would be Red and Yellow.

That's how it USED to be in 4.6. Indeed a few years ago we had an entire workflow built around this behavior, and upon a recent upgrade to 5.x this behavior changed. So I think we already have a precedent (and possible regression) for viewable disabled custom field option values. How can I assist getting this re-established? Thanks!

@Stoob have you tested it on a 4.6 install? I don't recall one way or the other. FYI -- if you want to test my extension that extends visibility to option values you can get it here: https://github.com/lcdservices/biz.lcdservices.optionvaluevisibility

I haven't released it yet because I consider it still in beta. But if you kick the tires and find it working as needed, that may be the nudge I need to get it released.

Hi @lcdweb I just tested and behavior described above is present in 4.6.38. Attached are screenshots of the testing. Would it we worthwhile to determine why this behavior was changed after 4.6 and if the change was deliberate or could instead be considered a regression? If it's a regression it might be more valuable to re-establish this behavior in core (rather than an extension, although thx for making it), since 1) there seems to be agreement among several of us the behavior is useful, and 2) this behavior was already present up until recently.

export.csv

@Stoob If you realy can't work without the situation as it was before I made a pull request to the core that does this. It was not accepted because of the discussion that is being held here. We are using that code in our environment because we rely on that behavior as well. Unfortunately this means patching the civicrm core.

@lcdweb Is there a possibility you could create a README for that extension? I do think this is an important issue, and I'm happy to test the extension if it provides even a partial solution to the issues discussed here.

@JonGold yes, I'll work on that. However -- my extension doesn't address @Stoob's issue directly; it's a different approach which may or may not meet his needs (and I think the discussion about whether this is a regression or an intentional change should continue to be fleshed out). My extension let's you set the visibility of an option value to public or admin. So the option value remains enabled/active -- but you can control what is exposed to the frontend vs. the admin backend. That's a little different from the 4.6 behavior, which doesn't allow the value to be selected in either the frontend/backend, but retains the existing value in view/export. (I actually think that behavior is problematic because editing the inline block basically forces you to select another option; if I understand correctly, you wouldn't be able to preserve the existing, now disabled, option.)

@lcdweb makes a useful point, but that "problematic behavior" is in 5.5 and 4.6 alike and so apparently hasn't changed. To be clear, with the described behavior above, if you edit a inline block or a profile, a field with a previously selected disabled option value is set to null unless a different & enabled option value is selected. That's according to my continued testing today.

That might be more problematic in Contact-level data that persists and is re-edited time and again. But this behavior is less problematic in Contributions and Participant data, which are usually not re-edited.

So it appears that what we are looking at is specifically a regression of behavior change on whether the disabled option value is visible or not.

@Joost do you refer to https://github.com/civicrm/civicrm-core/pull/11817? Since based on my testing this behavior change appears to be a regression from core 4.6 to 5.x, I hope that your PR or something similar can be merged to core.

@JonGold I added info to the readme for that extension.

@Stoob Yes i meant that one, forgot to paste in the link

@Joost I've pinged @colemanw on this and hopefully he can evaluate your patch, to re-establish this functionality in core.

mentioned in issue #613 (closed)

mentioned in issue #723 (closed)

mentioned in issue #978 (closed)

mentioned in issue #1015 (closed)

mentioned in issue #1183 (closed)

mentioned in issue #1427 (closed)

mentioned in issue #2089 (closed)

mentioned in issue #2377 (closed)

mentioned in commit 081629dc

mentioned in issue #3698 (closed)

It's been noted that activity types could be considered a little bit different from regular option values, and so perhaps as described above for financial types, they might be handled differently. Wondering what people think about disabled activity types being displayed. Specifically https://github.com/civicrm/civicrm-core/pull/23980.

mentioned in issue #4267

mentioned in issue #4390 (closed)

mentioned in issue #5284 (closed)