Special characters become "?" On PDF report

Overview

There's an issue when exporting special characters to PDF format on civicrm report. All special characters turn into "?".

This problem can be replicated on https://dmaster.demo.civicrm.org. 

After discussing the problem on do-ocracy 2020 channel, we found out that dejavu fonts were removed on dompdf fonts folder to save about 12mb for civicrm. Thus, this issue emerged.

Comments

Anything else you would like the reviewer to note.

changed the description

Here's the link to the conversation for the background: https://chat.civicrm.org/civicrm/pl/qxhbtp4pjpfgb8yx6gp7nbwmhc

This sounds familiar. In January I updated the mailing labels to default to Deja Vu for the same reasons, but they use tcpdf. I remember briefly looking at dompdf but didn't follow it thru: #1502 (comment 29489)

Oh wow - maybe extending that fix would help - ug no - it still needs the deja-vu font to be present

As Luke pointed out on chat - this is the offending line https://github.com/civicrm/civicrm-core/blob/96879c169e7526004925e3eb6921ac03e7071985/tools/scripts/composer/dompdf-cleanup.sh#L125

the issue is that for sites using web based installs an extra 12MB might be problematic. We did talk about whether we could have a multilingual downloader extension - rather than the existing l10n tarball & solve this as part of that

Given this could be impacting half potential users (more?) – ie every language not using a latin typeface - what's the best way to ensure this doesn't get delayed much longer?

I do web-based upgrades/installs on Joomla, so can easily test with a larger tarball. There was an issue with 32mb as a limit in the past somewhere I think, but my memory is it was a server/hosting constraint and many/most have raised that now. Civi 5.31 beta 5 I see is 31.4mb so maybe this was about keeping under 32mb?

Just posting these two StackExchange posts here for reference (mentioned in @DaveD's comment):

https://civicrm.stackexchange.com/questions/29060/special-characters-become-question-marks-in-pdf-printing

https://civicrm.stackexchange.com/questions/27735/how-to-change-the-encoding-in-pdf-documents

Maybe we can try to clarify what the size limit was about. My understanding is we're not talking about something like a cpanel filemanager, but something like the drupal option described at https://docs.civicrm.org/installation/en/latest/drupal7/#download under "Download via Drupal Web UI", where it often doesn't work anyway with many PHP configurations.

I don't know if joomla has something similar - I don't see it in the civi docs, and I also don't think civi works in wordpress that way. So if we're just talking about a drupal 7 thing which often doesn't work anyway, then let's just remove those lines from the dompdf-cleanup script.

@DaveD One concern is for WP. If the install is done via the WP User Interface, we have to deal with the max upload size in php. 32MB has been a common limit.

Ok good to know. I didn't see that as an option in the install docs for WP.

My followup question then would be for people installing l10n files, they'd still have to do that separately, right? Not in the CMS UI. So at least for the short term, since that's the bulk of the target audience here, could update the docs to say how to install dejavu, or wkhtmltopdf?

It's not an option as we don't recommend it. But Stack Exchange and support over the years shows that users will try to install CiviCRM the 'normal' WP way.

I am not sure we need to preserve this functionality, but if we ever want to use WP's automatic updates, we have to be aware of the file size. That's a whole different conversation

My followup question then would be for people installing l10n files, they'd still have to do that separately, right?

Correct.

mentioned in merge request documentation/docs/installation!19 (merged)

Docs PR which explains what to do for now: documentation/docs/installation!19 (merged)

There was some discussion on Thursday's call about bundling the fonts in the *-l10n.tar.gz distribution. I'm not sure about the technical consequences, but at a superficial level it makes some sense because:

• anyone needing non-latin characters in PDFs would be downloading that package anyway.
• installing requires manually moving the l10n & sql directories into your civi root, so seems less likely to hit php upload limits (or easier to work around at least). ?

added git:civicrm-core sig:bug labels

added regression label

I have given this the regression label as I do understand that in the latest release of CiviCRM certain fonts are missing and therefor the pdf generation doesn't always render all characters correct. This would probably affact users in Norway, Germany etc... But I am not completly sure whether this is a regression so feel free to remove that label if you think otherwise.

Seems pretty old: https://github.com/civicrm/civicrm-svn/commit/c4a71302be4ab73a39cfc4938926546d75234a8b

It's not ALL pdf generation, just the ones that use dompdf, such as CiviReports.

Just musing some more thoughts:

I don't know what the scope of converting dompdf code in civi to use tcpdf would be or if it has all the needed features. The tcpdf included with civi includes dejavu fonts and somehow they only take up 2 MB.

Dompdf will also often crash for civireports (just take the stock demo Constituent Detail report for example) so dompdf may not be the best library for it anyway. There's wkhtmltopdf as another alternative which has dejavu but it's hard to bundle that by default in civi since it's operating system-specific.

Update: Seems it was attempted to convert to tcpdf https://issues.civicrm.org/jira/browse/CRM-14567 but then reverted because of lack of css support, but that was years ago and now seems to handle it.

removed regression label

added prioritised label

I tried fiddling with un-re-reverting the tcpdf implementation and it seems to still have the problem with civi's use of css. Also it crashed for me same as dompdf on medium-to-large input.

The install docs above were merged but that's just a workaround. If memory is always going to be an issue converting html to pdf, then maybe it needs an architecture change, or now that all browsers internally support saving to pdf, maybe there's some way to get the html output to trigger the browser's save-to-pdf feature.

Oh, no that last part won't work with scheduled mailed reports, unless there's a way to do it with headless chrome or something.

the issue is that for sites using web based installs an extra 12MB might be problematic

One concern is for WP. If the install is done via the WP User Interface, we have to deal with the max upload size in php. 32MB has been a common limit

Just noticed that the core tarball is now well over 32MB. Afform alone accounts for 68M when unzipped. Hmm.

I don't get it. On my local the afform directory is 1.3 mb. And that includes all the .git files which take up a lot of space and aren't included in the tarball.

Update: Oh, it's Monaco Editor. That thing is huge.

I've added a PR that would make it easier to add your own fonts and have it survive civi upgrades: https://github.com/civicrm/civicrm-core/pull/21423. Note that even if DejaVu was included in the tarball, the distribution would still be missing a good font with CJK characters, so here you can have your own arbitrary fonts.

mentioned in commit 31902b21

Docs PR: documentation/docs/installation!35 (merged)

removed prioritised label

mentioned in issue #3769 (closed)

Hi,

Thank you for raising an issue to help improve CiviCRM. As you may know, this issue has not had any activity for quite some time, so we have closed it.

We would like you to help us to determine if this issue should be re-opened:

• If this issue was reporting a bug, can you attempt to reproduce it on a latest version of CiviCRM?
• If this issue was proposing a new feature, can you verify if the feature proposal is still relevant? Did it get the concept-approved label? Have other people also shown interest? Could it be implemented as an extension?

If the answer to either question is yes, please feel free to comment or re-open the issue. Please also consider:

• Is it something that you could help implement, either by sending a patch or hiring someone who can?
• Would you be willing to fund the work, either through a CiviCRM partner, or the Core Team?

Thank you for your help and contributions to CiviCRM.

P.S. This is an automated message, see infra/gitlab issue 20. We understand that automatic responses are annoying, but given the number of open issues as the project evolves, we need a bit of help to triage and prioritise the most relevant issues.

closed