Bounce processing fails for invalid unicode characters
We received some bounces that where containing invalid unicode characters. Due to MySQL not accepting to write those incorrect string values to the database, the bounces are moved to the processed folder although they are not actually recorded.
Error Log:
[info] $backTrace = #0 /var/www/drupal/sites/all/modules/civicrm/CRM/Core/Error.php(912): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 [internal function](): CRM_Core_Error::exceptionHandler(Object(DB_Error))
#2 /var/www/drupal/sites/all/modules/civicrm/packages/PEAR.php(931): call_user_func((Array:2), Object(DB_Error))
#3 /var/www/drupal/sites/all/modules/civicrm/packages/DB.php(976): PEAR_Error->PEAR_Error("DB Error: unknown error", -1, 16, (Array:2), "INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#4 /var/www/drupal/sites/all/modules/civicrm/packages/PEAR.php(564): DB_Error->DB_Error(-1, 16, (Array:2), "INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#5 /var/www/drupal/sites/all/modules/civicrm/packages/DB/common.php(1905): PEAR->raiseError(NULL, -1, NULL, NULL, "INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...", "DB_Error", TRUE)
#6 /var/www/drupal/sites/all/modules/civicrm/packages/DB/mysql.php(895): DB_common->raiseError(-1, NULL, NULL, NULL, "1366 ** Incorrect string value: '\x81hrt ' for column 'bounce_reason' at row 1")
#7 /var/www/drupal/sites/all/modules/civicrm/packages/DB/mysql.php(328): DB_mysql->mysqlRaiseError()
#8 /var/www/drupal/sites/all/modules/civicrm/packages/DB/common.php(1216): DB_mysql->simpleQuery("INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#9 /var/www/drupal/sites/all/modules/civicrm/packages/DB/DataObject.php(2438): DB_common->query("INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#10 /var/www/drupal/sites/all/modules/civicrm/packages/DB/DataObject.php(1060): DB_DataObject->_query("INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#11 /var/www/drupal/sites/all/modules/civicrm/CRM/Core/DAO.php(468): DB_DataObject->insert()
#12 /var/www/drupal/sites/all/modules/civicrm/CRM/Mailing/Event/BAO/Bounce.php(81): CRM_Core_DAO->save()
#13 /var/www/drupal/sites/all/modules/civicrm/api/v3/Mailing.php(316): CRM_Mailing_Event_BAO_Bounce::create((Array:6))
#14 /var/www/drupal/sites/all/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php(89): civicrm_api3_mailing_event_bounce((Array:5))
#15 /var/www/drupal/sites/all/modules/civicrm/Civi/API/Kernel.php(96): Civi\API\Provider\MagicFunctionProvider->invoke((Array:9))
#16 /var/www/drupal/sites/all/modules/civicrm/api/api.php(23): Civi\API\Kernel->run("Mailing", "event_bounce", (Array:5), NULL)
#17 /var/www/drupal/sites/all/modules/civicrm/CRM/Utils/Mail/EmailProcessor.php(381): civicrm_api("Mailing", "event_bounce", (Array:5))
#18 /var/www/drupal/sites/all/modules/civicrm/CRM/Utils/Mail/EmailProcessor.php(58): CRM_Utils_Mail_EmailProcessor::_process(TRUE, Object(CRM_Core_DAO_MailSettings))
#19 /var/www/drupal/sites/all/modules/civicrm/api/v3/Job.php(361): CRM_Utils_Mail_EmailProcessor::processBounces()
#20 /var/www/drupal/sites/all/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php(89): civicrm_api3_job_fetch_bounces((Array:1))
#21 /var/www/drupal/sites/all/modules/civicrm/Civi/API/Kernel.php(96): Civi\API\Provider\MagicFunctionProvider->invoke((Array:9))
#22 /var/www/drupal/sites/all/modules/civicrm/api/api.php(23): Civi\API\Kernel->run("Job", "fetch_bounces", (Array:1), NULL)
#23 /var/www/drupal/sites/all/modules/civicrm/CRM/Core/JobManager.php(134): civicrm_api("Job", "fetch_bounces", (Array:1))
#24 /var/www/drupal/sites/all/modules/civicrm/CRM/Core/JobManager.php(79): CRM_Core_JobManager->executeJob(Object(CRM_Core_ScheduledJob))
#25 /var/www/drupal/sites/all/modules/civicrm/api/v3/Job.php(99): CRM_Core_JobManager->execute(FALSE)
#26 /var/www/drupal/sites/all/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php(89): civicrm_api3_job_execute((Array:1))
#27 /var/www/drupal/sites/all/modules/civicrm/Civi/API/Kernel.php(96): Civi\API\Provider\MagicFunctionProvider->invoke((Array:9))
#28 /var/www/drupal/sites/all/modules/civicrm/api/api.php(23): Civi\API\Kernel->run("job", "execute", (Array:1), NULL)
#29 /var/www/drupal/sites/all/modules/civicrm/drupal/drush/civicrm.drush.inc(1560): civicrm_api("job", "execute", (Array:1))
#30 [internal function](): drush_civicrm_api("job.execute")
#31 /usr/share/php/drush/includes/command.inc(334): call_user_func_array("drush_civicrm_api", (Array:1))
#32 /usr/share/php/drush/includes/command.inc(208): _drush_invoke_hooks("civicrm-api", (Array:1), "civicrm")
#33 [internal function](): drush_command("job.execute")
#34 /usr/share/php/drush/includes/command.inc(175): call_user_func_array("drush_command", (Array:1))
#35 /usr/share/php/drush/drush.php(92): drush_dispatch((Array:27))
#36 /usr/share/php/drush/drush.php(61): _drush_bootstrap_and_dispatch()
#37 /usr/share/php/drush/drush.php(16): drush_main()
#38 {main}
I will create PR with a unit test.