From b23b65f762aec3f2e5a0aec7f2a068e2fb1c5f2b Mon Sep 17 00:00:00 2001
From: Seamus Lee <seamuslee001@gmail.com>
Date: Mon, 21 Aug 2017 09:39:58 +1000
Subject: [PATCH] WIP on expanding events and merging in the other token
 documentation

---
 docs/framework/civimail.md | 138 +++++++++++++++++++++++++++++++++++--
 1 file changed, 133 insertions(+), 5 deletions(-)

diff --git a/docs/framework/civimail.md b/docs/framework/civimail.md
index 99d5cbe4..5c8e45a7 100644
--- a/docs/framework/civimail.md
+++ b/docs/framework/civimail.md
@@ -16,17 +16,127 @@ If the contact has multiple locations, a location preference may be set for each
 
 Note that a contact may only have one subscription record for the group, so the mailing will go to at most one of the contact's email addresses.
 
-## Tokens
+## Tokens (#tokens)
 
-In 4.7+ there was major changes to the Scheduled Reminders facility which also included changes to CiviMail in so far as how tokens are generated. There is a move to use more of the `Civi\Token\TokenProcessor` sub system as this is more robust. However there have been compatibility layers built in to use the older `CRM_Utils_Token` processors. Developers should aim to work off the `Civi\Token\TokenProcessor` where possible. However there are still some systems that haven't been refactored. Some of the key functions in the older systems are. 
+CiviCRM's token functionality originates in CiviMail, which focuses on writing and delivering newsletters to large constituencies. In its original form, the design placed heavy weight on:
+
+- **Performance**: Divide mail-composition into batches and split the batches among parallel workers. Moreover, when processing each batch, improve efficiency by minimizing the #SQL queries - i.e. fetch all records in a batch at once, and only fetch columns which are actually used.
+- **Security**: Do not trust email authors with a fully programmable language.
+- **Contact Records**: The main data for mail-merge came from contact records. Other data (contribution, event, participant, membership, etc) were not applicable.
+
+Over time, the token functionality evolved:
+
+- Add optional support for more powerful templates with conditions and loops (Smarty). (In the case of CiviMail, this was still disabled by default as a security consideration, but in other use-cases it might be enabled by default.)
+- Add a hook for custom tokens.
+- Expand to other applications, such as individual mailings, print letters, receipts for contributions, and scheduled reminders.
+
+As the use-cases grew, techniques from the original CiviMail code were duplicated and adapted, leading to a lengthy idiom which looks a bit like this:
+
+```php
+$subject = $template->subject;
+$body_html = $template->body_html;
+$body_text = $template->body_text;
+ 
+$subject = CRM_Utils_Token::replaceFooTokens($subject, $fooData... $encodingOptions...);
+$subject = CRM_Utils_Token::replaceBarTokens($subject, $barData... $encodingOptions...);
+$subject = CRM_Utils_Token::replaceHookTokens($subject, $encodingOptions...);
+if (smarty enabled) $subject = $smarty->display("string:$subject");
+ 
+$body_html = CRM_Utils_Token::replaceFooTokens($body_html, $fooData... $encodingOptions...);
+$body_html = CRM_Utils_Token::replaceBarTokens($body_html, $barData... $encodingOptions...);
+$body_html = CRM_Utils_Token::replaceHookTokens($body_html, $encodingOptions...);
+if (smarty enabled) $body_html = $smarty->display("string:$body_html");
+ 
+$body_text = CRM_Utils_Token::replaceFooTokens($body_text, $fooData... $encodingOptions...);
+$body_text = CRM_Utils_Token::replaceBarTokens($body_text, $barData... $encodingOptions...);
+$body_text = CRM_Utils_Token::replaceHookTokens($body_text, $encodingOptions...);
+if (smarty enabled) $body_text = $smarty->display("string:$body_text");
+```
+
+Some of the key functions of this system are:
 
 - `CRM_Utils_Token::getTokens` - Retrieves an array of tokens contained in the given string e.g. HTML of an email
 - `CRM_Utils_Token::getRequiredTokens` - What are the minimum required tokens for CiviMail
 - `CRM_Utils_Token::requiredTokens` - Check that the required tokens are there
 - `CRM_Utils_Token::&replace<type>Tokens` - Replaces x type of Tokens where x is User, Contact, Action, Resubscribe etc
-- `CRM_Utils_Token::get<type>TokenReplcaement` - Format and escape for use in Smarty the found content for Tokens for x type. This is usually called within `CRM_Utils_Token::&replace<type>Tokens`
+- `CRM_Utils_Token::get<type>TokenReplacement` - Format and escape for use in Smarty the found content for Tokens for x type. This is usually called within `CRM_Utils_Token::&replace<type>Tokens`
+
+
+In 4.7+ there was major changes to the Scheduled Reminders facility which also included potentail changes to CiviMail in so far as how tokens are generated see [CRM-13244](https://issues.civicrm.org/jira/browse/CRM-13244). There is a move to use more of the `Civi\Token\TokenProcessor` sub system as this is more robust. However there have been compatibility layers built in to use the older `CRM_Utils_Token` processors. Developers should aim to work off the `Civi\Token\TokenProcessor` where possible. However there are still some systems that haven't been refactored. Some of the key functions in the older systems are. 
 
-Extension Authors are also able to extend the list of tokens by implement ["hook_civicrm_tokens"](/hooks/hook_civicrm_tokens.md). The content of the custom token needs to be set with ["hook_civicrm_tokenValues"](/hooks/hook_civicrm_tokenValues.md).
+This new system of generating content for tokens has a number of advantages
+- Decreases the number of SQL Queries
+- Is not as tightly coupled with the one templating engine
+
+The basic process in the new subsystem is
+- Whenever an application's controller (e.g. for CiviMail or PDFs or scheduled reminders) needs to work with tokens, it instantiates `Civi\Token\TokenProcessor`.
+- The `controller` passes some information to `TokenProcessor` – namely, the `$context` and the list of `$rows`.
+- The `TokenProcessor` fires an event (`TOKEN_EVALUATE`). Other modules respond with the actual token content.
+- For each of the rows, the controller requests a rendered blob of text.
+
+```php
+$p = new TokenProcessor(Container::singleton()->get('dispatcher'), array(
+  'controller' => __CLASS__,
+  'smarty' => FALSE,
+));
+ 
+// Fill the processor with a batch of data.
+$p->addMessage('body_text', 'Hello {contact.display_name}!', 'text/plain');
+$p->addRow()->context('contact_id', 123);
+$p->addRow()->context('contact_id', 456);
+ 
+// Lookup/compose any tokens which are referenced in the message.
+// e.g. SELECT id, display_name FROM civicrm_contact WHERE id IN (...contextual contact ids...);
+$p->evaluate();
+ 
+// Display mail-merge data.
+foreach ($p->getRows() as $row) {
+  echo $row->render('body_text');
+}
+```
+
+### Entending the Token system
+
+In the old system the standard way extension authors would  extend the list of tokens by implement ["hook_civicrm_tokens"](/hooks/hook_civicrm_tokens.md). The content of the custom token needs to be set with ["hook_civicrm_tokenValues"](/hooks/hook_civicrm_tokenValues.md).
+
+To utilise the newer method extension authors should implement code similar to the following. This is able to be done because when executing `TokenProcessor::evaluate()`, the processor dispatches an event so that other classes may define token content.
+
+```php
+function example_civicrm_container($container) {
+  $container->addResource(new \Symfony\Component\Config\Resource\FileResource(__FILE__));
+  $container->findDefinition('dispatcher')->addMethodCall('addListener', 
+    array(\Civi\Token\Events::TOKEN_REGISTER, 'example_register_tokens')
+  );
+  $container->findDefinition('dispatcher')->addMethodCall('addListener', 
+    array(\Civi\Token\Events::TOKEN_EVALUATE, 'example_evaluate_tokens')
+  );
+}
+ 
+function example_register_tokens(\Civi\Token\Event\TokenRegisterEvent $e) {
+  $e->entity('profile')
+    ->register('viewUrl', ts('Profile View URL'))
+    ->register('viewLink', ts('Profile View Link'));
+}
+ 
+function example_evaluate_tokens(\Civi\Token\Event\TokenValueEvent $e) {
+  foreach ($e->getRows() as $row) {
+    /** @var TokenRow $row */
+    $row->format('text/html');
+    $row->tokens('profile', 'viewUrl', 'http://example.com/profile/' . $row->context['contact_id']);
+    $row->tokens('profile', 'viewLink', ts("<a href='%1'>Open Profile</a>", array(
+      1 => $row->tokens['profile']['viewUrl'],
+    )));
+  }
+}
+```
+
+Some notes on the the above
+
+- `$row->context['...']` returns contextual data, regardless of whether you declared it at the row level or the processor level.
+- To update a row's data, use the `context()` and `tokens()` methods. To read a row's data, use the $context and $tokens properties. These interfaces support several notations, which are described in the TokenRow class.
+- You have control over the loop. You can do individual data-lookups in the loop (for simplicity) – or you can also do prefetches and batched lookups (for performance).
+- The class `\Civi\Token\AbstractTokenSubscriber` provides a more structured/opinionated way to handle these events.
+- For background on the `event dispatcher` (e.g. `listeners` vs subscribers), see [Symphony Documentation](http://symfony.com/doc/current/components/event_dispatcher/introduction.html)
 
 ### Required Tokens
 
@@ -143,7 +253,9 @@ Job `status` can take one of 5 states.
 4.  `Paused`: A job can only be marked paused by the admin interface. The mailer will not act on paused jobs.
 5.  `Canceled`: Like paused, but cannot be placed back in the `Running` state.
 
-## Events
+## Inbound CiviMail Events
+
+Events within CiviMail are usually designed as where CiviMail receieves something back follwoing an email and does som processing
 
 - Delivery
     - Registered after a successful SMTP transaction.
@@ -153,3 +265,19 @@ Job `status` can take one of 5 states.
     - Action:
         - Add a new row in `Mailing_Event_Bounce` with the `queue_id`, `bounce_type` and `bounce_reason` returned by the bounce  processor
         - Count the bounce events for `email_id` and compare with the `hold_threshold` for the matching bounce type. If the email address has more than the threshold of any type of bounce, place it on bounce hold.
+- Unsbuscribe
+  - Registered after either a Successful SMTP transacftion or submission on the usubscribe webform
+  - Action 
+    - Removes the contact from the group leaving a note in the `civicrm_subscription_history` table indicating it was from an email and when it happend
+    - Add a row in `mailing_event_unsbuscribe` setting the `is_domain = 0` for the new row.
+- Opt Out 
+  - Registered after a successful SMTP transaction or on submisssion of the opt out form
+  - Action
+     - Adds a row in `mailing_event_unsubscrive` setting `is_domain = 1`. 
+     - Updaes the `is_opt_out` field to 1 for the contact
+- tracking url
+  - Regisreted when a successfull webrequest is recieved and processed
+  - Action adds a row into `mailing_event_trackable_url` with the current date and the `url_id` that was clicked
+- Reply
+  - Registered when CiviMail successfully processes an SMTP transaction
+
-- 
GitLab