added subquery and fixed value field spec

CHANGELOG.md

@@ -5,6 +5,8 @@
 * Added date filter to filter date with the PHP Date Format.
 * Added filtering on Contact (sub) type on the contact filter.
 * Added PDF Export Output
+* Added Union Query Data Flow.
+* Added a field specification for a fixed value
 
 # Version 1.3.0
 

Civi/DataProcessor/DataFlow/CombinedDataFlow/UnionQueryDataFlow.php

+<?php
+/**
+ * @author Jaap Jansma <jaap.jansma@civicoop.org>
+ * @license AGPL-3.0
+ */
+
+namespace Civi\DataProcessor\DataFlow\CombinedDataFlow;
+
+use Civi\DataProcessor\DataFlow\EndOfFlowException;
+use Civi\DataProcessor\DataFlow\InvalidFlowException;
+use Civi\DataProcessor\DataFlow\MultipleDataFlows\DataFlowDescription;
+use Civi\DataProcessor\DataFlow\MultipleDataFlows\MultipleSourceDataFlows;
+use Civi\DataProcessor\DataFlow\SqlDataFlow;
+use Civi\DataProcessor\DataFlow\SqlTableDataFlow;
+use Civi\DataProcessor\DataSpecification\DataSpecification;
+use Civi\DataProcessor\DataSpecification\SqlFieldSpecification;
+
+class UnionQueryDataFlow extends SqlDataFlow implements MultipleSourceDataFlows {
+
+  /**
+   * @var \Civi\DataProcessor\DataFlow\MultipleDataFlows\DataFlowDescription[]
+   */
+  protected $sourceDataFlowDescriptions = array();
+
+  /**
+   * @var String
+   */
+  protected $name;
+
+  public function __construct($name) {
+    parent::__construct();
+    $this->name = $name;
+  }
+
+  /**
+   * Adds a source data flow
+   *
+   * @param \Civi\DataProcessor\DataFlow\MultipleDataFlows\DataFlowDescription $dataFlowDescription
+   * @return void
+   * @throws \Civi\DataProcessor\DataFlow\InvalidFlowException
+   */
+  public function addSourceDataFlow(DataFlowDescription $dataFlowDescription) {
+    if (!$dataFlowDescription->getDataFlow() instanceof SqlDataFlow) {
+      throw new InvalidFlowException();
+    }
+    $this->sourceDataFlowDescriptions[] = $dataFlowDescription;
+  }
+
+  public function getName() {
+    return $this->name;
+  }
+
+  /**
+   * Returns the From Statement.
+   *
+   * @return string
+   */
+  public function getFromStatement() {
+    return "FROM {$this->getTableStatement()}";
+  }
+
+  /**
+   * Returns the Table part in the from statement.
+   *
+   * @return string
+   */
+  public function getTableStatement() {
+    $selectStatements = array();
+    foreach($this->sourceDataFlowDescriptions as $sourceDataFlowDescription) {
+      $sourceDataFlow = $sourceDataFlowDescription->getDataFlow();
+      if ($sourceDataFlow instanceof SqlDataFlow) {
+        $selectAndFrom = $sourceDataFlow->getSelectQueryStatement();
+        $where = $sourceDataFlow->getWhereStatement();
+        $groupBy = $sourceDataFlow->getGroupByStatement();
+        $selectStatements[] = "{$selectAndFrom} {$where} {$groupBy}";
+      }
+    }
+
+    $sql = implode(" UNION ", $selectStatements);
+
+    return "({$sql}) `{$this->getName()}`";
+  }
+
+  /**
+   * Returns an array with the fields for in the select statement in the sql query.
+   *
+   * @return string[]
+   * @throws \Civi\DataProcessor\DataSpecification\FieldExistsException
+   */
+  public function getFieldsForSelectStatement() {
+    $fields = array();
+    foreach($this->getDataSpecification()->getFields() as $field) {
+      if ($field instanceof SqlFieldSpecification) {
+        $fields[] = $field->getSqlSelectStatement($this->getName());
+      } else {
+        $fields[] = "`{$this->getName()}`.`{$field->name}` AS `{$field->alias}`";
+      }
+    }
+    return $fields;
+  }
+
+}

Civi/DataProcessor/DataSpecification/FixedValueSqlFieldSpecification.php

+<?php
+/**
+ * @author Jaap Jansma <jaap.jansma@civicoop.org>
+ * @license AGPL-3.0
+ */
+
+namespace Civi\DataProcessor\DataSpecification;
+
+class FixedValueSqlFieldSpecification extends FieldSpecification {
+
+  /**
+   * @var String
+   */
+  public $value;
+
+  public function __construct($value, $alias, $type, $title) {
+    $this->name = $alias;
+    $this->alias = $alias;
+    $this->type = $type;
+    $this->title = $title;
+    if ($value) {
+      $this->value = \CRM_Utils_Type::escape($value, $type);
+    } else {
+      $this->value = "NULL";
+    }
+  }
+
+  /**
+   * Returns the select statement for this field.
+   * E.g. COUNT(civicrm_contact.id) AS contact_id_count
+   *
+   * @param String $table_alias
+   *
+   * @return String
+   */
+  public function getSqlSelectStatement($table_alias) {
+    return "{$this->value} as `{$this->alias}`";
+  }
+
+  /**
+   * Returns the group by statement for this field.
+   * E.g. civicrm_contribution.financial_type_id
+   * or MONTH(civicrm_contribution.receive_date)
+   *
+   * @param String $table_alias
+   *
+   * @return String
+   */
+  public function getSqlGroupByStatement($table_alias) {
+    return "`{$this->alias}`";
+  }
+
+  /**
+   * Returns the SQL column name for this field.
+   * This could be used in join statements
+   *
+   * @param $table_alias
+   *
+   * @return string
+   */
+  public function getSqlColumnName($table_alias) {
+    return "`{$this->alias}`";
+  }
+
+
+}

README.md

@@ -3,7 +3,7 @@
 The data processor is an extension with which system administrator can do the following:
 
 * Create custom searches with optional the possibility to export the results
-* Create an API to fetch data. This is quite useful if you have external systems which need data from CiviCRM. 
+* Create an API to fetch data. This is quite useful if you have external systems which need data from CiviCRM.
 * ... and developers can enhance the outputs of the data processor so that much more is possible, even things we haven't thought of yet.
 
 Below a screenshot of the configuration screen
@@ -12,7 +12,7 @@ Below a screenshot of the configuration screen
 
 And below a screenshot of the above data processor in action as a custom search:
 
-![Screen of data processor in action as a search](docs/images/dataprocessor_2.png) 
+![Screen of data processor in action as a search](docs/images/dataprocessor_2.png)
 
 The extension is licensed under [AGPL-3.0](LICENSE.txt).
 
@@ -35,6 +35,7 @@ The extension is licensed under [AGPL-3.0](LICENSE.txt).
 * [Add your own data source for a CiviCRM Entity](docs/add_your_own_datasource.md)
 * Add your own data source for a CSV File
 * How to store a data processor in code in your extension
+* [Data Flow Classes](docs/dev/DataFlowClasses.md) - Overview of all the available data flow classes.
 * [Develop PHPUnit TestCase for the extension](docs/how_to_create_test.md)
 
 ## Installation
@@ -43,7 +44,7 @@ Install this extension by downloading it from https://lab.civicrm.org/extensions
 and then upload it to your civicrm server in the extension folder.
 And then press install in the Administer -->  System Settings --> Extensions screen.
 
-## Optional 
+## Optional
 
 To get more information about each fields in DataProcessor. A CiviTutorial has been created, to view tutorial install CiviTutorial Extension.
 

docs/dev/DataFlowClasses.md

+# Data Flow Classes
+
+A Data Flow Class determines how the data flows through the data processor. Most of the time this would be an SQL Query.
+
+The following data flow classes are available:
+
+* `InMemoryDataFlow` - Use this to have the data in memory. E.g. when your data comes from a CSV file you can use this
+  data flow to hold all data.
+* `CsvDataFlow` - Data flow for loading data from a CSV file.
+* `CombinedDataFlow` - Combine multiple data flows into one.
+* `SqlTableDataFlow` - Use this data flow to retrieve data from a database table.
+* `CombinedSqlDataFlow` - Use this data flow to create an sql query with a join on multiple tables.
+* `SubqueryDataFlow` - Use this data flow to create a query based on a sub query.
+* `UnionQueryDataFlow` - Use this data flow to create a sub query with an Union statement.
+
+Abstract classes for data flow:
+
+* `AbstractDataFlow` - This is the base class for the data flow. All Data Flows should inherit this class.
+* `SqlDataFlow` - This is the base class for all the SQL based data flows.
+
+## Data Flow Classes: CombinedDataFlow, CombinedSqlDataFlow and SubQueryDataFlow
+
+When you use a class which combines multiple data flows. Such as the `CombinedSqlDataFlow` then you add the other data flows by
+calling `addSourceDataFlow` which expects a `DataFlowDescription` class. The `DataFlowDescription` describes the data flow and
+how the data flow should be joined to the other data flows.
+
+## Data Flow Classes: UnionQueryDataFlow
+
+This class generates a query like
+
+```sql
+    SELECT id, display_name FROM civicrm_contact WHERE contact_type = 'Individual'
+    UNION
+    SELECT id, display_name FROM civicrm_contact WHERE contact_type = 'Household'
+```
+
+When you use the `UnionQueryDataFlow` class you can add additional query by calling the method `addSourceDataFlow`. This
+method expects a `DataFlowDescription` which will then hold the `DataFlow` for the next select statement.
+Each added data flow should hold the same number of columns in the same order.
+