README.md 8.07 KB
Newer Older
everykittysdaydream's avatar
everykittysdaydream committed
1 2
# com.example.osdi

everykittysdaydream's avatar
everykittysdaydream committed
3 4
This is an extension for importing from and exporting to OSDI endpoints from CiviCRM. We suport syncs for Contact resources between CiviCRM to CiviCRM instances, as well as ActionNetwork to CiviCRM instances. Syncs are updated every day automatically through jobs that take groups of contacts on a remote instance of CiviCRM / ActionNetwork and ensure that these contacts are store and up to date on the host CiviCRM instance. All actions are negotiated through the OSDI implementation, and this extension provides an OSDI API endpoint to access contact instances through an OSDI format. 

everykittysdaydream's avatar
everykittysdaydream committed
5 6 7
For a demo video, look [here](https://drive.google.com/file/d/1Xlcxlr52bE4hvSUPRXw0hT6Qm_zmBUVO/view).

For more information on how to use or extend this application, please check the `docs` folder.
everykittysdaydream's avatar
everykittysdaydream committed
8

everykittysdaydream's avatar
everykittysdaydream committed
9 10
[![Coverage Status](https://coveralls.io/repos/github/4ndygu/civicrm_osdi/badge.svg?branch=master)](https://coveralls.io/github/4ndygu/civicrm_osdi?branch=master)

everykittysdaydream's avatar
everykittysdaydream committed
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43
The extension is licensed under [AGPL-3.0](LICENSE.txt).

## Requirements

* PHP v5.4+
* CiviCRM (*FIXME: Version number*)

## Installation (Web UI)

This extension has not yet been published for installation via the web UI.

## Installation (CLI, Zip)

Sysadmins and developers may download the `.zip` file for this extension and
install it with the command-line tool [cv](https://github.com/civicrm/cv).

```bash
cd <extension-dir>
cv dl com.example.osdi@https://github.com/FIXME/com.example.osdi/archive/master.zip
```

## Installation (CLI, Git)

Sysadmins and developers may clone the [Git](https://en.wikipedia.org/wiki/Git) repo for this extension and
install it with the command-line tool [cv](https://github.com/civicrm/cv).

```bash
git clone https://github.com/FIXME/com.example.osdi.git
cv en osdi
```

## Usage

everykittysdaydream's avatar
everykittysdaydream committed
44
### how to set up sync
everykittysdaydream's avatar
everykittysdaydream committed
45

everykittysdaydream's avatar
everykittysdaydream committed
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68
In order to implement 2-way sync, we have implemented 4 main functions.

- one time import 
- one time export 
- scheduled job import
- scheduled job import

### setting up "one time" jobs

After installing the extension, the configuration page can be spotted at `Contacts -> Import` via OSDI. The config page should look like this:

![config image](https://raw.githubusercontent.com/4ndygu/civicrm_osdi/master/civicrm_osdi_configure.png "config image")

The endpoint and resources can be selected from a dropdown menu. Please offer the APIKey of the appropriate Action Network resource in the API Key field. 

In the Rule ID field, users can supply a dedupe rule ID, and the extension will call the supervised rule at the end of the one time import and export. This is deprecated for now - as a design decision, we chose to dedupe on name and email on insert. As a CiviCRM user, you can call dedupe rules yourself after import / export at the `Contact -> Find and Merge Duplicate Contacts` menu. Please do reach my handle at @everykittysdaydream if you explicitly would like this functionality back. 

In the Group ID rule, you can specify the group you want to export from or the group you want to import into. Please specify the group ID number.

In specify required fields, you can specify a space_delimited string. In Import, this will check if specific fields in Action Network are present in the incoming data. In Export, this will check if specific fields in CiviCRM are present in the incoming data.

#### actually importing

everykittysdaydream's avatar
everykittysdaydream committed
69 70
The import job above only adds the job *to be imported* in a queue that sits in `$_SESSION["extractors"]`. In order to continue with the job, you have to schedule it.

everykittysdaydream's avatar
everykittysdaydream committed
71
To continue scheduling the job and getting contacts into CiviCRM, you need to set up two endpoints: Importer.Schedule and OSDIQueue.Run
everykittysdaydream's avatar
everykittysdaydream committed
72
To do this, you can navigate to the `/civicrm/admin/job?reset=1` endpoint and press "Add New Scheduled Job". There are two jobs to configure. You can name them whatever you like, but make sure the two jobs use the Importer entity + Schedule action, and the OSDIQueue entity and run action, respectively. There are no parameters to these two tasks.
everykittysdaydream's avatar
everykittysdaydream committed
73 74 75

![job image](https://github.com/4ndygu/civicrm_osdi/blob/master/civicrm_job_schedule.png?raw=true "job image")

everykittysdaydream's avatar
everykittysdaydream committed
76 77
You can schedule the job by calling out to the Importer.Schedule endpoint. There are no parameters to be made. In order to set up the import pipeline, you must go to `/civicrm/admin/job` and configure Importer.Schedule as a cron job to be run at an interval of your discretion. This will add all tasks to a queue.

everykittysdaydream's avatar
everykittysdaydream committed
78 79
These jobs can be timed in cron, but can also be tested by pressing `more -> Execute now`.

everykittysdaydream's avatar
everykittysdaydream committed
80 81 82 83 84 85 86 87 88 89
The Scheduler will return information in the following format:

    'email' => [
        'valid' => True or false,
        'new' => True or false,
    ]

Valid determines if the imported contact was valid, or if it contains a first name, last name, and email plus whatever space_delimited OSDI keys that you deem are necessary. New determines if the imported contact is newer than whatever matching contact sits in CiviCRM already. If there is no matching contact, this automatically returns true. The scheduler loads all contacts as separate jobs in a CiviCRM Queue class.

If the Scheduler returns the information "no variable set", that means that the scheduler is empty. You can import more jobs like above. 
everykittysdaydream's avatar
everykittysdaydream committed
90 91 92

After you call schedule, you have to run the tasks in the queue. You can do this by calling out to the OSDIQueue.run endpoint. There are no parameters to be made. In order to set up the import pipeline, you must go to `/civicrm/admin/job` and configure OSDIQueue.run as a cron job to be run at an interval of your discretion. This will run everybody in the queue.

everykittysdaydream's avatar
everykittysdaydream committed
93 94
The OSDIQueue.run job will run *everybody* in the queue. 

everykittysdaydream's avatar
everykittysdaydream committed
95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133
Currently, the architecture is constructed this way because I cannot configure the queue to only run a few elements without throwing a `failed to obtain next task` error. Please do let me know if you have gotten past this!

#### actually exporting 

The first time you press the export button on the configuration page, you are actually just loading up the task by calling the Exporter.bulk endpoint. To actually start exporting via the Person Signup Helper, you must call it more times, preferably in a scheduled job. Every subsequent call of the API with the given key and endpoint will attempt to export 100 contacts. 

I realize as I write this that this is not particularly good UI and will make a note to fix this.

### setting up update jobs

Update jobs exist so they can be called as a scheduled job, presumably every day, that will update all newly modified contacts into both the external endpoint and the CiviCRM instance. 

#### setting up import updates

I have provided an Updater.update endpoint that does the same thing as the original /config field. This is to be set as a scheduled job that runs daily. Here are the parameters:

    key - this is the api key
    endpoint - this is the root of the external api, such as https://actionnetwork.org/api/v2/
    rule - this functions the same as the RuleID from earlier
    group - this is analogous to the groupID from the config page

Like the bulk import, you must call Importer.Schedule and OSDIQueue.run for these jobs to actually end up as CiviCRM contacts. If all jobs are configured via scheduled jobs, the pipeline should work.

#### setting up export updates

To update your exports, you can call Exporter.bulk again with certain parameters. We provide the following extra parameters:

    key - this is the api key
    endpoint - this is the root of the api's people endpoint, such as https://actionnetwork.org/api/v2/people/
    group - this is analogous to the groupID from the config page
    updatejob - set this to 1 if you want to update
    updateendpoint - set this to the endpoint of the external party's person signup helper
    required - this is the same as the space-delimited Required field 

You can set update as a scheduled job that runs daily.

### A generic OSDI-compliant endpoint.

You can find it at `/civicrm/osdi/response?object=contact`. This is currently still in alpha, but it allows users to page through all users with emails, first, and last names. I suppose everything here is in alpha, but this endpoint is currently in even more alpha than the above task.
everykittysdaydream's avatar
everykittysdaydream committed
134 135 136
## Known Issues

(* FIXME *)