Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
C
Core
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 975
    • Issues 975
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Operations
    • Operations
    • Incidents
  • Analytics
    • Analytics
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • Development
  • Core
  • Issues
  • #1307

Closed
Open
Opened Oct 09, 2019 by eileen@eileen🎱Owner

Add tracking table for import jobs

A current blocker to refactoring the import jobs to use queue processes / non-timeout UI methods is persistence of output data.

We discussed this in Barcelona and came up with the following proposal

  1. create a new table civicrm_user_job with the fields
  • contact_id (or created_id?)
  • job_identifier (or name?) - we probably would need this to be unique but the BAO could handle appending to it if it is not.
  • start_timestamp
  • end_timestamp
  • job_hash - used for file naming
  1. files created as a result of the job could have a standard naming convention - e.g import_{job_hash}validation_errors.csv import{job_hash}_duplicates.csv

  2. We would need a cleanup job which would also remove the files - e.g more than one week old

  3. The UserJob BAO would obviously run pre & post hooks - allowing tracking by extensions

  4. Permissioning of file retrieval can be via Attachment api - Tim thinks we know how that works but he is wrong.

  5. When viewing the results of the import this creates options for us to potentially present a user with their imports (plural) results & for them to choose which they want. (Perhaps exposing the job_idenfier field to them would help here?

@totten @pfigel @seamuslee

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
None
Due date
None
Reference: dev/core#1307