Develop "patch" system in bundle format

[au5ton]

Idea

Develop common format so that most future data additions won't require updating @cougargrades/importer.

"Patch" system would theoretically have individual objects which:

• contain a field like "target" that denotes which Firestore objects should be affected by their absolute document path (ex: catalog/CHEM 1331)
  • archetype: 'document' | 'collection'
  • path: path to what is being referred to
• contain a field like "actions" that denotes which actions should be taken such as
  • write: write the provided object to the Firestore document at the location (will overwrite existing objects) (only for document)
  • merge: merge the provided object with the object at the existing location (will fail if object doesn't exist) (only for document)
  • append: appending to an array-type with a provided amount (any) (only for document)
  • increment: incrementing to a field by a provided amount (number) (only for document)
  • create: write the provided object to a new document in the provided collection (only for collection)

Distinction

Patches are NOT intended to be flexible or programmatic. They should be discrete ways to modify a specific Firestore document in an explicit or "hard-coded" way. Patch files should be used as a medium of interchange and NOT as a method of describing data updates. Ideally, an intelligent tool will have generated a "patch" file that it is certain will be correct.

"Patch" files should be mostly idempotent.

Usage

This "Patch" system would allow the inclusion of supplementary data such as extra fields from various data in this repository. This is especially applicable to the use of:

• edu.uh.core_by_component
  • usage of core component across Firestore
• com.collegescheduler.uh.subjects
  • usage of subjectName across Firestore
• edu.uh.publications.courses
  • linking to the official UH catalog
• possible RateMyProfessor integration in the future 👀

Possible models

write Example

{
  "format": "io.cougargrades.publicdata.patch",
  "target": {
    "archetype": "document",
    "path": "catalog/CHEM 1331"
  },
  "actions": [
    {
      "operation": "write",
      "payload": {
        "department": "CHEM",
        "catalogNumber": "1331",
        "description": "Fundamentals of Chemistry"
      }
    }
  ]
}

merge Example

{
  "format": "io.cougargrades.publicdata.patch",
  "target": {
    "archetype": "document",
    "path": "catalog/CHEM 1331"
  },
  "actions": [
    {
      "operation": "merge",
      "payload": {
        "publicationUrl": "http://publications.uh.edu/preview_course_nopop.php?catoid=34&coid=165599"
      }
    }
  ]
}

append Example

{
  "format": "io.cougargrades.publicdata.patch",
  "target": {
    "archetype": "document",
    "path": "catalog/CHEM 1331"
  },
  "actions": [
    {
      "operation": "append",
      "arrayfield": "instructors",
      "datatype": "firebase.firestore.DocumentReference",
      "payload": "/instructors/Zaitsev, Vladimir G"
    }
  ]
}

increment Example

{
  "format": "io.cougargrades.publicdata.patch",
  "target": {
    "archetype": "document",
    "path": "catalog/CHEM 1331"
  },
  "actions": [
    {
      "operation": "increment",
      "field": "sectionCount",
      "payload": 1
    }
  ]
}

create Example

{
  "format": "io.cougargrades.publicdata.patch",
  "target": {
    "archetype": "collection",
    "path": "users"
  },
  "actions": [
    {
      "operation": "create",
      "payload": {
        "name": "John Doe",
        "score": 9001
      }
    }
  ]
}

[au5ton]

The adoption of this idea would involve generating a io.cougargrades.publicdata.patch/ folder inside the generated bundles which will contain many patch-XXXX.json files to be iterated by @cougargrades/importer in a future update.

[au5ton]

Implemented in c39c3bb

[au5ton]

append operation spec will be updated that allows new property many: boolean to exist which would be present to specify if the payload is an array of datatype or not

[au5ton]

patchfile phase 0 is now denoted as a special phase that runs before traditional CSV records are processed