[IT-3230] Move agora data manager CI

.github/workflows/main.yaml

@@ -6,20 +6,14 @@ on:
 
 jobs:
   deploy:
-    # self hosted runners are setup in github to match branch names
+    # self hosted runner labels are setup in github to match branch names
     runs-on: [self-hosted, ${{ github.ref_name }}]
-    # context environment variables are setup in github to match branch names
+    # variables in context environments are setup in github to match branch names
     environment:
       name: ${{ github.ref_name }}
 
     steps:
       - uses: actions/checkout@v4
-      - uses: BSFishy/pip-action@v1
-        with:
-          packages: |
-            synapseclient[pandas,pysftp]
-      - name: Setup mongo tools - mongo-tools and mongo-shell
-        uses: boly38/action-mongo-tools@stable
       - name: Import Synapse Data
         run: ./import-data.sh $BRANCH $SYNAPSE_USERNAME $SYNAPSE_PASSWORD $DB_HOST $DB_USER $DB_PASS
         env:

README.md

@@ -1,5 +1,6 @@
 # Overview
-Agora Data Manager is a tool that loads the JSON files into Agora's document database instances in our AWS environments.
+Agora Data Manager is a tool that loads the JSON files into Agora's document database
+instances in our AWS environments.
 
 # Purpose
 This project allows Agora maintainers to update the Agora database with
@@ -55,17 +56,19 @@ Context specific secrets for each environment that corresponds to a git branch (
 
 ## Self hosted runners
 
-We setup a bastian host in AWS for each environment which have access to the
-databases.  Then we associate those bastian hosts to
-[Github self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners#about-self-hosted-runners).
-The update is executed from these runners.  Each runner corresponds
-to an environment which  corresponds to a git branch.  When a push happens on a
-branch (i.e. develop), the update is executed on the `agora-bastian-develop` runner
-which in turn updates the development database.
+[agora2-infra] repository deploys a bastian host in AWS for each environment which have access to
+the databases.  We manually configure a [Github self-hosted runner] for each bastian host,
+a label is applied to each runner to match the corresponding deployment branch name (develop/staging/prod).
+Each runner corresponds to an environment which corresponds to a git branch. The update is
+executed from these runners.  When a push happens on a branch (i.e. develop), the update
+is executed on the `agora-bastian-develop` runner which in turn updates the development database.
+
 
 ![alt text][self_hosted_runners]
 
 
 [db_update]: agora-db-update.drawio.png "update diagram"
 [github_secrets]: github_secrets.png "github secrets screen"
 [self_hosted_runners]: self-hosted-runners.png "self hosted runners"
+[agora2-infra]: https://github.com/Sage-Bionetworks/agora2-infra "agora2-infra repository"
+[Github self-hosted runners]: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners#about-self-hosted-runners

import-data.sh

@@ -13,12 +13,12 @@ DB_USER=$5
 DB_PASS=$6
 
 CURRENT_DIR=$(pwd)
-PARENT_DIR="$(dirname "$CURRENT_DIR")"
-TMP_DIR=/tmp
-WORKING_DIR=$TMP_DIR/work
+WORKING_DIR=$CURRENT_DIR
 DATA_DIR=$WORKING_DIR/data
 TEAM_IMAGES_DIR=$DATA_DIR/team_images
 
+mkdir -p $TEAM_IMAGES_DIR
+
 # Version key/value should be on his own line
 DATA_VERSION=$(cat $WORKING_DIR/data-manifest.json | grep data-version | head -1 | awk -F: '{ print $2 }' | sed 's/[",]//g' | tr -d '[[:space:]]')
 DATA_MANIFEST_ID=$(cat $WORKING_DIR/data-manifest.json | grep data-manifest-id | head -1 | awk -F: '{ print $2 }' | sed 's/[",]//g' | tr -d '[[:space:]]')