create cli-pull-ot command to sync assessments from OneTrust to disk #375

abrantesarthur · 2025-01-08T18:59:13Z

to test this locally, run yarn ts-node ./src/cli-pull-ot.ts --hostname=app-eu.onetrust.com --auth=$ONE_TRUST_OAUTH_TOKEN --file=./oneTrustAssessment.json, where ONE_TRUST_OAUTH_TOKEN should be the authentication token (you can find it in the parent ticket's description).
for now, this only pulls assessment in json format. After the PR merges, I will update it to support converting to CSV.

Related Issues

closes https://transcend.height.app/T-40816

height · 2025-01-08T18:59:17Z

This pull request has been linked to and will mark 1 task as "Pending Deploy" when merged:

~~T-40816 Build new functionality in the CLI to fetch assessments data from OT API~~ (unlink task)

💡Tip: You can link multiple Height tasks to a pull request.

anotherminh

@abrantesarthur can you make sure to update the version in package.json and also add README updates for this command? 🙏

abrantesarthur · 2025-01-08T22:25:12Z

@abrantesarthur can you make sure to update the version in package.json and also add README updates for this command? 🙏

for sure!

abrantesarthur · 2025-01-08T23:02:43Z

@abrantesarthur can you make sure to update the version in package.json and also add README updates for this command? 🙏

for sure!

done

lvonessen

Overall looks good, I think I get what's going on!

lvonessen · 2025-01-08T21:19:42Z

src/oneTrust/types.ts

+  /** Overall risk score after considering existing controls. */
+  residualRiskScore: number;
+  /** Result of the assessment. NOTE: This field will be deprecated soon. Please reference the 'resultName' field instead. */
+  result: 'Approved' | 'AutoClosed' | 'Rejected';


The linked documentation says these are the appropriate value for result, but the example response provided there looks like it has result and resultId the same, rather than result and resultName. Not sure if that makes a difference to us.

"result": "b1b0a831-b7d7-410e-b3e5-4a3e5e141851", "resultId": "b1b0a831-b7d7-410e-b3e5-4a3e5e141851", "resultName": "Pending",

(I didn't check the rest of this file closely, I was just curious how you'd gotten the info about which values a field might be limited to and that detail happened to pop out to me)

Yeah, checking the actual response that I got, I see that sometimes "result" = "resultId", other times "result" = "resultName", other times "result" = "Approved" but "resultName" = "Approved - Remediation required". In general, since their docs say that "result" will be deprecated, I will just trust "resultId" + "resultName" for purposes of importing into Transcend. Thanks for the heads up!

lvonessen · 2025-01-08T23:08:47Z

README.md

@@ -571,6 +573,43 @@ tr-pull --auth=./transcend-api-keys.json --resources=consentManager --file=./tra

 Note: This command will overwrite the existing transcend.yml file that you have locally.

+### tr-pull-ot
+
+Pulls resources from a OneTrust instance. For now, it only supports retrieving OneTrust Assessments. It sends a request to the (Get List of Assessments)[https://developer.onetrust.com/onetrust/reference/getallassessmentbasicdetailsusingget] endpoint to fetch a list of all Assessments in your account. Then, it queries the (Get Assessment)[https://developer.onetrust.com/onetrust/reference/exportassessmentusingget] and (Get Risk)[https://developer.onetrust.com/onetrust/reference/getriskusingget] endpoints to enrich these assessments with more details such as respondents, approvers, assessment questions and responses, and assessment risks. Finally, it syncs the enriched resources to disk in the specified file and format.


I think the link syntax here is reversed--I think we want [link text](link address)

Good point!

michaelfarrell76

good stuff!

michaelfarrell76 · 2025-01-09T19:41:45Z

README.md

+
+```sh
+# Writes out file to ./oneTrustAssessments.json
+tr-pull-ot --auth=$ONE_TRUST_OAUTH_TOKEN --hostname=trial.onetrust.com --file=./oneTrustAssessments.json


what do you do with the file after its pulled?

For now we do nothing, but I am going to create another command that accepts the file path as input and then pushes it to Transcend via the importOneTrustAssessmentForms mutation.

got it, maybe use the same command but have a --dryRun parameter or something?

or have it dump into the transcend.yml shape and use tr-push to push up (would need to add assessment support to that)

That could work as well! So if dryRun = true we push to transcend. Otherwise, we save it as a file?

was thinking opposite, default behavior is to pull from OT and push to transcend. probaly would make sense to rename to tr-sync-ot

then if you specify --dryRun=true this would only pull from OT and write to file instead of syncing to transcend

makes sense! I'll do that.

michaelfarrell76 · 2025-01-09T19:42:08Z

src/cli-pull-ot.ts

+    parseCliPullOtArguments();
+
+  try {
+    if (resource === OneTrustPullResource.Assessments) {


nit: best to move these into helper functions so that they can be imported as code if needed

will do in a follow up PR!

michaelfarrell76 · 2025-01-09T19:42:29Z

src/oneTrust/getAssessment.ts

+    `api/assessment/v2/assessments/${assessmentId}/export?ExcludeSkippedQuestions=false`,
+  );
+
+  return JSON.parse(body) as OneTrustGetAssessmentResponse;


nit: should use an io-ts codec to validate here

Created the codecs here https://github.com/transcend-io/privacy-types/pull/205/files and battle-tested them agains the OneTrust API responses. Caught lots of stuff! Thanks for the suggestion.

michaelfarrell76 · 2025-01-09T19:42:45Z

src/oneTrust/getListOfAssessments.ts

+  logger.info('Getting list of all assessments from OneTrust...');
+  while (currentPage < totalPages) {
+    // eslint-disable-next-line no-await-in-loop
+    const { body } = await oneTrust.get(


nit sjould use io-ts codec validation here

will do in a follow up PR as well!

abrantesarthur added 9 commits January 8, 2025 18:46

create cli-pull-ot file

d5c4c11

modify cli-pull-ot

c91e6c6

create createOneTrustGotInstance helper

ce3b7dc

create helpers

d1ece52

create getAssessment helper

0a35092

create index.ts

f29068a

update enums

8233efd

update package.json

a398525

fix bug in write

1f0c2cc

abrantesarthur requested review from lvonessen, michaelfarrell76, anotherminh and a team January 8, 2025 19:00

fix compilation

8abbc4a

anotherminh approved these changes Jan 8, 2025

View reviewed changes

anotherminh reviewed Jan 8, 2025

View reviewed changes

abrantesarthur added 2 commits January 8, 2025 22:26

update package version

1aa6857

update README.md

1a3090e

lvonessen approved these changes Jan 8, 2025

View reviewed changes

add default debug argument

99e8d82

lvonessen reviewed Jan 8, 2025

View reviewed changes

abrantesarthur added 6 commits January 8, 2025 23:09

update parseCliPullOtArguments comment

25a73a8

fix readme

52c8e7a

nit

5e1dc0d

improveOneTrust types based on responses

2f99e72

improve types

1dcc90e

more types improvements

a811105

abrantesarthur merged commit df0943d into main Jan 9, 2025
10 checks passed

abrantesarthur deleted the pullOt branch January 9, 2025 16:13

michaelfarrell76 reviewed Jan 9, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

create cli-pull-ot command to sync assessments from OneTrust to disk #375

create cli-pull-ot command to sync assessments from OneTrust to disk #375

abrantesarthur commented Jan 8, 2025 •

edited

Loading

height bot commented Jan 8, 2025 •

edited

Loading

anotherminh left a comment

abrantesarthur commented Jan 8, 2025

abrantesarthur commented Jan 8, 2025

lvonessen left a comment

lvonessen Jan 8, 2025

abrantesarthur Jan 8, 2025

lvonessen Jan 8, 2025

abrantesarthur Jan 8, 2025

abrantesarthur Jan 8, 2025

michaelfarrell76 left a comment

michaelfarrell76 Jan 9, 2025

abrantesarthur Jan 13, 2025

michaelfarrell76 Jan 13, 2025

michaelfarrell76 Jan 13, 2025

abrantesarthur Jan 13, 2025

michaelfarrell76 Jan 13, 2025

abrantesarthur Jan 13, 2025

michaelfarrell76 Jan 9, 2025

abrantesarthur Jan 13, 2025

michaelfarrell76 Jan 9, 2025

abrantesarthur Jan 13, 2025

michaelfarrell76 Jan 9, 2025

abrantesarthur Jan 13, 2025

create cli-pull-ot command to sync assessments from OneTrust to disk #375

create cli-pull-ot command to sync assessments from OneTrust to disk #375

Conversation

abrantesarthur commented Jan 8, 2025 • edited Loading

Related Issues

height bot commented Jan 8, 2025 • edited Loading

anotherminh left a comment

Choose a reason for hiding this comment

abrantesarthur commented Jan 8, 2025

abrantesarthur commented Jan 8, 2025

lvonessen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelfarrell76 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abrantesarthur commented Jan 8, 2025 •

edited

Loading

height bot commented Jan 8, 2025 •

edited

Loading