From 4ac44384eb1c0334fe3c4207e7702316a11b8d84 Mon Sep 17 00:00:00 2001 From: Paul Bugni Date: Thu, 9 Jan 2025 12:04:07 -0800 Subject: [PATCH 1/8] initial cut at schema compliant with FHIR AuditEvent --- README.md | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5b0cecb..c16939a 100644 --- a/README.md +++ b/README.md @@ -3,10 +3,58 @@ API and storage for logging event and audit messages, persisted in JSON within a PostgreSQL db, with a thin frontend provided by [PostgREST](http://postgrest.org/en/v7.0.0/index.html) +The logserver serves as the **Audit Record Repository**, as detailed in the +Basic Audit Log Patterns +[(BALP)](https://profiles.ihe.net/ITI/BALP/volume-1.html#1-52-basic-audit-log-patterns) +implementation guide. + ## Event Schema **logserver** is agnostic to the format, provided it's valid JSON. Any number -of database tables can be used, but only the single **"events"** table is built -in, containing a PostgreSQL JSONB column, **"event"**. +of database tables can be used, but only the single `events` table is built +in, containing a PostgreSQL JSONB column, `event`. + +It is desirable to generate log events complaint with the [FHIR audit event]( +https://www.hl7.org/fhir/auditevent.html) resource. To generate an AuditEvent +resource nested within each `event` is cumbersome, however the following field +parity is recommended: + +- `category`: major type of the event, such as: + - `authentication`: Events related to login or authentication. + - `authorization`: Events related to access control or permissions changes. + - `security`: General security-related events. + - `data-access`: Events where healthcare data is accessed or modified. + - `configuration`: Events involving system or configuration changes. +- `code`: specific type of event. See [audit-event-sub-type valueset]( + https://www.hl7.org/fhir/valueset-audit-event-sub-type.html) for full list. + - `login` + - `create` + - `read` + - `update` + - `delete` + - `search` +- `action`: required element to describe the type of operation performed. + - `C`: create - Creating a new resource, such as adding a patient. + - `R`: read/view/search - Data retrieved or viewed w/o modification. + - `U`: update - indicates existing data was modified. + - `D`: delete - indicates data was removed or deleted. + - `E`: execute - indicates execution of operation or procedure. +- `occurred`: defined only when reliance on message timestamp is inadequate +- `patient`: the **subject** of the activity, i.e. `Patient/ab-123-ef` +- `agent`: actor involved in the event, i.e. `Practitioner/123-abc` +- `souce`: event reporter or system generating the audit event +- `entity`: data or objects used + - `detail`: tagged value pairs for conveying additional information + - `query`: query parameters for query-type entities +- `outcome`: result of event. + - `code` + - `fatal` + - `error` + - `warning` + - `information` + - `success` + - `detail`: additional outcome detail + +## Historical Schema Information Below The following suggestions for the format of each "event" entry enable common query syntax and meet expectations. From de6162f6665a3cd0e90b66d24740d12cda9f4e91 Mon Sep 17 00:00:00 2001 From: Paul Bugni Date: Tue, 14 Jan 2025 16:34:17 -0800 Subject: [PATCH 2/8] minor edits and cleanup; `source` to include version info --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index c16939a..4173808 100644 --- a/README.md +++ b/README.md @@ -19,11 +19,11 @@ resource nested within each `event` is cumbersome, however the following field parity is recommended: - `category`: major type of the event, such as: - - `authentication`: Events related to login or authentication. - - `authorization`: Events related to access control or permissions changes. - - `security`: General security-related events. - - `data-access`: Events where healthcare data is accessed or modified. - - `configuration`: Events involving system or configuration changes. + - `authentication`: events related to login or authentication. + - `authorization`: events related to access control or permissions changes. + - `security`: general security-related events. + - `data-access`: events where healthcare data is accessed or modified. + - `configuration`: events involving system or configuration changes. - `code`: specific type of event. See [audit-event-sub-type valueset]( https://www.hl7.org/fhir/valueset-audit-event-sub-type.html) for full list. - `login` @@ -33,15 +33,15 @@ parity is recommended: - `delete` - `search` - `action`: required element to describe the type of operation performed. - - `C`: create - Creating a new resource, such as adding a patient. - - `R`: read/view/search - Data retrieved or viewed w/o modification. + - `C`: create - creating a new resource, such as adding a patient. + - `R`: read/view/search - data retrieved or viewed w/o modification. - `U`: update - indicates existing data was modified. - `D`: delete - indicates data was removed or deleted. - `E`: execute - indicates execution of operation or procedure. - `occurred`: defined only when reliance on message timestamp is inadequate - `patient`: the **subject** of the activity, i.e. `Patient/ab-123-ef` - `agent`: actor involved in the event, i.e. `Practitioner/123-abc` -- `souce`: event reporter or system generating the audit event +- `source`: event reporter or system generating the audit event including version - `entity`: data or objects used - `detail`: tagged value pairs for conveying additional information - `query`: query parameters for query-type entities From a59fe98f05b16337188994baa0c7db70b4b9ffab Mon Sep 17 00:00:00 2001 From: Paul Bugni Date: Mon, 27 Jan 2025 16:49:07 -0800 Subject: [PATCH 3/8] Clearly marked required fields and some minor text improvements. --- README.md | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 4173808..513af17 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ Basic Audit Log Patterns implementation guide. ## Event Schema -**logserver** is agnostic to the format, provided it's valid JSON. Any number +**logserver** is agnostic to the format, provided it is valid JSON. Any number of database tables can be used, but only the single `events` table is built in, containing a PostgreSQL JSONB column, `event`. @@ -18,6 +18,26 @@ https://www.hl7.org/fhir/auditevent.html) resource. To generate an AuditEvent resource nested within each `event` is cumbersome, however the following field parity is recommended: +NB: all fields are considered optional unless marked as **required** + +- `severity`: **required** element. Use built in log level rather than adding + an additional `severity` field: + - `critical`: Critical condition with application. Include `emergency` and + `alert` levels in this category. + - `error`: Error condition with the application. + - `warning`: Warning needing attention before escalation to error. + - `info`: Normal operational messages not requiring action. Include + `notice` level in this category (normal but significant). + - `debug`: Debug level messages, useful to application developers. +- `action`: **required** element to describe the type of operation performed. + - `C`: create - creating a new resource, such as adding a patient. + - `R`: read/view/search - data retrieved or viewed w/o modification. + - `U`: update - indicates existing data was modified. + - `D`: delete - indicates data was removed or deleted. + - `E`: execute - system or application function such as log-on, program + execution or perform a query/search. +- `occurred`: Use built in log message time rather than adding an additional + `occurred` field. - `category`: major type of the event, such as: - `authentication`: events related to login or authentication. - `authorization`: events related to access control or permissions changes. @@ -32,20 +52,13 @@ parity is recommended: - `update` - `delete` - `search` -- `action`: required element to describe the type of operation performed. - - `C`: create - creating a new resource, such as adding a patient. - - `R`: read/view/search - data retrieved or viewed w/o modification. - - `U`: update - indicates existing data was modified. - - `D`: delete - indicates data was removed or deleted. - - `E`: execute - indicates execution of operation or procedure. -- `occurred`: defined only when reliance on message timestamp is inadequate - `patient`: the **subject** of the activity, i.e. `Patient/ab-123-ef` - `agent`: actor involved in the event, i.e. `Practitioner/123-abc` - `source`: event reporter or system generating the audit event including version - `entity`: data or objects used - `detail`: tagged value pairs for conveying additional information - `query`: query parameters for query-type entities -- `outcome`: result of event. +- `outcome`: result of event (i.e. success or failure). - `code` - `fatal` - `error` From 9e41764822eec97477e693686027cd894c2ff750 Mon Sep 17 00:00:00 2001 From: Paul Bugni Date: Tue, 28 Jan 2025 16:08:18 -0800 Subject: [PATCH 4/8] Enhancements & simplifications following meeting. --- README.md | 59 +++++++++++++++++++++++-------------------------------- 1 file changed, 25 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index 513af17..121fd8a 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ implementation guide. of database tables can be used, but only the single `events` table is built in, containing a PostgreSQL JSONB column, `event`. -It is desirable to generate log events complaint with the [FHIR audit event]( +It is desirable to generate log events compliant with the [FHIR audit event]( https://www.hl7.org/fhir/auditevent.html) resource. To generate an AuditEvent resource nested within each `event` is cumbersome, however the following field parity is recommended: @@ -29,43 +29,34 @@ NB: all fields are considered optional unless marked as **required** - `info`: Normal operational messages not requiring action. Include `notice` level in this category (normal but significant). - `debug`: Debug level messages, useful to application developers. +- `version`: **required** logserver schema version. - `action`: **required** element to describe the type of operation performed. - - `C`: create - creating a new resource, such as adding a patient. - - `R`: read/view/search - data retrieved or viewed w/o modification. - - `U`: update - indicates existing data was modified. - - `D`: delete - indicates data was removed or deleted. - - `E`: execute - system or application function such as log-on, program - execution or perform a query/search. -- `occurred`: Use built in log message time rather than adding an additional - `occurred` field. -- `category`: major type of the event, such as: - - `authentication`: events related to login or authentication. - - `authorization`: events related to access control or permissions changes. - - `security`: general security-related events. - - `data-access`: events where healthcare data is accessed or modified. - - `configuration`: events involving system or configuration changes. -- `code`: specific type of event. See [audit-event-sub-type valueset]( - https://www.hl7.org/fhir/valueset-audit-event-sub-type.html) for full list. - - `login` - - `create` - - `read` - - `update` - - `delete` - - `search` -- `patient`: the **subject** of the activity, i.e. `Patient/ab-123-ef` -- `agent`: actor involved in the event, i.e. `Practitioner/123-abc` + - `create`: creating a new resource, such as adding a patient. + - `read`: read/view/search - data retrieved or viewed w/o modification. + - `update`: indicates existing data was modified. + - `delete`: indicates data was removed or deleted. + - `execute`: system or application function such as, program execution or + perform a query/search. + - `login`: specific category for the log-in action + - `logout`: specific category for the log-out action +- `occurred`: Date-Time of the event, including timestamp information. This may + duplicate the logging system timestamp (such as `asctime`) but will always + capture the time the event took place, not when it hit the logging server. +- `subject`: the **subject** of the activity, i.e. `Patient/ab-123-ef` +- `agent`: actor involved in the event, generally the logged-in user: + - `ip_address`: end user or requesting system's IP Address + - `type`: i.e. `system` or `user` + - `who`: i.e. `Practitioner/123-abc` - `source`: event reporter or system generating the audit event including version + - `observer`: base URL of the system generating the audit message + - `type`: system type such as `dhair2` or other predefined project category. + - `version`: version of the observer (in contrast to top level `version`) - `entity`: data or objects used - - `detail`: tagged value pairs for conveying additional information + - `detail`: list of tagged value pairs for conveying additional information. + example pair might include `url`: `` in contrast to `source.observer` - `query`: query parameters for query-type entities -- `outcome`: result of event (i.e. success or failure). - - `code` - - `fatal` - - `error` - - `warning` - - `information` - - `success` - - `detail`: additional outcome detail +- `outcome`: details in event of a failure or warning. use `severity` to capture + level. ## Historical Schema Information Below From 6bc7b47e7a2ab7b22e69c44c4cc12fa5ae004535 Mon Sep 17 00:00:00 2001 From: Paul Bugni Date: Tue, 28 Jan 2025 16:12:04 -0800 Subject: [PATCH 5/8] remove outdated schema information --- README.md | 58 +------------------------------------------------------ 1 file changed, 1 insertion(+), 57 deletions(-) diff --git a/README.md b/README.md index 121fd8a..0fbeb9f 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Basic Audit Log Patterns [(BALP)](https://profiles.ihe.net/ITI/BALP/volume-1.html#1-52-basic-audit-log-patterns) implementation guide. -## Event Schema +## Event Schema (version 3.0) **logserver** is agnostic to the format, provided it is valid JSON. Any number of database tables can be used, but only the single `events` table is built in, containing a PostgreSQL JSONB column, `event`. @@ -58,62 +58,6 @@ NB: all fields are considered optional unless marked as **required** - `outcome`: details in event of a failure or warning. use `severity` to capture level. -## Historical Schema Information Below - -The following suggestions for the format of each "event" entry enable common -query syntax and meet expectations. - -A distinction is made between "top level" attributes, and those nested under -"message". The "message" may be a simple text string, or any level of valid -JSON data, intended to capture the intent of the event, such as "new consent -signed" or "search for <...> found 0 matches". Message generally captures -the specific context from the code of the event being tracked, with all other -details collected by a routine that can collect and populate the other top -level attributes as specified below. - -Since we anticipate that events will undergo automated processing and there is only a single 'message' attribute, it seems likely that a message will have several json attributes, rather than be a simple string. But, either is legal. - -The following should be common to all events on a given system: -```json -{ - "event_version": "1", // the event schema version - "asctime": "", // ISO-8601 format including time-zone offset - "name": "", // Application code package name, often built in to the logging system and difficult to manipulate - "level": "INFO", // Built in to the logging package, options also include DEBUG, WARN, ERROR -``` - -System identifiers to uniquely specify the source of the event: -```json - "clinical-site": "", // unique name when appropriate to define jurisdiction, institution or clinic, such as "UW Harborview", - "deployment": "", // one of ["dev", "test", "demo", "stage", "prod"] - "system-type": "", // such as "remote" or "kiosk", if applicable - "system-url": "", // system identifier URL -``` - -Authenticated user, or string identifier for system run jobs, etc. -```json - "user": "User/1", // alternative nested JSON with attributes is fine; ideally consistent per application -``` - -If acting on an identifiable entity "subject": -```json - "subject": "Patient/12", -``` - -List of topics (effectively a message "type" or "reason") useful for filtering: -```json - "tags": ["patient", "launch", "logout", "search"], // one or more tags -``` - -And finally, and details in the message itself, that aren't captured above, -nesting any valid JSON within message if appropriate. The details captured -in the "message" often come from deep in the application stack, where all of -the above isn't so easily obtained. -```json - "message": "Description of action" // replace string with nested JSON when applicable -} -``` - ### Example event schemas in use for the respective projects: * [COSRI - version 0](./docs/cosri_v0.md) From 7f197cbd290e6e0cd162b26ab035e0dfef671fd0 Mon Sep 17 00:00:00 2001 From: Paul Bugni Date: Tue, 28 Jan 2025 16:28:57 -0800 Subject: [PATCH 6/8] Remove redundant clause. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0fbeb9f..2c71bd9 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ NB: all fields are considered optional unless marked as **required** - `ip_address`: end user or requesting system's IP Address - `type`: i.e. `system` or `user` - `who`: i.e. `Practitioner/123-abc` -- `source`: event reporter or system generating the audit event including version +- `source`: event reporter or system generating the audit event. - `observer`: base URL of the system generating the audit message - `type`: system type such as `dhair2` or other predefined project category. - `version`: version of the observer (in contrast to top level `version`) From 0289d1fa4013e61419567734f4532ff6cd76beec Mon Sep 17 00:00:00 2001 From: Paul Bugni Date: Thu, 6 Feb 2025 16:14:48 -0800 Subject: [PATCH 7/8] include example implementation of v3 schema for dhair2 --- README.md | 3 ++- docs/dhair2_v3.md | 29 +++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) create mode 100644 docs/dhair2_v3.md diff --git a/README.md b/README.md index 2c71bd9..9f13a3c 100644 --- a/README.md +++ b/README.md @@ -60,8 +60,9 @@ NB: all fields are considered optional unless marked as **required** ### Example event schemas in use for the respective projects: -* [COSRI - version 0](./docs/cosri_v0.md) +* [dhair2 - version 3](./docs/dhair2_v3.md) * [COSRI - version 1](./docs/cosri_v1.md) +* [COSRI - version 0](./docs/cosri_v0.md) ## Config Copy ``default.env`` to ``.env`` and edit. Don't quote strings! diff --git a/docs/dhair2_v3.md b/docs/dhair2_v3.md new file mode 100644 index 0000000..4a23aee --- /dev/null +++ b/docs/dhair2_v3.md @@ -0,0 +1,29 @@ +# Example Message from dhair2 +```json +{ + "level": "info", + "version": "3.0", + "source": { + "observer": "https://ubu.mcjustin.dev.cirg.uw.edu", + "type": "dhair2/inform" + }, + "occurred": "2025-01-31T00:44:18Z", + "agent": { + "type": "user", + "who": "607", + "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0", + "ip_address": "10.102.17.250" + }, + "subject": [ + "patients/607" + ], + "entity": { + "detail": { + "controller": "Answer", + "action": "afterSave created" + }, + "query": "question_id = 2320; state = ; body_text = 6; option_id = 8040; value = ; iteration = 0; survey_session_id = 78900258;" + }, + "action": "create" +} +``` From e7d0a49727b47c91ed9a4062910449f0c5b5710f Mon Sep 17 00:00:00 2001 From: Paul Bugni Date: Thu, 6 Feb 2025 16:23:30 -0800 Subject: [PATCH 8/8] adding overlooked example dhair2 v3 audit entries --- docs/dhair2_v3.md | 65 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 64 insertions(+), 1 deletion(-) diff --git a/docs/dhair2_v3.md b/docs/dhair2_v3.md index 4a23aee..4fc6cf7 100644 --- a/docs/dhair2_v3.md +++ b/docs/dhair2_v3.md @@ -1,4 +1,6 @@ -# Example Message from dhair2 +# Example Audit Entries from dhair2 + +## Creating an answers record: ```json { "level": "info", @@ -27,3 +29,64 @@ "action": "create" } ``` + +## View survey page: +```json +{ + "level": "info", + "version": "3.0", + "source": { + "observer": "https://ubu.mcjustin.dev.cirg.uw.edu", + "type": "dhair2/inform" + }, + "occurred": "2025-01-31T00:47:17Z", + "agent": { + "type": "user", + "who": "607", + "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0", + "ip_address": "10.102.17.250" + }, + "subject": [ + "patients/607" + ], + "entity": { + "detail": { + "controller": "surveys", + "action": "show" + }, + "query": "1680" + }, + "action": "read" +} +``` + +## SurveySession copy: +("restart" for inform; I'll note that this also creates a ton of log entries for copying the session's answer records): +```json +{ + "level": "info", + "version": "3.0", + "source": { + "observer": "https://ubu.mcjustin.dev.cirg.uw.edu", + "type": "dhair2/inform" + }, + "occurred": "2025-01-31T01:49:39Z", + "agent": { + "type": "user", + "who": "607", + "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0", + "ip_address": "10.102.17.250" + }, + "subject": [ + "patients/607" + ], + "entity": { + "detail": { + "controller": "SurveySession", + "action": "afterSave created" + }, + "query": "partial_finalization = ; finished = ; auto_finish = ; user_id = 607; project_id = 3; started = 2025-01-30 00:52:20; patient_id = 607; type = ELECTIVE; appointment_id = ; reportable_datetime = 2025-01-31 01:34:54; external_id = ; page_id_last_interaction = 1705;" + }, + "action": "create" +} +```