[GCP][CDR] Add actor.entity.id and target.entity.id fields to audit logs #11983

kubasobon · 2024-12-03T12:17:39Z

Proposed commit message

Add actor.entity.id and target.entity.id fields to properly identify events' origins and targets. It is a requirement for https://github.com/elastic/security-team/issues/9352.

Warning

~~To be merged after #11762. Please remember to change the base to main.~~
Merged. The base is now main.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Required by https://github.com/elastic/security-team/issues/9352
Closes https://github.com/elastic/security-team/issues/10398

@efd6

co-authored by: @efd6

elasticmachine · 2024-12-03T12:20:38Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

kubasobon · 2024-12-06T14:41:51Z

@romulets The issue stated origin.id and target.id, but I've seen actor and target.ENTITY.id formats elsewhere. Can you confirm what is the one we're going with?

efd6 · 2024-12-08T22:18:08Z

Can you confirm what is the one we're going with?

This is the kind of question that emphasises why we want to be working from accepted specifications. Decisions should already have been made and be publicly visible so that we know that we are all on the same page and so that our users know what behaviour they can expect.

packages/gcp/data_stream/audit/fields/base-fields.yml

elastic-vault-github-plugin-prod · 2024-12-11T17:19:40Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2024-12-11T17:20:24Z

💚 Build Succeeded

Buildkite Build
Commit: a74d888

History

💔 Build #19368 failed 47a0c46
💚 Build #19032 succeeded d4a3d0b
💔 Build #19015 failed a84fd8f
💔 Build #19014 failed ae5fd2c

cc @kubasobon

elastic-sonarqube · 2024-12-11T17:20:32Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

kubasobon · 2024-12-17T10:27:55Z

@elastic/obs-service-integrations & @elastic/security-service-integrations I would appreciate someone taking a look here :)

ishleenk17

I had some Nits and those are addressed. Thanks.
The changes mainly pertain to GCP audit logs owned by Security Service Integrations. I am providing code owner approval from obs-infraobs side.
Please get the code reviewed from security team before merging.

chrisberkhout

Minor question and comment.
Already fine.

chrisberkhout · 2024-12-17T15:01:02Z

packages/gcp/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+        boolean isKubernetes = false;
+        if (ctx.json?.resource?.type != null) {
+          String typ = ctx.json.resource.type;
+          isKubernetes = (typ == "k8s_cluster" || typ == "gke_cluster" || typ == "kubernetes");
+        }


Fine as is, but here's an alternative:

Suggested change

boolean isKubernetes = false;

if (ctx.json?.resource?.type != null) {

String typ = ctx.json.resource.type;

isKubernetes = (typ == "k8s_cluster" || typ == "gke_cluster" || typ == "kubernetes");

}

boolean isKubernetes = ["k8s_cluster", "gke_cluster", "kubernetes"].contains(ctx.json?.resource?.type);

That's a nice, concise way of setting the flag. I'll keep it in mind for the future.

chrisberkhout · 2024-12-17T15:05:25Z

packages/gcp/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+        String serviceName = ctx.json?.protoPayload?.serviceName ?: '';
+        if (serviceName == "compute.googleapis.com") {
+          if (ctx.json?.protoPayload?.request?.networkInterfaces instanceof List) {
+            for (def e: ctx.json.protoPayload.request.networkInterfaces) {
+              addValue(entities, e.network);
+            }
+          }
+          if (ctx.json?.protoPayload?.request?.serviceAccounts instanceof List) {
+            for (def e: ctx.json.protoPayload.request.serviceAccounts) {
+              addValue(entities, e.email);
+            }
+          }
+          if (ctx.json?.protoPayload?.request?.disks instanceof List) {
+            for (def e: ctx.json.protoPayload.request.disks) {
+              addValue(entities, e.source);
+            }
+          }
+        } else if (serviceName == "cloudresourcemanager.googleapis.com") {
+          if (ctx.json?.protoPayload?.request?.policy?.bindings instanceof List) {
+            for (def e: ctx.json.protoPayload.request.policy.bindings) {
+              addValue(entities, e.role);
+              for (def m: e.members) {
+                addValue(entities, m);
+              }
+            }
+          }
+          if (ctx.json?.protoPayload?.response?.bindings instanceof List) {
+            for (def e: ctx.json.protoPayload.response.bindings) {
+              addValue(entities, e.role);
+              for (def m: e.members) {
+                addValue(entities, m);
+              }
+            }
+          }
+        } else if (serviceName == "iamcredentials.googleapis.com") {
+          if (ctx.json?.protoPayload?.metadata?.identityDelegationChain instanceof List) {
+            for (def e: ctx.json.protoPayload.metadata.identityDelegationChain) {
+              addValue(entities, e);
+            }
+          }
+        }


Are the outer ifs necessary here? is it okay to just collect entities from any of these locations if they exist?

This could potentially be reduced to something closer to a short list of locations by using Streams, similar to this.

We were using streams for similar PRs, but we got pointed toward this approach in #11762.
I agree that nested if clauses are an overkill and we could just try to grab everything. However this way we get a nice readability bump, I think. It tells a story of where each value originates from, and lays solid groundwork for any future PRs.

elastic-vault-github-plugin-prod · 2024-12-19T08:43:47Z

Package gcp - 2.40.0 containing this change is available at https://epr.elastic.co/package/gcp/2.40.0/

…ogs (elastic#11983)

kubasobon added 10 commits November 18, 2024 11:56

document related.entity field

258fddd

add related.entity to audit logs and test it

88a3007

update changelog entry

71de5e8

fix code smells

ee85a15

use List instead of []String

dad99d4

apply review remarks

0e0f938

co-authored by: @efd6

extract origin and target IDs

356d569

fix test cases

91aaded

add new fields

d46f836

update docs

0634daf

kubasobon self-assigned this Dec 3, 2024

kubasobon changed the base branch from main to gcp-related-entity December 3, 2024 12:18

update changelog and manifest

ae5fd2c

kubasobon marked this pull request as ready for review December 3, 2024 12:20

kubasobon requested review from a team as code owners December 3, 2024 12:20

kubasobon added 2 commits December 3, 2024 13:31

add case for "Customer-Supplied-Encryption-Key-Event"

a84fd8f

update docs

d4a3d0b

kubasobon requested a review from a team December 4, 2024 13:24

ishleenk17 reviewed Dec 9, 2024

View reviewed changes

packages/gcp/data_stream/audit/fields/base-fields.yml Show resolved Hide resolved

kubasobon added 2 commits December 11, 2024 17:35

rename origin to actor

3de6116

refactor target structure

b146d42

fix field types

47a0c46

Base automatically changed from gcp-related-entity to main December 11, 2024 16:42

kubasobon added 3 commits December 11, 2024 17:49

Merge branch 'main' into gcp-origin-and-target

07cd89c

fix formatting

ab08697

fix test cases

a74d888

kubasobon requested a review from ishleenk17 December 16, 2024 15:26

ishleenk17 approved these changes Dec 17, 2024

View reviewed changes

chrisberkhout approved these changes Dec 17, 2024

View reviewed changes

chrisberkhout changed the title ~~[GCP][CDR] Add origin.id and target.id fields to audit logs~~ [GCP][CDR] Add actor.entity.id and target.entity.id fields to audit logs Dec 17, 2024

kubasobon merged commit a5c586f into main Dec 19, 2024
5 checks passed

kubasobon deleted the gcp-origin-and-target branch December 19, 2024 08:27

harnish-elastic pushed a commit to harnish-elastic/integrations that referenced this pull request Feb 4, 2025

[GCP][CDR] Add actor.entity.id and target.entity.id fields to audit l…

704c017

…ogs (elastic#11983)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GCP][CDR] Add actor.entity.id and target.entity.id fields to audit logs #11983

[GCP][CDR] Add actor.entity.id and target.entity.id fields to audit logs #11983

kubasobon commented Dec 3, 2024 •

edited

Loading

elasticmachine commented Dec 3, 2024

kubasobon commented Dec 6, 2024

efd6 commented Dec 8, 2024

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

elasticmachine commented Dec 11, 2024

elastic-sonarqube bot commented Dec 11, 2024

kubasobon commented Dec 17, 2024

ishleenk17 left a comment

chrisberkhout left a comment

chrisberkhout Dec 17, 2024

kubasobon Dec 18, 2024

chrisberkhout Dec 17, 2024

chrisberkhout Dec 17, 2024

kubasobon Dec 18, 2024

elastic-vault-github-plugin-prod bot commented Dec 19, 2024

[GCP][CDR] Add actor.entity.id and target.entity.id fields to audit logs #11983

[GCP][CDR] Add actor.entity.id and target.entity.id fields to audit logs #11983

Conversation

kubasobon commented Dec 3, 2024 • edited Loading

Proposed commit message

Checklist

Related issues

elasticmachine commented Dec 3, 2024

kubasobon commented Dec 6, 2024

efd6 commented Dec 8, 2024

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

🚀 Benchmarks report

elasticmachine commented Dec 11, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Dec 11, 2024

Quality Gate passed

kubasobon commented Dec 17, 2024

ishleenk17 left a comment

Choose a reason for hiding this comment

chrisberkhout left a comment

Choose a reason for hiding this comment

chrisberkhout Dec 17, 2024

Choose a reason for hiding this comment

kubasobon Dec 18, 2024

Choose a reason for hiding this comment

chrisberkhout Dec 17, 2024

Choose a reason for hiding this comment

chrisberkhout Dec 17, 2024

Choose a reason for hiding this comment

kubasobon Dec 18, 2024

Choose a reason for hiding this comment

elastic-vault-github-plugin-prod bot commented Dec 19, 2024

kubasobon commented Dec 3, 2024 •

edited

Loading