Add support for namespace suffixes

[kubasobon]

Summary of your changes

This pull request allows Cloudbeat to publish events to non-default namespaces. For example, findings are sent to the logs-cloud_security_posture.findings-{namespace} Data Stream. Until now, the {namespace} always defaulted to default, regardless of Agent Policy. After merging this change, Cloudbeat will respect the Agent Policy and set publish events to the appropriate stream.

Screenshot/Data

CSPM using non-default namespace

Non-default Data Streams present when running modified cloudbeat

Findings published to the new Data Stream successfully

EDIT:

Vulnerabilities published to the new Data Stream successfully

Related Issues

Closes #2289

Checklist

• I have added tests that prove my fix is effective or that my feature works
• I have added the necessary README/documentation (if appropriate)

[mergify]

This pull request does not have a backport label. Could you fix it @kubasobon? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit
  NOTE: backport-skip has been added to this pull request.

[github-actions]

📊 Allure Report - 💚 No failures were reported.

Result	Count
🟥 Failed	0
🟩 Passed	359
⬜ Skipped	33

[kubasobon]

Our integration has the correct ECS data_stream fields defined, and we could use those to build the index, like so {data_stream.type}-{data_stream.dataset}-{data_stream.namespace}.
However, the namespace field is empty for current release of the integration. The Policy/Config also comes with the index field, which contains a ready-to-use index name in the correct format.

@kfirpeled Do you know where does index field come from and is it reliable enough for us to use?

[romulets]

How I understand is that a integration has the permissions to to publish to a specific index. See the policy template once you install it. So essentially the integration already knows where you can publish to. The index template is decided by the integration

[kubasobon]

Yeah, that I know :) I wanted to know if there is anything preventing us from relying on the index field.

[kfirpeled]

I'm not familiar with index in the agent config. I'm only familiar with agent's policy.

Here's a snippet:

inputs:
  - id: logfile-system-1c2284c7-c5ae-481d-a7bc-d9e3ce61545a
    name: system-3
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 1.58.1
    data_stream:
      namespace: default
    package_policy_id: 1c2284c7-c5ae-481d-a7bc-d9e3ce61545a
    streams:
      - id: logfile-system.auth-1c2284c7-c5ae-481d-a7bc-d9e3ce61545a
        data_stream:
          dataset: system.auth
          type: logs

So I can only guess from here how it is being done on the beats side

[romulets]

Why is this called default it doesn't mention any namespace?

[kubasobon]

This is why it's a prefix. The whole index is constructed as {prefix}-{namespace}. The prefix can vary - see the prefix for Vulnerability events.

[kfirpeled]

Here's an example what is being sent for CNVM

  - id: cloudbeat/vuln_mgmt_aws-vuln_mgmt-6921c0f7-ae37-4a1b-910b-f158a7c1af64
    name: cnvm
    revision: 1
    type: cloudbeat/vuln_mgmt_aws
    use_output: default
    meta:
      package:
        name: cloud_security_posture
        version: 1.9.0
    data_stream:
      namespace: default
    package_policy_id: 6921c0f7-ae37-4a1b-910b-f158a7c1af64
    streams:
      - id: >-
          cloudbeat/vuln_mgmt_aws-cloud_security_posture.vulnerabilities-6921c0f7-ae37-4a1b-910b-f158a7c1af64
        data_stream:
          dataset: cloud_security_posture.vulnerabilities
          type: logs
        period: 24h

So the datastream is built by <type>-<dataset>-<namespace>

[romulets]

How I understand is that a integration has the permissions to to publish to a specific index. See the policy template once you install it. So essentially the integration already knows where you can publish to. The index template is decided by the integration

[kfirpeled]

It would be best to approach #fleet or find some documentation regardless beats and how send events to datastreams and reading the agent policy. I'm not familiar with the best practices