Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch Bulk API with bundled messages FOR COMPLIANCE INGESTION #74

Open
vjeffrey opened this issue Feb 25, 2019 · 1 comment
Open
Labels
performance enhancement This issue or pull request relates to performance enhancements for Automate

Comments

@vjeffrey
Copy link

User Story

In https://github.com/chef/a2/pull/5067 the client runs ingestion pipeline was modified to use the elasticsearch bulk api with bundled messages. This is a huge improvement for the pipeline, so let's implement it in the compliance ingestion pipeline too.
Please see https://github.com/chef/a2/pull/5067/files for more details.

Definition of Done

compliance ingestion uses es bulk api/bundles msgs

@vjeffrey vjeffrey transferred this issue from another repository Apr 14, 2019
@vjeffrey vjeffrey added performance enhancement This issue or pull request relates to performance enhancements for Automate chef-conf This issue is being targeted for delivery by chef conf and removed chef-conf This issue is being targeted for delivery by chef conf labels Apr 14, 2019
@lancewf lancewf self-assigned this May 23, 2019
@lancewf
Copy link
Contributor

lancewf commented May 29, 2019

The Elasticsearch Bulk API cannot be currently used for the compliance report ingestion. The reason the Bulk API cannot be used is that for each report an elasticsearch update-by-query is run to unmark the previous report from being the latest. The reason config-mgmt-service does not have this problem is that it has a separate index that only contains the latest run. Updating one document with its ID in this index can use the Bulk API. The reason update-by-query cannot use the Bulk API is, it is searching for the document that needs to be changed.

https://github.com/chef/automate/blob/master/components/compliance-service/ingest/ingestic/ingestic.go#L141

@lancewf lancewf removed their assignment Oct 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance enhancement This issue or pull request relates to performance enhancements for Automate
Projects
None yet
Development

No branches or pull requests

2 participants