Skip to content
This repository has been archived by the owner on Mar 24, 2021. It is now read-only.

Iteratively load data into ElasticSearch during indexing #242

Open
christabone opened this issue Mar 20, 2017 · 1 comment
Open

Iteratively load data into ElasticSearch during indexing #242

christabone opened this issue Mar 20, 2017 · 1 comment
Assignees
@christabone
Labels
backend data enhancement re-index-required
Milestone
0.6.0

Comments

@christabone
Copy link
Contributor

To help alleviate memory issues during indexing we need to change the data loading to become a more iterative process.

Starting this work now, will update with more information asap.
@christabone christabone added this to the 0.6.0 milestone Mar 20, 2017
@christabone christabone self-assigned this Mar 20, 2017
@christabone
Copy link
Contributor Author

christabone commented Mar 21, 2017 •
edited
Loading

Indexing on a per-MOD basis appears to limit the max memory usage to about 1.2 GB. Still working on improving garbage collection and removing unused elements when they are no longer required (fairly confident it can be lower). This is only using 0.3.0 data, DIOPT is not yet incorporated.

Will hopefully have a PR tomorrow for folks to review.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@christabone christabone

Labels
backend data enhancement re-index-required
Projects
None yet
Milestone
0.6.0
Development

No branches or pull requests
1 participant
@christabone