Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Impove performance when retrieving time series for set of epochs #308

Open
1 of 2 tasks
ehennestad opened this issue Jan 7, 2025 · 4 comments
Open
1 of 2 tasks

Comments

@ehennestad
Copy link
Collaborator

ehennestad commented Jan 7, 2025

For the NDI-Marder project, profiling the element method plotEpochs uncovered some performance issue.

Profiling was done on session '412691d3730ea2a0_40cb949cfbed9eba', element: ntrode/dgn-1

Total time before making changes: 21.1 seconds

  1. Repeated calls to ndi.file.navigator/getepochfiles_number
    Todo:

    • Cache the result in a map where epoch number is the key and the resulting file path(s) is the value.
  2. Repeated calls the the database to check for ingested documents with the same query parameters.

    epoch_query = ndi.query('','isa','epochfiles_ingested') & ...
    ndi.query('','depends_on','filenavigator_id',ndi_filenavigator_obj.id()) & ...
    ndi.query('base.session_id','exact_string',ndi_filenavigator_obj.session.id());
    d_ingested = ndi_filenavigator_obj.session.database_search(epoch_query);

    This is called once per each epoch (107), resulting in total call time of ~3 seconds to mksqlite.
    @stevevanhooser What about caching the result in a temporary cache with a timeout of e.g 1 minute? Other ideas?

  3. Repeated calls to abfloaf
    There are 107 epochs, but 555 calls to read abc_header, totalling ~2 seconds.
    Todo:

    • Can this be improved?
@ehennestad
Copy link
Collaborator Author

Total time after doing 1: 15.5 seconds

@stevevanhooser
Copy link
Contributor

Cool, thanks. Should this be a pull request yet or should we wait?

@ehennestad
Copy link
Collaborator Author

Still want to work on points 2 & 3

@ehennestad
Copy link
Collaborator Author

Total time after doing 2 & 3: 11.1 seconds

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants