-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move from mongodb to tiled-backed catalog #339
Conversation
…or support for ADs.
Developed on the beamline, tests fail.
As-written, the tiled writer exists as a run engine callback. Previously, we put documents on the kafka topic and read them out using the Maybe that should be a different PR, where the run_engine factory lets you attach the kafka producer instead of the tiled writer and databroker separately? |
Also, I think documentation for this belongs on the wiki, since it extends beyond the scope of Haven. |
I added a TiledConsumer module to the queueserver package that reads documents from Kafka and sends them to Tiled. I will do a separate PR that adds a kafka produce to the run engine (the queueserver does this automatically). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for changing the data structure for me steal!
The bluesky project is moving from storing raw scan documents in a mongo database via databroker to storing tabular data in a Tiled catalog backed by a relational (sqlite or postgres) database. This PR updates Haven and Firefly to use the Tiled catalog by default.
This change is necessary to enable ophyd-async detector data to be read back out by Tiled.
The aim with this PR is keep top-level features (e.g. run browser) the same as with the old database structure. This PR does not add features that will be useful with the new catalog, such as filtering by beamline; these will be added in a separate PR.
To avoid losing data, the raw documents should still be saved in the mongo database. This way, when we are ready to fully switch over we will create a new tiled catalog and stream the mongodb documents into the new catalog. I suggest we keep that old mongodb going until we are confident we don't need them.
Things to do before merging:
write docs