Skip to content

Commit

Permalink
change contents
Browse files Browse the repository at this point in the history
correct some docs

manually migrate some files

missed one line

correct the file formatting

change to a python block

add index

got the docs to build locally without sphinx warnings
  • Loading branch information
stan-dot authored and callumforrester committed May 23, 2024
1 parent 43b595b commit a3f023f
Show file tree
Hide file tree
Showing 19 changed files with 231 additions and 384 deletions.
32 changes: 10 additions & 22 deletions docs/explanations/architecture.md
Original file line number Diff line number Diff line change
@@ -1,46 +1,34 @@
Architecture
============
# Architecture


Blueapi performs a number of tasks:

* Managing the Bluesky RunEngine_, giving it instructions and handling its errors. Traditionally this job has been done by a human with an IPython_ terminal, so it requires automating.
* Managing the Bluesky [RunEngine](https://nsls-ii.github.io/bluesky/run_engine_api.html), giving it instructions and handling its errors. Traditionally this job has been done by a human with an [IPython](https://ipython.org/) terminal, so it requires automating.
* Maintaining a registry of plans and devices. In the aforementioned IPython_ case, these would have just been global variables.
* Communicating with the outside world, accepting instructions to run plans, providing updates on plan progress etc.

These responsibilities are kept separate in the codebase to ensure a clean, maintainable architecture.

Key Components
--------------
## Key Components

.. figure:: ../images/blueapi-architecture.png
:width: 600px
:align: center

main components
![blueapi architecture main components](../images/blueapi-architecture.png)


The ``BlueskyContext`` Object
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
### The `BlueskyContext` Object

Holds registries of plans and devices as well as a number of helper methods for
registering en-masse from a normal Python module.

### The Worker Object

The Worker Object
^^^^^^^^^^^^^^^^^

Wraps the Bluesky ``RunEngine`` and accepts requests to run plans. The requests include the name
Wraps the Bluesky `RunEngine` and accepts requests to run plans. The requests include the name
of the plan and a dictionary of parameters to pass. The worker validates the parameters against
the known expectations of the plan, passes it to the ``RunEngine`` and handles any errors.
the known expectations of the plan, passes it to the `RunEngine` and handles any errors.


The Service Object
^^^^^^^^^^^^^^^^^^
### The Service Object

Handles communications and the API layer. This object holds a reference to the worker
can interrogate it/give it instructions in response to messages it recieves from the message
bus. It can also forward the various events generated by the worker to topics on the bus.


.. _RunEngine: https://nsls-ii.github.io/bluesky/run_engine_api.html
.. _IPython: https://ipython.org/
15 changes: 5 additions & 10 deletions docs/explanations/decisions/0003-no-queues.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,22 @@
3. No Queues
============
# 3. No Queues

Date: 2023-05-22

Status
------
## Status

Accepted

Context
-------
## Context

In asking whether this service should hold and execute a queue of tasks.

Decision
--------
## Decision

We will not hold any queues. The worker can execute one task at a time and will return
an error if asked to execute one task while another is running. Queueing should be the
responsibility of a different service.

Consequences
------------
## Consequences

The API must be kept queue-free, although transactions are permitted where the server
caches requests.
15 changes: 5 additions & 10 deletions docs/explanations/decisions/0004-api-case.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,21 @@
4. API Model Case
=================
# 4. API Model Case

Date: 2023-05-23

Status
------
## Status

Accepted

Context
-------
## Context

Considering whether keys in JSON blobs from the API should be in snake_case or camelCase.
This includes plan parameters which may be user-defined.

Decision
--------
## Decision

The priority is not to confuse users, so we will not alias any field names defined in Python.

Consequences
------------
## Consequences

Most code will be written with pep8 enforcers which means most field names will be snake_case.
Some user defined ones may differ.
48 changes: 16 additions & 32 deletions docs/explanations/events.md
Original file line number Diff line number Diff line change
@@ -1,36 +1,30 @@
Events Emitted by the Worker
============================
# Events Emitted by the Worker

Blueapi adds new events on top of the `bluesky event model`_.
Blueapi adds new events on top of the [`bluesky event model`](https://blueskyproject.io/event-model/main/index.html).

Reasons
-------
## Reasons

Since the ``RunEngine`` is traditionally used by a human in front of an IPython terminal, it

Since the `RunEngine` is traditionally used by a human in front of an IPython terminal, it
sometimes assumes intuitive behaviour. The worker replaces the human and so must fill in the
gaps.

The base engine programatically emits data events conforming to the `bluesky event model`_. These
are meant to be handled by other subscribing code (e.g. databroker) and are decoupled from concerns such as whether
a plan has started, finished, paused, errored etc. See the example below:

.. figure:: ../images/bluesky-events.png
:width: 600px
:align: center

sequence of event emission compared to plan start/finish, in a complicated case
![sequence of event emission compared to plan start/finish, in a complicated case](../images/bluesky-events.png)

Note the gap between the start of the plan and the issue of the first `run start document`_, and the similar gap
Note the gap between the start of the plan and the issue of the first [`run start document`](https://blueskyproject.io/event-model/main/user/explanations/data-model.html#run-start-document), and the similar gap
for the stop document vs end of the plan, thsse are typically used for setup and cleanup.
Also note that a plan can produce an arbitrary number of runs. This decoupling is fine in an IPython terminal
because a human user can see when a plan has started, can see when it's finished and can see which runs are
associated with which plans.

New Events
----------
## New Events

For the case of automation, we introduce a new set of events outside of the event model, specifically
pertaining to the running of the plan and state of the ``RunEngine``. At a mimimum, an event is emitted
pertaining to the running of the plan and state of the `RunEngine`. At a mimimum, an event is emitted
every time the engine:

* Starts a new plan
Expand All @@ -40,28 +34,18 @@ every time the engine:
In the latter case, information about the error is also included.


Correlation ID
--------------
## Correlation ID


When controlling plans programatically, it can be useful to verify that event model documents really are related to
the plan you just asked the worker to run. The worker will therefore bundle a correlation ID into the headers of
messages containing documents.

.. seealso:: `Microsoft Playbook on Correlation IDs`_
see also [`Microsoft Playbook on Correlation IDs`](https://microsoft.github.io/code-with-engineering-playbook/observability/correlation-id/)

ActiveMQ will give this header a different name depending on the protocol you use.

.. list-table:: Correlation ID Headers
:widths: 25 25
:header-rows: 1

* - Protocol
- Header name
* - JMS
- jms_correlationID
* - STOMP
- correlation-id

.. _`bluesky event model`: https://blueskyproject.io/event-model/main/index.html
.. _`run start document`: https://blueskyproject.io/event-model/main/user/explanations/data-model.html#run-start-document
.. _`Microsoft Playbook on Correlation IDs`: https://microsoft.github.io/code-with-engineering-playbook/observability/correlation-id/
| Protocol | Header name |
|----------|------------------|
| JMS | jms_correlationID|
| STOMP | correlation-id |
55 changes: 22 additions & 33 deletions docs/explanations/lifecycle.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
Lifecycle of a Plan
===================
# Lifecycle of a Plan

The following demonstrates exactly what the code does with a plan through its lifecycle
of being written, loaded and run. Take the following plan.

.. code:: python

```
from typing import Any, List, Mapping, Optional, Union
import bluesky.plans as bp
Expand Down Expand Up @@ -42,23 +39,21 @@ of being written, loaded and run. Take the following plan.
"""
yield from bp.count(detectors, num, delay=delay, md=metadata)
```



Loading and Registration
------------------------
## Loading and Registration

Blueapi will load this plan into its context if configured to load either this module or a module that
imports it. The ``BlueskyContext`` will go through all global variables in the module and register them
imports it. The `BlueskyContext` will go through all global variables in the module and register them
if it detects that they are plans.

At the point of registration it will inspect the plan's parameters and their type hints, from which it
will build a pydantic_ model of the parameters to validate against. In other words, it will build something
will build a [pydantic](https://docs.pydantic.dev/) model of the parameters to validate against. In other words, it will build something
like this:


.. code:: python

```
from pydantic import BaseModel
class CountParameters(BaseModel):
Expand All @@ -71,31 +66,26 @@ like this:
arbitrary_types_allowed = True
validate_all = True
.. note::

This is for illustrative purposes only, this code is not actually generated, but an object
resembling this class is constructed in memory.
The default arguments will be validated by the context to inject the "det" device when the
plan is run. The existence of the "det" default device is not checked until this time.
```

The model is also stored in the context.


Startup
-------
## Startup

On startup, the context is passed to the worker, which is passed to the service.
The worker also holds a reference to the ``RunEngine`` that can run the plan.
The worker also holds a reference to the `RunEngine` that can run the plan.


Request
-------
## Request

A user can send a request to run the plan to the service, which includes values for the parameters.
It takes the form of JSON and may look something like this:

.. code:: json

```
{
"name": "count",
"params": {
Expand All @@ -107,33 +97,32 @@ It takes the form of JSON and may look something like this:
"delay": 0.1
}
}
```

The ``Service`` receives the request and passes it to the worker, which holds it in an internal queue
The `Service` receives the request and passes it to the worker, which holds it in an internal queue
and executes it as soon as it can.


Validation
----------
## Validation

The pydantic model from earlier, as well as the plan function itself, is loaded out of the registry
The parameter values in the request are validated against the model, this includes looking up devices
with names ``andor`` and ``pilatus`` or, if detectors was not passed ``det``.
with names `andor` and `pilatus` or, if detectors was not passed `det`.

See also [type validators](./type_validators.md)

.. seealso:: `./type_validators`

Execution
---------
## Execution

The validated parameter values are then passed to the plan function, which is passed to the RunEngine.
The plan is executed. While it is running, the ``Worker`` will publish
The plan is executed. While it is running, the `Worker` will publish

* Changes to the state of the ``RunEngine``
* Changes to the state of the `RunEngine`
* Changes to any device statuses running within a plan (e.g. when a motor changes position)
* Event model documents emitted by the ``RunEngine``
* Event model documents emitted by the `RunEngine`
* When a plan starts, finishes or fails.

If an error occurs during any of the stages from "Request" onwards it is sent back to the user
over the message bus.

.. _pydantic: https://docs.pydantic.dev/

Loading

0 comments on commit a3f023f

Please sign in to comment.