Added introductory section for queue documentation

docs/source/dev/queue.md

@@ -1,7 +1,7 @@
 # Queue system
-Today's MXCuBE queue system was first implemented in version 2.x of MXCuBE. The aim was to create a queuing system that provided means for automation and that integrated well with the sample changer robots, the ISPyB LIMS system, and workflow engines. The queue was designed to have manual and automatic modes of operation.
+Today's MXCuBE queue system was first implemented in version 2.x of MXCuBE. The aim was to create a queuing system that provided means for automation and that integrated well with the sample changer robots, the ISPyB LIMS system, and workflow engines. The system was also designed to be flexible to handle the rapid technical and scientific evolution of the beamlines.
 
-To enable automatic data collection over several samples, one needs to associate tasks to be performed on a set of samples. The decision was to represent the queue as a tree of nodes, where each node defines the logic for a certain task or collection protocol. Each node can have a set of child nodes that are executed after the main logic of its parent. There are no constraints on the structure of the tree to not limit the possibilities for more complex collection protocols, the nesting of the various types of nodes can be done freely. For conventional colletions with sample changer containing pins with samples. A sample node, for example, defines how a sample is mounted, and its child nodes define what is done on that sample once it is mounted.
+To enable automatic data collection over several samples, there is a need to associate a set of tasks to be performed on the given samples. The decision was to represent the queue as a tree of nodes, where each node defines the logic for a certain task or collection protocol. Each node can have a set of child nodes that are executed after the main logic of its parent. There are no constraints on the structure of the tree, to not limit the possibilities for more complex collection protocols. The nesting and reuse of the various types of nodes can be done freely. For conventional collections with a sample changer containing pins with samples. A sample node, for example, defines how a sample is mounted, and its child nodes define what is done on that sample once it is mounted.
 
 ```{figure} /assets/queue_overview_node_concept.svg
 :width: 400
@@ -10,18 +10,19 @@ To enable automatic data collection over several samples, one needs to associate
 *Typical structure of a queue for conventional colletions with sample changer containing pins with samples.*
 ```
 
-**_NOTE:_** There are no constraints on the structure of the tree. However, today: the top level always contain sample nodes,
-and the nesting occours at the group level. The depth of nesting is currently maximum (3? to be confirmed)
+**_NOTE:_** There are no constraints on the structure of the tree. However, today: the top level always contains sample nodes,
+and the nesting occurs at the group level. The depth of nesting is currently maximum 3 - Sample -
+Data collection group - Datacollection. Group is the name given to a node that groups together several other nodes, for instance, data collections. A group is just like any other node and can have its behavior defined in an `execute` method. A group can, for instance, commonly be a workflow that consists of several different steps.
 
 
-```{figure} /assets/queue_overview_node_complex.svg
+```{figure} /assets/queue_complex_node_concept.svg
 :width: 400
-:alt: Queue as bad as it gets
+:alt: More complex queue
 
-*A nested strcture with several wedges, one in each group*
+*A nested strucutre, A group (Group1) with several sub groups each with several data collections*
 ```
 
-The execution order is depth first, so that all the children of a node are executed before the node's siblings. Each node has a `execute` method that defines its main logic and `pre_execute`and `post_execute` methods that are executed before and after the main `execute` method.
+The execution order is depth first, so that all the children of a node are executed before the node's siblings. Each node has a `execute` method that defines its main logic and `pre_execute`and `post_execute` methods that are executed before and after the main `execute` method. There are also means to stop, pause and skip entries (by raising `QueueSkippEntryException`).
 
 This design was thought to map well to the semantics of how data is collected with a sample changer and the upload of metadata to LIMS. The sample changer has a number of samples, each having a set of data collections (child nodes). The results are uploaded to the lims system after each data collection, (`post_execute`). Workflow engines can further group native collection protocols into grouping nodes, which in turn can be used as building blocks for even more complex collection strategies.
 
@@ -31,9 +32,9 @@ This design was thought to map well to the semantics of how data is collected wi
 *`pre_execute` and `post_execute` methods that are executed before and after the main `execute` method.*
 ```
 
-The queue system and node, mentioned above, can be seen as having two conceptual parts: a logic part and a model part. The logic part consists of an object called `QueueManager` and objects are often referred to as *queue entries*. Queue entry objects are simply objects inheriting `BaseQueueEntry`, for instance `DataCollectionQueueEntry`. The queue entry objects define the behavior (logic) of a specific collection protocol. The queue consists of several queue entry objects that are executed via the `QueueManager`.
+The queue system and node, mentioned above, can be seen as having two conceptual parts: a behavioral part and a model part. The behavioral part consists of an object called `QueueManager` and objects are often referred to as *queue entries*. Queue entry objects are simply objects inheriting `BaseQueueEntry`, for instance `DataCollectionQueueEntry`. The queue entry objects define the behavior (logic) of a specific collection protocol. The queue consists of several queue entry objects that are executed via the `QueueManager`.
 
-The model consists of an object called `QueueModel` that has a tree structure of `TaskNode` objects. The `TaskNode` object is often referred to as a *queue model object* and defines the data associated with a collection protocol. There is a one-to-one mapping between a specific queue model object and a specific queue entry, for example, `DataCollectionTaskNode` and `DataCollectionQueueEntry`. The mapping is defined in a structure called MODEL_QUEUE_ENTRY_MAPPINGS in mxcubecore/queue_entry/base_queue_entry.py.
+The model consists of an object called `QueueModel` that has a tree structure of `TaskNode` objects. The `TaskNode` object is often referred to as a *queue model object* and defines the data associated with a collection protocol. There is a one-to-one mapping between a specific queue model object and a specific queue entry, for example, `DataCollectionTaskNode` and `DataCollectionQueueEntry`. The mapping is defined in a structure called [MODEL_QUEUE_ENTRY_MAPPINGS](https://github.com/mxcube/mxcubecore/blob/develop/mxcubecore/queue_entry/__init__.py#L83)
 
 The combination of a `QueueEntry` and its model (`QueueModel`) is often referred to as a *task*, and makes up the entity that will be executed when the queue is executed.
 
@@ -45,9 +46,9 @@ The one-to-one mapping makes it possible to represent a queue and construct the
 A task is created by creating a `Tasknode` and  attaching it to a corresponding `QueueEntry`. The `QueueEntry` is added or *enqueued* to the queue via the method `QueueManager.enqueue`.
 
 ### Differences between the Web and the Qt user interfaces
-When the Web interface was designed, the scientists wanted to change some aspects of how the queue operates and how it is presented to the user. The decision was to work on one sample at a time, rather than for the entire queue, as is done in the Qt version. This also means  that the Web interface is centered around displaying the contents of the queue for one sample. It was also decided to not display `DataCollectionGroupQueueEntry` in the queue. The `DataCollectionGroupQueueEntry` still exists in the Web interface, but there is no view created for it.
+When the Web interface was designed, the scientists wanted to change some aspects of how the queue operates and how it is presented to the user. The decision was to work on one sample at a time, instead of the entire queue at once, as is done in the Qt version. This also means that the Web interface is centered around displaying the contents of the queue for one sample at the time. It was also decided to not display `DataCollectionGroupQueueEntry` in the queue. The `DataCollectionGroupQueueEntry` still exists in the Web interface, but there is no view created for it.
 
-On a technical level, there is a tight coupling between the `QueueEntry` base class to a Qt view object that makes the creation of the object slightly different between the two user interfaces (Qt and Web). To handle the tight coupling to Qt a `Mock` object used in the Web version instead of a Qt view and the handling of the updating of the view in the web version handled through signals passed over websockets to the client.
+On a technical level, there is a tight coupling between the `QueueEntry` base class and a Qt view object that makes the creation of the `QueueEntry` object slightly different between the two user interfaces (Qt and Web). To handle the tight coupling to Qt, a `Mock` object is used in the web version instead of a Qt View. The Qt interface can access the view directly via a reference, something that is impossible in the Web interface. Because the view and the `QueueEntry` do not execute in the same process, updates to the view in the web version are instead done through signals passed over websockets to the client.
 
 ### Creation of QueueEntry and QueueModel
 A simple example of the creation of a task: data collection is added to a group, which in turn is added to a sample.
@@ -74,4 +75,6 @@ sample_entry.enqueue(group_entry)
 group_entry.enqueue(dc_entry)
 ```
 
-Each type of data collection protocol has a piece of code similar to the code above associated with it. There is a mapping, MODEL_QUEUE_ENTRY_MAPPINGS, in mxcubecore/queue_entry/base_queue_entry.py, that is used to associate the types of models and entries. In the Web version, this is done in mxcubeweb.components.queue.py in the methods called add_**task_type**, for instance `add_data_collection`. In the Qt version, the creation is done in two different places: first in the widget where the queue model object is created, and then through `QueueModel.view_created`.
+Each type of data collection protocol has a piece of code similar to the code above associated with it. In the Web version, this is done in mxcubeweb.components.queue.py in the methods called add_**task_type**, for instance `add_data_collection`. In the Qt version, the creation is done in two different places: first in the widget where the queue model object is created, and then through `QueueModel.view_created`.
+
+## Execution