diff --git a/src/glossary.rst b/src/glossary.rst index 6daa019174..bd0ec44725 100644 --- a/src/glossary.rst +++ b/src/glossary.rst @@ -68,10 +68,41 @@ Glossary don't get overwritten. + active + active task + An active task is a task which is near ready to run, in the process of + running, or which requires user intervention. + + Active tasks are: + + - Tasks which have some, but not all of their prerequisites satisfied. + - ``waiting`` tasks, that are actively waiting on: + + - :term:`xtriggers `. + - :ref:`internal queues ` + - :ref:`runahead limit ` + - ``preparing`` tasks - i.e. tasks in the process of submitting jobs + - ``submitted`` and ``running`` tasks - i.e. those with active jobs + - :term: `incomplete` tasks which have completed without satisfying all of their + :term:`required outputs `, (e.g. a failed task + where success was required). + + Active tasks are in the ``n=0`` :term:`window ` which means they + will always be displayed in the GUI and Tui. + + The distinction between active and non-active tasks is important for + the computing of the :term:`runahead limit`. + + + active cycle + A cycle is active if it contains any :term:`active tasks `. + + Active cycles are counted towards the :term:`runahead limit`. + + window n-window active window - workflow window active task pool This is a :term:`graph`-based window or view of the workflow at runtime, including tasks out to ``n`` graph edges from current :term:`active @@ -155,19 +186,6 @@ Glossary the same as :term:`workflow id`. - active - active task - An active task is a task in the submitted or running state. - - - active-waiting - active-waiting task - An active-waiting task is a task in the :term:`scheduler's ` - ``n=0`` :term:`active window` that is ready to run according to its task - prerequisites, but is still waiting on a limiting mechanism such as a - :term:`clock trigger`, task :term:`hold`, or :term:`internal queue`. - - external trigger xtrigger External triggers allow :term:`tasks ` in the :term:`graph` to @@ -1675,13 +1693,18 @@ Glossary runahead limit runahead - In a :term:`cycling workflow`, the runahead limit determines the maximum - number of consecutive cycle points that can be active at once. + In a :term:`cycling workflow`, the runahead limit determines how far + ahead of the oldest :ref:`active cycle point` the workflow is permitted + to run. + + The "oldest active cycle point" is the first cycle in the workflow to contain + any :term:`active tasks ` (e.g. running tasks). .. seealso:: * :cylc:conf:`[scheduling]runahead limit` * :ref:`Runahead Limiting` + * :term:`active cycle` workflow completion diff --git a/src/hyperlinks.rst.include b/src/hyperlinks.rst.include index aad84f1841..9a22b11f2e 100644 --- a/src/hyperlinks.rst.include +++ b/src/hyperlinks.rst.include @@ -38,6 +38,9 @@ .. _Cylc UI Server: https://github.com/cylc/cylc-uiserver .. _UI Server: https://github.com/cylc/cylc-uiserver + +.. Task Icons (inline) + .. |task-waiting| image:: /img/task-job-icons/task-waiting.png :height: 18px :align: middle @@ -71,6 +74,8 @@ :align: middle +.. Task Modifiers (inline) + .. |task-held| image:: /img/task-job-icons/task-isHeld.png :height: 18px :align: middle @@ -84,6 +89,23 @@ :align: middle +.. Task Modifiers (inline with modifiers as superscript) + +.. |task-held-super| image:: /img/task-job-icons/task-isHeld.png + :height: 24px + :align: middle + +.. |task-runahead-super| image:: /img/task-job-icons/task-isRunahead.png + :height: 24px + :align: middle + +.. |task-queued-super| image:: /img/task-job-icons/task-isQueued.png + :height: 24px + :align: middle + + +.. Job Icons (inline) + .. |job-submitted| image:: /img/task-job-icons/job-submitted.png :height: 18px :align: middle diff --git a/src/img/n-window.gif b/src/img/n-window.gif new file mode 100644 index 0000000000..351a27b47b Binary files /dev/null and b/src/img/n-window.gif differ diff --git a/src/user-guide/running-workflows/index.rst b/src/user-guide/running-workflows/index.rst index fce48c4a85..18665c24c1 100644 --- a/src/user-guide/running-workflows/index.rst +++ b/src/user-guide/running-workflows/index.rst @@ -7,7 +7,6 @@ Running Workflows :maxdepth: 2 scheduler-start-up - task-job-states tasks-jobs-ui retrying-tasks tracking-task-state diff --git a/src/user-guide/running-workflows/reflow.rst b/src/user-guide/running-workflows/reflow.rst index 3bac689f60..23bbe42651 100644 --- a/src/user-guide/running-workflows/reflow.rst +++ b/src/user-guide/running-workflows/reflow.rst @@ -122,12 +122,14 @@ Triggering with No Active Flows same as if you had triggered the task just before the workflow completed. Special Case: Triggering ``n=0`` Tasks - Tasks in the ``n=0`` window are :term:`active`, :term:`active-waiting`, or - :term:`incomplete`. Their flow membership is already determined - that of + Tasks in the ``n=0`` window are :term:`active tasks `. + Their flow membership is already determined - that of the parent tasks that spawned them. - - Triggering an :term:`active task` has no effect (it is already triggered). - - Triggering an :term:`active-waiting task` queues it to run in the same flow. + - Triggering a task with a submitted or running job has no effect + (it is already triggered). + - Triggering other :term:`active tasks ` e.g. (a waiting + task which is held) queues it to run in the same flow. - Triggering an :term:`incomplete task` queues it to re-run in the same flow. diff --git a/src/user-guide/running-workflows/task-job-states.rst b/src/user-guide/running-workflows/task-job-states.rst deleted file mode 100644 index 91dfe0cc97..0000000000 --- a/src/user-guide/running-workflows/task-job-states.rst +++ /dev/null @@ -1,91 +0,0 @@ -.. _task-job-states: - -Task & Job States -================= - -**Tasks** are a workflow abstraction; they represent future and past jobs as -well as current active jobs. In the Cylc UI, task states have monochromatic -icons like this: |task-running|. - -**Jobs** represent real job scripts submitted to run -on a :term:`job platform`. In the Cylc UI, job states have coloured icons like -this: |job-running|. - -A single task can have multiple jobs, by automatic retry or manual triggering. - - -.. table:: - - ============== ==================== =================== ==================================== - State Task Icon Job Icon Description - ============== ==================== =================== ==================================== - waiting |task-waiting| waiting on prerequisites - preparing |task-preparing| job being prepared for submission - submitted |task-submitted| |job-submitted| job submitted - running |task-running| |job-running| job running - succeeded |task-succeeded| |job-succeeded| job succeeded - failed |task-failed| |job-failed| job failed - submit-failed |task-submit-failed| |job-submit-failed| job submission failed - expired |task-expired| will not submit job (too far behind) - ============== ==================== =================== ==================================== - -The running task icon contains a clock face which shows the time elapsed -as a proportion of the average runtime. - -.. image:: ../../img/task-job-icons/task-running-0.png - :width: 50px - :height: 50px - :align: left - -.. image:: ../../img/task-job-icons/task-running-25.png - :width: 50px - :height: 50px - :align: left - -.. image:: ../../img/task-job-icons/task-running-50.png - :width: 50px - :height: 50px - :align: left - -.. image:: ../../img/task-job-icons/task-running-75.png - :width: 50px - :height: 50px - :align: left - -.. image:: ../../img/task-job-icons/task-running-100.png - :width: 50px - :height: 50px - :align: left - -.. NOTE: these pipe characters are functional! They create a line break. - -| - -| - - -Task Modifiers --------------- - -Tasks are run as soon as their dependencies are satisfied, however, there are -some other conditional which can prevent tasks from being run. These are -given "modifier" icons which appear to the top-left of the task icon: - -.. list-table:: - :class: grid-table - :align: left - :widths: 20, 80 - - * - .. image:: ../../img/task-job-icons/task-isHeld.png - :width: 60px - :height: 60px - - **Held:** Task has been manually :term:`held ` back from - running. - * - .. image:: ../../img/task-job-icons/task-isRunahead.png - :width: 60px - :height: 60px - - **Runahead:** Task is held back by the :term:`runahead limit`. - * - .. image:: ../../img/task-job-icons/task-isQueued.png - :width: 60px - :height: 60px - - **Queued:** Task has been held back by an :term:`internal queue`. diff --git a/src/user-guide/running-workflows/tasks-jobs-ui.rst b/src/user-guide/running-workflows/tasks-jobs-ui.rst index f7bc6b0ef4..813c16bfd6 100644 --- a/src/user-guide/running-workflows/tasks-jobs-ui.rst +++ b/src/user-guide/running-workflows/tasks-jobs-ui.rst @@ -1,45 +1,153 @@ +.. _task-job-states: + +Tasks in the GUI/Tui +==================== + +Task & Job States +----------------- + +**Tasks** are a workflow abstraction; they represent future and past jobs as +well as current active jobs. In the Cylc UI, task states have monochromatic +icons like this: |task-running|. + +**Jobs** represent real job scripts submitted to run +on a :term:`job platform`. In the Cylc UI, job states have coloured icons like +this: |job-running|. + +A single task can have multiple jobs, by automatic retry or manual triggering. + + +.. table:: + + ============== ==================== =================== ==================================== + State Task Icon Job Icon Description + ============== ==================== =================== ==================================== + waiting |task-waiting| waiting on prerequisites + preparing |task-preparing| job being prepared for submission + submitted |task-submitted| |job-submitted| job submitted + running |task-running| |job-running| job running + succeeded |task-succeeded| |job-succeeded| job succeeded + failed |task-failed| |job-failed| job failed + submit-failed |task-submit-failed| |job-submit-failed| job submission failed + expired |task-expired| will not submit job (too far behind) + ============== ==================== =================== ==================================== + +The running task icon contains a clock face which shows the time elapsed +as a proportion of the average runtime. + +.. image:: ../../img/task-job-icons/task-running-0.png + :width: 50px + :height: 50px + :align: left + +.. image:: ../../img/task-job-icons/task-running-25.png + :width: 50px + :height: 50px + :align: left + +.. image:: ../../img/task-job-icons/task-running-50.png + :width: 50px + :height: 50px + :align: left + +.. image:: ../../img/task-job-icons/task-running-75.png + :width: 50px + :height: 50px + :align: left + +.. image:: ../../img/task-job-icons/task-running-100.png + :width: 50px + :height: 50px + :align: left + +.. NOTE: these pipe characters are functional! They create a line break. + +| + +| + + +Task Modifiers +-------------- + +Tasks are run as soon as their dependencies are satisfied, however, there are +some other conditions which can prevent tasks from being run. These are +given "modifier" icons which appear to the top-left of the task icon: + +.. list-table:: + :class: grid-table + :align: left + :widths: 20, 80 + + * - .. image:: ../../img/task-job-icons/task-isHeld.png + :width: 60px + :height: 60px + - **Held:** Task has been manually :term:`held ` back from + running. + * - .. image:: ../../img/task-job-icons/task-isRunahead.png + :width: 60px + :height: 60px + - **Runahead:** Task is held back by the :term:`runahead limit`. + * - .. image:: ../../img/task-job-icons/task-isQueued.png + :width: 60px + :height: 60px + - **Queued:** Task has been held back by an :term:`internal queue`. + + + .. _n-window: -Tasks in the UI -=============== +The "n" Window +-------------- .. versionchanged:: 8.0.0 Cylc workflow :term:`graphs ` can be very large, even infinite for :term:`cycling workflows ` with no :term:`final cycle point`. -Consequently the UI often can't display "all of the tasks" at once. Instead, it -displays tasks that belong to a graph-based :term:`window` extending a -configured number of graph edges out from active tasks. -The ``n = 0`` or *active task* window includes: +Consequently the GUI often can't display "all of the tasks" at once. Instead +it displays all :term:`active tasks ` (e.g. running tasks) +as well as any tasks which are a configurable number of tasks away from +them in the task dependency :term:`graph`. -- ``preparing`` tasks -- ``submitted`` and ``running`` tasks - i.e. those with active jobs -- ``waiting`` tasks, that are :term:`actively waiting ` on: +.. image:: ../../img/n-window.png + :align: center - - :ref:`clock triggers ` - - :ref:`external triggers
` - - :ref:`internal queues ` - - :ref:`runahead limit ` -- finished tasks retained as *incomplete*, in expectation of user intervention: +n=0: + The ``n=0`` window contains :term:`active tasks `. An active + task is a task which is near ready to run, in the process of running, or + which requires user intervention (see the :term:`glossary ` + for a more detailed description). +n=1: + The ``n=1`` window contains all "active tasks" as well as any tasks one + "edge" out from them, i.e. their dependencies (the tasks that come immediately + before them in the graph) and their descendants (the tasks that come + immediately after them in the graph). +n=2: + The ``n=2`` window contains all "active tasks" as well as any tasks *two* + edges out from them, and so on. - - ``submit-failed`` tasks, if successful submission was not :term:`optional - ` - - ``succeeded`` or ``failed`` tasks that did not complete :term:`required - outputs ` +This animation shows how the n-window advances as a workflow runs, tasks are +colour coded according to their n-window value with the colours changing from +``n=0`` (blue) to ``n=8`` (pink): -The default window extent is ``n = 1``, i.e. tasks out to one graph edge from -current active tasks. +.. image:: ../../img/n-window.gif + :align: center -Tasks ahead of the ``n=0`` window are displayed by the UI as ``waiting`` but -the scheduler is not actively managing them yet. +| -.. image:: ../../img/n-window.png - :align: center +By default the GUI/Tui displays the ``n=1`` window. You can change this using +the "Set Graph Window Extent" command which is currently only available in the +GUI. .. note:: - The graph-based active task window is intended to display the essential - information without overwhelming the UI. Even very large workflows - typically do not have a massive number of tasks active at once. + The "graph window extent" is a property of the workflow not a property of + the GUI so persists between sessions. Better visibility and easier control + over the n-window are planned in future releases of Cylc. + +.. warning:: + + High "graph window extent" values can cause a Cylc scheduler and the GUI + to run slowly. diff --git a/src/user-guide/writing-workflows/scheduling.rst b/src/user-guide/writing-workflows/scheduling.rst index f8f018587e..19961d3188 100644 --- a/src/user-guide/writing-workflows/scheduling.rst +++ b/src/user-guide/writing-workflows/scheduling.rst @@ -2002,103 +2002,137 @@ When using message triggers in this way there are two things to be aware of: Check that you understand how your tasks work, if they use custom outputs. -Limiting Workflow Activity --------------------------- - -Cylc will usually try to trigger any task with met dependencies. If this -risks running more tasks than you wish - if it would overwhelm the job -platform for example - :ref:`RunaheadLimit` and :ref:`InternalQueues` provide -tools to limit workflow activity. - .. _RunaheadLimit: Runahead Limiting -^^^^^^^^^^^^^^^^^ +----------------- + +Runahead limiting prevents a workflow from getting too far ahead of the oldest +active cycle point by holding back tasks in cycles beyond a specified limit. + +The runahead limit is defined as an interval measured from the oldest active cycle. +A cycle is considered to be "active" if it contains any :term:`active` tasks +(e.g. running tasks). + +Tasks in cycles which are beyond the limit are called :term:`runahead` tasks +and are displayed in the GUI/Tui with small circle above them: + +.. image:: ../../img/task-job-icons/task-isRunahead.png + :width: 60px + :height: 60px -The scheduler runahead limit determines how many consecutive cycle points can -be active at once. The base point of the runahead calculation is the current -lowest-valued point with :term:`active` or :term:`incomplete` tasks present. +As the workflow advances and active cycles complete, the runahead limit moves +forward allowing tasks in later cycles to run. -The runahead limit can be set with :cylc:conf:`[scheduling]runahead limit`, -to an integer interval (for datetime or integer cycling workflows) or to -a datetime interval (datetime cycling only). +There are two ways of defining the interval which defines the runahead limit, +as an integer number of cycles, or as a datetime interval. -The integer format ``Pn`` is an interval that spans ``n+1`` consecutive cycle -points, regardless of how the cycle point values increment from one point to -the next (i.e. regardless of the cycling interval). -For example, the default runahead limit of ``P4`` allows 5 active cycle points -(base point plus the 4 next points) whether the cycling is defined by -``R/^/P1`` or ``R/^/P3`` etc. (for integer cycling) or ``R/^/PT6H`` etc. (for -datetime cycling). +Integer Format +^^^^^^^^^^^^^^ + +The runahead limit can be defined as an integer interval with the format +``P``, where ``N`` is an integer. -In the following example, tasks ``1/foo`` through ``4/foo`` will submit to -run immediately at start-up, but ``5/foo`` will be held back until ``1/foo`` -finishes and allows the runahead limit to move on: +For example the default runahead limit is ``P4`` (an interval of four cycles), +which means that up to five cycles may be active simultaneously, the oldest +active cycle and the next four after it. + +E.G. for this example workflow: .. code-block:: cylc [scheduling] cycling mode = integer initial cycle point = 1 - runahead limit = P3 # max 4 active points + runahead limit = P4 # max 5 active points (the default) [[graph]] - P2 = foo # cycle points 1, 3, 5, 7, 9, ... + P1 = foo + +When this workflow starts, the initial cycle point is 1 and the runahad limit +is four cycles after this (i.e. cycle 4). So the task ``foo`` will immediately +submit in cycles 1, 2, 3 and 4, however, the tasks in cycles 5 onwards will +wait until earlier cycles complete, and the runahead limit advances. -The active cycle points in this workflow, at some point during the run, -might be ``29, 31, 33, 35``, for example. +* 1 |task-submitted| - **initial cycle point** +* 2 |task-submitted| +* 3 |task-submitted| +* 4 |task-submitted| - **runahead limit** +* 5 |task-runahead-super| (held back by runahead limit) +* 6 |task-runahead-super| (held back by runahead limit) +* X |task-runahead-super| (held back by runahead limit) + +As the workflow advances and earlier cycles complete, the runahead limit +moves on. E.G. Once the cycles 1 & 2 have completed, the runahead limit will +advance to cycle 6. + +The integer format counts the number of cycles irrespective of the cycling +interval, so if we change the cycling interval from ``P1`` to ``P2Y``: .. code-block:: cylc + :emphasize-lines: 2,5 [scheduling] - initial cycle point = 2050 - runahead limit = P3 # max 4 active points + initial cycle point = 2000 # date time cycling + runahead limit = P4 # max 5 active points [[graph]] - P2Y = foo # cycle points 2050, 2052, 2054, ... + P2Y = foo # cycle points 1, 3, 5, 7, 9, ... -The active points in this workflow, at some point during the run, might be -``2060, 2062, 2064, 2068``. +Then, the task ``foo`` would submit immediately in the cycles 1, 3, 5 and 7. +Cycles from 9 onwards will be held back. -With a datetime interval, on the other hand, the number of active cycle points -depends on the cycling intervals: +* 2000 |task-submitted| - **initial cycle point** +* 2002 |task-submitted| +* 2004 |task-submitted| +* 2006 |task-submitted| - **runahead limit** +* 2008 |task-runahead-super| (held back by runahead limit) +* 2010 |task-runahead-super| (held back by runahead limit) +* XXXX |task-runahead-super| (held back by runahead limit) + + +Datetime Format +^^^^^^^^^^^^^^^ + +The runahead interval can also be specified as an :term:`ISO8601 duration`. +This approach *does* depend on the cycling intervals, e.g: .. code-block:: cylc + :emphasize-lines: 3 [scheduling] - initial cycle point = 2050 + initial cycle point = 2000 runahead limit = P4Y # max active point: base point + P4Y [[graph]] P2Y = foo # cycle points 2050, 2052, 2054, ... -The active points in this workflow, at some point during the run, might be -``2060, 2062, 2064``. If the cycling interval was ``P1Y`` instead of ``P2Y``, -they might be ``2060, 2061, 2062, 2063, 2064``. - - -.. note:: +When this workflow starts, the task foo in the first three cycles will run: - To restrict activity to a single cycle point at a time (just the base point) - use a null runahead interval: ``P0`` or (e.g.) ``PT0H``. +* 2000 |task-submitted| - **initial cycle point** +* 2002 |task-submitted| +* 2004 |task-submitted| - **runahead limit** +* 2006 |task-runahead-super| (held back by runahead limit) +* 2008 |task-runahead-super| (held back by runahead limit) +* XXXX |task-runahead-super| (held back by runahead limit) -.. note:: - - Runahead limiting does not restrict activity within a cycle point. - Workflows with a large number of tasks per cycle may need :ref:`internal - queues ` to constrain activity in absolute terms. +Runahead Limit Notes +^^^^^^^^^^^^^^^^^^^^ +To restrict activity to a single cycle point at a time (just the base point) +use a null runahead interval: ``P0`` or (e.g.) ``PT0H``. -.. note:: +Runahead limiting does not restrict activity within a cycle point. +Workflows with a large number of tasks per cycle may need :ref:`internal +queues ` to constrain activity in absolute terms. - The scheduler may automatically raise the runahead limit to accommodate - :term:`future triggered` tasks without stalling - the workflow. +The scheduler may automatically raise the runahead limit to accommodate +:term:`future triggered` tasks without stalling the workflow. .. _InternalQueues: Internal Queues -^^^^^^^^^^^^^^^ +--------------- Large workflows can potentially overwhelm the system by submitting too many jobs at once. Internal queues can prevent this by limiting the number of diff --git a/src/user-guide/writing-workflows/suicide-triggers.rst b/src/user-guide/writing-workflows/suicide-triggers.rst index 3f2fbc9d38..4583558d88 100644 --- a/src/user-guide/writing-workflows/suicide-triggers.rst +++ b/src/user-guide/writing-workflows/suicide-triggers.rst @@ -52,12 +52,8 @@ To remove a task after any one of several events, use an OR operator: .. note:: - * There's no point removing tasks that are not in the ``waiting`` state - * In Cylc 8 all waiting tasks in front of the active window are virtual and - don't need to be "removed" - * The only non-virtual waiting tasks in Cylc 8 are those :term:`actively - waiting ` on an external trigger; these might need to - be removed if they will never run + At present, only :term:`active tasks ` that are waiting + (e.g. queued tasks) can be removed by ``cylc remove``. .. _remaining-use-cases: @@ -65,7 +61,7 @@ To remove a task after any one of several events, use an OR operator: Remaining Use Case ------------------ -Suicide triggers may be needed to remove an :term:`active-waiting task` when it +Suicide triggers may be needed to remove a waiting :term:`active task` when it can be inferred from the status of another task that the external trigger will never be satisfied.