dagster-io · neverett · Jan 22, 2025 · Jan 21, 2025 · Jan 21, 2025 · Jan 21, 2025
@@ -1,3 +1,6 @@
+[!IMPORTANT]
+This is the directory for the legacy documentation. Please make documentation changes to the [docs-beta directory](https://github.com/dagster-io/dagster/blob/master/docs/docs-beta/README.md) instead.
+
 # Running the docs, v2
 
 > **Submitting a pull request to update the docs?** If so, please verify that the updates follow the [docs style checklist](https://github.com/dagster-io/dagster/blob/master/docs/DOC_CHECKLIST.md).

@@ -1,4 +1,4 @@
-# Dagster Docs - Beta
+# Dagster Docs
 
 This is the home of the Dagster documentation. The documentation site is built using [Docusaurus](https://docusaurus.io/), a modern static website generator.
 

@@ -1,14 +1,8 @@
 ---
-title: "Deployment settings"
+title: "Full deployment settings"
 sidebar_position: 200
 ---
 
-# Dagster+ deployment settings reference
-
-:::note
-This article is applicable to Dagster+.
-:::
-
 This reference describes the settings that can be configured for full deployments in Dagster+.
 
 Refer to the [Managing deployments in Dagster+ guide](managing-deployments) for information about configuring settings in the Dagster+ interface or using the `dagster-cloud` CLI.

@@ -1,7 +1,15 @@
 ---
-title: "Deployments"
+title: "Full deployments"
+sidebar_position: 50
 ---
 
+In Dagster+, there are two types of deployments:
+
+- [**Branch deployments**](/dagster-plus/features/ci-cd/branch-deployments), which are temporary deployments built for testing purposes.
+- **Full deployments**, which are persistent, fully-featured deployments intended to perform actions on a recurring basis.
+
+This section focuses on **full deployments**, hereafter referred to as "deployments".
+
 import DocCardList from '@theme/DocCardList';
 
 <DocCardList />
@@ -1,31 +1,23 @@
 ---
-title: "Managing deployments in Dagster+"
+title: "Managing full deployments in Dagster+"
 ---
 
-:::note
+Full deployments are standalone environments, allowing you to operate independent instances of Dagster with separately managed permissions.
 
-This guide is applicable to Dagster+.
+When a Dagster+ organization is created, a single deployment named `prod` will also be created. To create additional full deployments, you must sign up for a [Pro plan](https://dagster.io/pricing).
 
-:::
+Each full deployment can have one or multiple [code locations](/dagster-plus/deployment/code-locations).
+
+:::note Full deployments vs branch deployments
 
 In Dagster+, there are two types of deployments:
 
-- [**Branch deployments**](/dagster-plus/features/ci-cd/branch-deployments), which are temporary deployments built for testing purposes.
+- [**Branch deployments**](/dagster-plus/features/ci-cd/branch-deployments), which are temporary deployments built for testing purposes. We recommend using branch deployments to test your changes, even if you're able to create additional deployments. Branch deployments are available for all Dagster+ users, regardless of plan.
 - **Full deployments**, which are persistent, fully-featured deployments intended to perform actions on a recurring basis.
 
-This guide will focus on **full deployments**, hereafter referred to simply as deployments.
-
-## About deployments
-
-Deployments are standalone environments, allowing you to operate independent instances of Dagster with separately managed permissions.
+This guide focuses on **full deployments**, hereafter referred to as **deployments**.
 
-When a Dagster+ organization is created, a single deployment named `prod` will also be created. To create additional deployments, a [Pro plan](https://dagster.io/pricing) is required.
-
-Each deployment can have one or multiple [code locations](/dagster-plus/deployment/code-locations).
-
-**Concerned about testing environments?** We recommend using Branch Deployments to test your changes, even if you're able to create additional deployments. Branch deployments are available for all Dagster+ users, regardless of plan.
-
-Refer to the [Branch Deployment docs](/dagster-plus/features/ci-cd/branch-deployments) for more info, or the [Testing against production using Branch Deployments guide](/dagster-plus/features/ci-cd/branch-deployments/testing) for a real-world example.
+:::
 
 ## Viewing and switching deployments
 

@@ -1,6 +1,6 @@
 ---
 title: Environment variables
-sidebar_position: 30
+sidebar_position: 20
 ---
 
 Environment variables, which are key-value pairs configured outside your source code, allow you to dynamically modify application behavior depending on environment.

@@ -1,6 +1,6 @@
 ---
 title: Managing compute logs and error messages
-sidebar_position: 50
+sidebar_position: 200
 ---
 
 import ThemedImage from '@theme/ThemedImage';

@@ -1,6 +1,7 @@
 ---
 title: "Managing multiple projects and teams with Dagster+ Hybrid"
 description: "How to set up and manage multiple projects with Dagster+ Hybrid deployments."
+sidebar_position: 300
 ---
 
 
@@ -21,122 +22,10 @@ If you want to manage complexity or divide your work into areas of responsibilit
 
 Refer to the following table for more information, including the pros and cons of each approach.
 
-<table
-  className="table"
-  style={{
-    width: "100%",
-  }}
->
-  <thead>
-    <tr>
-      <th
-        style={{
-          width: "14%",
-        }}
-      >
-        Approach
-      </th>
-      <th
-        style={{
-          width: "43%",
-        }}
-      >
-        Multiple directories in a single repository
-      </th>
-      <th>Multiple repositories</th>
-    </tr>
-  </thead>
-  <tbody>
-    <tr>
-      <td>
-        <strong>How it works</strong>
-      </td>
-      <td>
-        You can use a single repository to manage multiple projects by placing
-        each project in a separate directory. Depending on your VCS, you may be
-        able to set{" "}
-        <a href="https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners">
-          code owners
-        </a>{" "}
-        to restrict who can modify each project.
-      </td>
-      <td>
-        For stronger isolation, you can use multiple repositories to manage
-        multiple projects.
-      </td>
-    </tr>
-    <tr>
-      <td>
-        <strong>Pros</strong>
-      </td>
-      <td>
-        <ul
-          style={{
-            marginTop: "0px",
-          }}
-        >
-          <li
-            style={{
-              marginTop: "0px",
-            }}
-          >
-            Simple to implement
-          </li>
-          <li>Facilitates code sharing between projects</li>
-        </ul>
-      </td>
-      <td>
-        <ul
-          style={{
-            marginTop: "0px",
-          }}
-        >
-          <li
-            style={{
-              marginTop: "0px",
-            }}
-          >
-            Stronger isolation between projects and teams
-          </li>
-          <li>
-            Each project has its own CI/CD pipeline and be deployed
-            independently
-          </li>
-          <li>Dependencies between projects can be managed independently</li>
-        </ul>
-      </td>
-    </tr>
-    <tr>
-      <td>
-        <strong>Cons</strong>
-      </td>
-      <td>
-        <ul
-          style={{
-            marginTop: "0px",
-          }}
-        >
-          <li
-            style={{
-              marginTop: "0px",
-            }}
-          >
-            All projects share the same CI/CD pipeline and cannot be deployed
-            independently
-          </li>
-          <li>
-            Shared dependencies between projects may cause conflicts and require
-            coordination between teams
-          </li>
-        </ul>
-      </td>
-      <td>
-        Code sharing between projects require additional coordination to publish
-        and reuse packages between projects
-      </td>
-    </tr>
-  </tbody>
-</table>
+| Approach | How it works | Pros | Cons |
+|----------|--------------|------|------|
+| **Multiple directories in a single repository** | You can use a single repository to manage multiple projects by placing each project in a separate directory. Depending on your VCS, you may be able to set code owners to restrict who can modify each project. | <ul><li>Simple to implement</li><li>Facilitates code sharing between projects</li></ul> | <ul><li>All projects share the same CI/CD pipeline and cannot be deployed independently</li><li>Shared dependencies between projects may cause conflicts and require coordination between teams</li></ul> |
+| **Multiple repositories** | For stronger isolation, you can use multiple repositories to manage multiple projects. | <ul><li>Stronger isolation between projects and teams</li><li>Each project has its own CI/CD pipeline and be deployed independently</li><li>Dependencies between projects can be managed independently</li></ul> | Code sharing between projects require additional coordination to publish and reuse packages between projects |
 
 ### Deployment configuration
 

@@ -1,6 +1,6 @@
 ---
 title: Dagster+ rate limits
-sidebar_position: 60
+sidebar_position: 100
 ---
 
 Dagster+ enforces several rate limits to smoothly distribute the load. Deployments are limited to:

@@ -1,6 +1,6 @@
 ---
 title: Tokens
-sidebar_position: 20
+sidebar_position: 30
 ---
 
 import DocCardList from '@theme/DocCardList';

@@ -1,7 +1,6 @@
 ---
 title: Using resources in sensors
 sidebar_position: 100
-unlisted: true
 ---
 
 Dagster's [resources](/guides/build/external-resources/) system can be used with sensors to make it easier to call out to external systems and to make components of a sensor easier to plug in for testing purposes.

@@ -1,6 +1,6 @@
 ---
 title: "Components"
-sidebar_position: 70
+sidebar_position: 80
 unlisted: true
 ---
 

@@ -1,6 +1,6 @@
 ---
 title: External pipelines
-sidebar_position: 50
+sidebar_position: 60
 ---
 
 Dagster Pipes provides a powerful mechanism for invoking code outside of Dagster, while providing all the benefits of scheduling, reporting, and observability of native Dagster pipelines. While Dagster is written in Python, you can run code in other languages and send information back to Dagster.

@@ -1,6 +1,6 @@
 ---
 title: External resources
-sidebar_position: 20
+sidebar_position: 40
 ---
 
 Dagster resources are objects that provide access to external systems, databases, or services. Resources are used to manage connections to external systems, and are used by Dagster assets and ops. For example, a simple ETL (Extract Transform Load) pipeline fetches data from an API, ingests it into a database, and updates a dashboard. External tools and services this pipeline uses could be:

@@ -1,5 +1,6 @@
 ---
 title: "Graphs"
+sidebar_position: 300
 ---
 
 A graph is a set of interconnected [ops](ops) or sub-graphs. While individual ops typically perform simple tasks, ops can be assembled into a graph to accomplish complex tasks.

@@ -17,6 +17,8 @@ For assets that follow this pattern, an I/O manager can streamline the code that
 
 This article assumes familiarity with: [assets](/guides/build/assets/) and [resources](/guides/build/external-resources/)
 
+:::
+
 ## Before you begin
 
 **I/O managers aren't required to use Dagster, nor are they the best option in all scenarios.** If you find yourself writing the same code at the start and end of each asset to load and store data, an I/O manager may be useful. For example:

@@ -0,0 +1,18 @@
+---
+title: "Jobs"
+sidebar_position: 200
+---
+
+Jobs are the main unit of execution and monitoring in Dagster. They allow you to execute a portion of a graph of [asset definitions](/guides/build/assets/defining-assets) or [ops](/guides/build/ops) based on a schedule or an external trigger.
+
+When a job begins, it kicks off a _run_. A run is a single execution of a job in Dagster. Runs can be launched and viewed in the Dagster UI.
+
+## Benefits
+
+With jobs, you can
+
+* View and launch runs of jobs in the Dagster UI
+* Automate the execution of your Dagster pipelines with [schedules](/guides/automate/schedules/) and [sensors](/guides/automate/sensors/).
+* Attach information using [metadata and tags](/guides/build/assets/metadata-and-tags)
+* Apply custom prioritization rules to how job runs are prioritized and executed if you are using a [run queue](/guides/deploy/execution/run-coordinators)
+* Make your pipelines more efficient if you apply concurrency limits to job runs. For more information, see "[Managing concurrency of Dagster assets, jobs, and Dagster instances](/guides/operate/managing-concurrency)".
@@ -0,0 +1,8 @@
+---
+title: "ML pipelines"
+sidebar_position: 70
+---
+
+import DocCardList from '@theme/DocCardList';
+
+<DocCardList />
-Original file line number
+Diff line change
@@ Expand Up @@
     This article assumes familiarity with: [assets](/guides/build/assets/) and [resources](/guides/build/external-resources/)
+    :::
     ## Before you begin
     **I/O managers aren't required to use Dagster, nor are they the best option in all scenarios.** If you find yourself writing the same code at the start and end of each asset to load and store data, an I/O manager may be useful. For example:
@@ Expand Down @@