Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New docs fixes #27240

Merged
merged 20 commits into from
Jan 22, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
[!IMPORTANT]
This is the directory for the legacy documentation. Please make documentation changes to the [docs-beta directory](https://github.com/dagster-io/dagster/blob/master/docs/docs-beta/README.md) instead.

# Running the docs, v2

> **Submitting a pull request to update the docs?** If so, please verify that the updates follow the [docs style checklist](https://github.com/dagster-io/dagster/blob/master/docs/DOC_CHECKLIST.md).
Expand Down
2 changes: 1 addition & 1 deletion docs/docs-beta/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Dagster Docs - Beta
# Dagster Docs

This is the home of the Dagster documentation. The documentation site is built using [Docusaurus](https://docusaurus.io/), a modern static website generator.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,14 +1,8 @@
---
title: "Deployment settings"
title: "Full deployment settings"
sidebar_position: 200
---

# Dagster+ deployment settings reference

:::note
This article is applicable to Dagster+.
:::

This reference describes the settings that can be configured for full deployments in Dagster+.

Refer to the [Managing deployments in Dagster+ guide](managing-deployments) for information about configuring settings in the Dagster+ interface or using the `dagster-cloud` CLI.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,15 @@
---
title: "Deployments"
title: "Full deployments"
sidebar_position: 50
---

In Dagster+, there are two types of deployments:

- [**Branch deployments**](/dagster-plus/features/ci-cd/branch-deployments), which are temporary deployments built for testing purposes.
- **Full deployments**, which are persistent, fully-featured deployments intended to perform actions on a recurring basis.

This section focuses on **full deployments**, hereafter referred to as "deployments".

import DocCardList from '@theme/DocCardList';

<DocCardList />
Original file line number Diff line number Diff line change
@@ -1,31 +1,23 @@
---
title: "Managing deployments in Dagster+"
title: "Managing full deployments in Dagster+"
---

:::note
Full deployments are standalone environments, allowing you to operate independent instances of Dagster with separately managed permissions.

This guide is applicable to Dagster+.
When a Dagster+ organization is created, a single deployment named `prod` will also be created. To create additional full deployments, you must sign up for a [Pro plan](https://dagster.io/pricing).

:::
Each full deployment can have one or multiple [code locations](/dagster-plus/deployment/code-locations).

:::note Full deployments vs branch deployments

In Dagster+, there are two types of deployments:

- [**Branch deployments**](/dagster-plus/features/ci-cd/branch-deployments), which are temporary deployments built for testing purposes.
- [**Branch deployments**](/dagster-plus/features/ci-cd/branch-deployments), which are temporary deployments built for testing purposes. We recommend using branch deployments to test your changes, even if you're able to create additional deployments. Branch deployments are available for all Dagster+ users, regardless of plan.
- **Full deployments**, which are persistent, fully-featured deployments intended to perform actions on a recurring basis.

This guide will focus on **full deployments**, hereafter referred to simply as deployments.

## About deployments

Deployments are standalone environments, allowing you to operate independent instances of Dagster with separately managed permissions.
This guide focuses on **full deployments**, hereafter referred to as **deployments**.

When a Dagster+ organization is created, a single deployment named `prod` will also be created. To create additional deployments, a [Pro plan](https://dagster.io/pricing) is required.

Each deployment can have one or multiple [code locations](/dagster-plus/deployment/code-locations).

**Concerned about testing environments?** We recommend using Branch Deployments to test your changes, even if you're able to create additional deployments. Branch deployments are available for all Dagster+ users, regardless of plan.

Refer to the [Branch Deployment docs](/dagster-plus/features/ci-cd/branch-deployments) for more info, or the [Testing against production using Branch Deployments guide](/dagster-plus/features/ci-cd/branch-deployments/testing) for a real-world example.
:::

## Viewing and switching deployments

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Environment variables
sidebar_position: 30
sidebar_position: 20
---

Environment variables, which are key-value pairs configured outside your source code, allow you to dynamically modify application behavior depending on environment.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Managing compute logs and error messages
sidebar_position: 50
sidebar_position: 200
---

import ThemedImage from '@theme/ThemedImage';
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
title: "Managing multiple projects and teams with Dagster+ Hybrid"
description: "How to set up and manage multiple projects with Dagster+ Hybrid deployments."
sidebar_position: 300
---


Expand All @@ -21,122 +22,10 @@ If you want to manage complexity or divide your work into areas of responsibilit

Refer to the following table for more information, including the pros and cons of each approach.

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "14%",
}}
>
Approach
</th>
<th
style={{
width: "43%",
}}
>
Multiple directories in a single repository
</th>
<th>Multiple repositories</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<strong>How it works</strong>
</td>
<td>
You can use a single repository to manage multiple projects by placing
each project in a separate directory. Depending on your VCS, you may be
able to set{" "}
<a href="https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners">
code owners
</a>{" "}
to restrict who can modify each project.
</td>
<td>
For stronger isolation, you can use multiple repositories to manage
multiple projects.
</td>
</tr>
<tr>
<td>
<strong>Pros</strong>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Simple to implement
</li>
<li>Facilitates code sharing between projects</li>
</ul>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Stronger isolation between projects and teams
</li>
<li>
Each project has its own CI/CD pipeline and be deployed
independently
</li>
<li>Dependencies between projects can be managed independently</li>
</ul>
</td>
</tr>
<tr>
<td>
<strong>Cons</strong>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
All projects share the same CI/CD pipeline and cannot be deployed
independently
</li>
<li>
Shared dependencies between projects may cause conflicts and require
coordination between teams
</li>
</ul>
</td>
<td>
Code sharing between projects require additional coordination to publish
and reuse packages between projects
</td>
</tr>
</tbody>
</table>
| Approach | How it works | Pros | Cons |
|----------|--------------|------|------|
| **Multiple directories in a single repository** | You can use a single repository to manage multiple projects by placing each project in a separate directory. Depending on your VCS, you may be able to set code owners to restrict who can modify each project. | <ul><li>Simple to implement</li><li>Facilitates code sharing between projects</li></ul> | <ul><li>All projects share the same CI/CD pipeline and cannot be deployed independently</li><li>Shared dependencies between projects may cause conflicts and require coordination between teams</li></ul> |
| **Multiple repositories** | For stronger isolation, you can use multiple repositories to manage multiple projects. | <ul><li>Stronger isolation between projects and teams</li><li>Each project has its own CI/CD pipeline and be deployed independently</li><li>Dependencies between projects can be managed independently</li></ul> | Code sharing between projects require additional coordination to publish and reuse packages between projects. |

### Deployment configuration

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Dagster+ rate limits
sidebar_position: 60
sidebar_position: 100
---

Dagster+ enforces several rate limits to smoothly distribute the load. Deployments are limited to:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Tokens
sidebar_position: 20
sidebar_position: 30
---

import DocCardList from '@theme/DocCardList';
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ To create a new catalog view, you have two options:

![Screenshot of new catalog view modal](/images/dagster-plus/features/asset-catalog/new-catalog-view.png)

Give your view a name and optionally a description and icon. Next, you can select one or more filters to apply to your view by clicking the **Add filters** button. Filters can select a subset of assets based on their [metadata]/guides/build/assets/metadata-and-tags/organizing-assets-with-tags-and-metadata), tags, kinds, owners, asset groups, or other properties.
Give your view a name and optionally a description and icon. Next, you can select one or more filters to apply to your view by clicking the **Add filters** button. Filters can select a subset of assets based on their [metadata]/guides/build/assets/metadata-and-tags/), tags, kinds, owners, asset groups, or other properties.

### Creating a new catalog view from your current set of filters

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ def fruits_in_stock(): ...
### Metadata

{/* Change Tracking can detect when an [asset's definition metadata](/concepts/metadata-tags/asset-metadata#attaching-definition-metadata) has changed, whether it's been added, modified, or removed. */}
Change Tracking can detect when an [asset's definition metadata](/guides/build/assets/metadata-and-tags/organizing-assets-with-tags-and-metadata) has changed, whether it's been added, modified, or removed.
Change Tracking can detect when an [asset's definition metadata](/guides/build/assets/metadata-and-tags/) has changed, whether it's been added, modified, or removed.

<Tabs>
<TabItem value="Asset in the Dagster UI">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ You'll need one or more assets that emit the same metadata key at run time. Insi
are most valuable when you have multiple assets that emit the same kind of metadata, such as
such as the number of rows processed or the size of a file uploaded to object storage.

Follow [the metadata guide](/guides/build/assets/metadata-and-tags/organizing-assets-with-tags-and-metadata#runtime-metadata) to add numeric metadata
Follow [the metadata guide](/guides/build/assets/metadata-and-tags/index.md#runtime-metadata) to add numeric metadata
to your asset materializations.

## Step 2: Enable viewing your metadata in Dagster+ Insights
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
---
title: Using resources in sensors
sidebar_position: 100
unlisted: true
---

Dagster's [resources](/guides/build/external-resources/) system can be used with sensors to make it easier to call out to external systems and to make components of a sensor easier to plug in for testing purposes.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,6 @@ When an asset with a code version is materialized, the generated `AssetMateriali

## Next steps

- Enrich Dagster's built-in data catalog with [asset metadata]/guides/build/assets/metadata-and-tags/organizing-assets-with-tags-and-metadata)
- Enrich Dagster's built-in data catalog with [asset metadata]/guides/build/assets/metadata-and-tags/)
- Learn to [pass data between assets](/guides/build/assets/passing-data-between-assets)
- Learn to use a [factory pattern](/guides/build/assets/creating-asset-factories) to create multiple, similar assets
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Asset observations"
description: Dagster provides functionality to record metadata about assets.
sidebar_position: 400
sidebar_position: 500
---

An asset observation is an event that records metadata about a given asset. Unlike asset materializations, asset observations do not signify that an asset has been mutated.
Expand Down
Loading
Loading