Skip to content

Commit

Permalink
styles
Browse files Browse the repository at this point in the history
  • Loading branch information
prha committed Aug 27, 2024
1 parent 95cd32c commit fbafc9f
Showing 1 changed file with 7 additions and 7 deletions.
14 changes: 7 additions & 7 deletions docs/docs-beta/docs/guides/external-systems/io-managers.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: "Managing stored data with I/O Managers"
title: "Managing stored data with I/O managers"
sidebar_position: 50
sidebar_label: "Storing data with I/O Managers"
sidebar_label: "Storing data with I/O managers"
---

I/O Managers in Dagster provide a way to separate the code that's responsible for logical data transformation from the code that's responsible for reading and writing the results. This can help reduce boiler plate code and make it easy to swap out where your data is stored.
I/O managers in Dagster provide a way to separate the code that's responsible for logical data transformation from the code that's responsible for reading and writing the results. This can help reduce boiler plate code and make it easy to swap out where your data is stored.

<details>
<summary>Prerequisites</summary>
Expand All @@ -13,7 +13,7 @@ I/O Managers in Dagster provide a way to separate the code that's responsible fo
- Familiarity with [Resources](/concepts/resources)
</details>

## Extract I/O logic from an asset into a reuseable I/O Manager
## Extract I/O logic from an asset into a reusable I/O manager

In many Dagster pipelines, assets can be broken down as the following steps:
1. reading data from some data store into memory
Expand All @@ -28,7 +28,7 @@ For example, both assets in the code below are constructing a DuckDB connection

To switch to using I/O managers, we can use the DuckDB / Pandas I/O manager provided by the `dagster_duckdb_pandas` package. The I/O manager will be used to read and write data instead of the `DuckDBResource`.

To load an upstream asset using an I/O manager, that asset is specified as an input parameter to the asset function rather than within the `deps` list on the `@asset` decorator. The `DuckDBPandasIOManager` reads the DuckDB table with the same name as the upstream asset and passes the data into the asset function as a Pandas dataframe.
To load an upstream asset using an I/O manager, that asset is specified as an input parameter to the asset function rather than within the `deps` list on the `@asset` decorator. The `DuckDBPandasIOManager` reads the DuckDB table with the same name as the upstream asset and passes the data into the asset function as a Pandas DataFrame.

To store data using an I/O manager, return the data in the asset function. The data returned must be a valid type. The `DuckDBPandasIOManager` accepts Pandas data frames and writes them to DuckDB as a table with the same name as the asset.

Expand All @@ -39,9 +39,9 @@ Refer to the individual I/O manager documentation for details on valid types and

## Swapping data stores

With I/O managers, swapping data stores consists of changing the implementation of the I/O manager resource. The asset definitions, which only contain transformational logic, will not need to be changed.
With I/O managers, swapping data stores consists of changing the implementation of the I/O manager resource. The asset definitions, which only contain transformational logic, won't need to change.

<CodeExample filePath="guides/external-systems/assets-with-snowflake-io-manager.py" language="python" title="Asset with Snowflake I/O manager" />

Dagster offers a number of [library implementations for I/O managers] - TODO ADD LINK.
Dagster offers built-in [library implementations for I/O managers](/todo) for popular data stores and in-memory formats.

0 comments on commit fbafc9f

Please sign in to comment.