Update coding-practice-trips-by-week-asset.md #23819
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adapt the solution to address the requirement of not loading the full data into memory.
graphite-app
bot
added
the
area: dagster-university
graphite-app
bot
requested review from
C00ldudeNoonan
and removed request for
PedramNavid
Graphite Automations"Label and add CE on all Docs" took an action on this PR • (09/28/24)2 reviewers were added and 1 label was added to this PR based on Pedram Navid's automation. |
|
@cmpadden mind taking a look? |
cmpadden
requested review from
dehume
and removed request for
erinkcochran87 and
C00ldudeNoonan
|
Apologies that this PR went a bit stale, @edsoncezar16 . We're going to close the PR, but your suggestion is good. Instead of making the code more complex, we're going to have a follow-up to improve the wording of the lesson itself to hopefully be more clear. Thanks again! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adapt the solution to address the requirement of not loading the full data into memory.
Summary & Motivation
The challenge asks to design the code assuming that the whole trips data does not fit into memory but a week of data does. When fetching data from the database, the previous solution did the good job of filtering week data, but then went on to append this data to a initially empty dataframe. That approach would still find the same memory overflow.
How I Tested These Changes
Defined a trips by week asset in the local project following the Dagster University Course and successfully materialized it.
Changelog [New | Bug | Docs]
NOCHANGELOG