-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[components] Remove translator terminology from SlingReplicationCollectionComponent #26927
[components] Remove translator terminology from SlingReplicationCollectionComponent #26927
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What would the update to the README (in Notion) look like?
e.g. right now we have
type: dagster_components.dbt_project
params:
dbt:
project_dir: ../../../dbt/jdbt
translator:
key: "target/main/{{ node.name }}"
at one point.
What does it look like in this new regime?
54fe9ad
to
e533733
Compare
Updated this such that this would end up looking like: type: dagster_components.dbt_project
params:
dbt:
project_dir: ../../../dbt/jdbt
asset_attributes:
key: "target/main/{{ node.name }}" |
e533733
to
65347e4
Compare
22c7cdb
to
f232896
Compare
65347e4
to
2402ae9
Compare
f232896
to
a615bdd
Compare
e556feb
to
f62635e
Compare
a615bdd
to
9dbf553
Compare
f62635e
to
386ea31
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should it just be "transforms" since it might select asset checks only?
sounds good to me |
9dbf553
to
aafd7b9
Compare
386ea31
to
4caa07f
Compare
|
||
|
||
def get_wrapped_translator_class(translator_type: type): | ||
"""Temporary hack to allow wrapping of many methods of a given translator class.""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is it temporary? What is the "permanent" fix?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This does strike me as wacky and something that could end up living for a long time.
I would expect a more "normal" adapter class per component that wraps as instance of the original.
The "heavy lifiting" in this class is _get_rendered_attribute
which could be called as a standalone function
@dataclass
class RenderingInfo:
obj_name: str
asset_attributes: AssetAttributesModel,
value_renderer: TemplatedValueRenderer
def get_rendered_attr(...): ...
def get_asset_key(self, obj, default) -> AssetKey:
return self.get_rendered_attr("key", obj, default)
class RenderingSlingTranslator(SlingTranslator)
rendering_info: RenderingInfo
wrapped_translator: SlingTranslator
def get_asset_key(self, obj) -> AssetKey:
return self.rendering_info.get_asset_key(obj, self.wrapper_translator)
Something like this is a little more code but I think much more obvious
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dynamically created class seems too magical
aafd7b9
to
28c32aa
Compare
4caa07f
to
25b8727
Compare
added explanation, refactored the implementation to be "ready for the future"
28c32aa
to
726490d
Compare
25b8727
to
eb16a9d
Compare
726490d
to
8d1f82c
Compare
eb16a9d
to
7be98c0
Compare
7be98c0
to
bba23e4
Compare
…ctionComponent (dagster-io#26927) ## Summary & Motivation The translator language was fairly confusing. Went through some iterations but arrived in a place where the translation layer can be overridden in yaml using the `asset_attributes` field. This field will have the domain-specific object in scope, meaning you can use that to set attributes if desired. At the same time, this leaves open the ability for users to specify a python-based translator to override the default one. Made the following changes: 1. Because we want to use the key `asset_attributes` to replace `translator`, this conflicts with the global `asset_attributes` field, so I renamed this to be `asset_transforms`. This gives a better sense of the fact that you're doing specific operations over all assets (imo) 2. Removed the weird subclassing stuff with the existing asset transforms -- this was only necessary in a world where merge / replace supported different argument sets, but this is no longer the case so the extra complexity was unnecessary 3. Added a generic wrapper layer to make it easier to create these sorts of translator wrappers that incorporate both a base translator as well as jinja templating ## How I Tested These Changes ## Changelog NOCHANGELOG
Summary & Motivation
The translator language was fairly confusing. Went through some iterations but arrived in a place where the translation layer can be overridden in yaml using the
asset_attributes
field. This field will have the domain-specific object in scope, meaning you can use that to set attributes if desired.At the same time, this leaves open the ability for users to specify a python-based translator to override the default one.
Made the following changes:
asset_attributes
to replacetranslator
, this conflicts with the globalasset_attributes
field, so I renamed this to beasset_transforms
. This gives a better sense of the fact that you're doing specific operations over all assets (imo)How I Tested These Changes
Changelog
NOCHANGELOG