[dagster-aws] extend main ruff config + move boto3-stubs to stubs extras

[danielgafni]

Summary & Motivation

1. We currently have boto3-stubs-lite in dagster-aws runtime dependencies. This might be causing type-checkers to fail for our users if they have "bad" boto3 code (which previously wasn't checked because no stubs package was installed). This PR moves boto3-stubs-lite to a new stubs extras of dagster-aws, ensuring it won't be installed by default. It's now only imported under if TYPE_CHECKING blocks. Packages like mypy_boto3_glue were added to banned top-level modules (via ruff rule).

2. Additionally, dagster-aws's ruff config now extends the root config. This mostly fixed import ordering, previously dagster_aws imports weren't coming after all other packages.

How I Tested These Changes

Changelog [New | Bug | Docs]

NOCHANGELOG

[danielgafni]

• [dagster-aws] extend main ruff config + move boto3-stubs to stubs extras #24153 👈
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @danielgafni and the rest of your teammates on Graphite

[danielgafni]

pyright failing because of broken master (will try to rebase, seems to be fixed now)

[schrockn]

Hmm this seems complicated.

The errors that users would see are not phantom errors, correct? They would be real issues?

I don't think this is an unreasonable change, and users can easily ignore it with type: ignore.

Is this in response to user feedback?

[danielgafni]

The errors that users would see are not phantom errors, correct? They would be real issues?

Well, kinda, but not exactly. But there are 2 arguments against them:

• This will break existing builds for dagster-aws users once they update to a newer version. The errors will be popping out in random places, anywhere where boto3 is used, even outside of Dagster code. It might be confusing for users unaware of boto3-stubs.
• The number of errors might be overwhelming. boto3-stubs is really strict (and probably that's technically correct and aligned to AWS API specs). For example, most of response fields are set to NotRequired, so the user must do response.get("field") instead of response["field"]. And this might be a problem, because pyright doesn't have a way to automatically insert # type: ignore comments (if you remember, I made a script for that). Because vanilla boto3 is basically untyped, old codebases will be drowning in sudden type-checking errors.

In conclusion, this doesn't seem right to me, it should be a conscious decision to enable type-checking for boto3 on the user side. We shouldn't be forcing best-practices onto users.

Is this in response to user feedback?

No, but I am confident the users are going to complain about it

[schrockn]

Ah it will be break all all boto3 API calls. Yes then I agree this is too aggressive.

I just don't know if this is worth all the trouble and complexity at this point. How many people are going to go through the trouble of installing the appropriate extra? Let's discuss in our meeting today.

[schrockn]

Per offline discussion let's name the extra [stubs] and this is useful for our own internal development. Let's rename it and move forward.

[danielgafni]

@schrockn done