[dagster-dbt] Add Profiles Directory to prepare_manifest

[bkimjin]

Summary & Motivation

Every time I ran dagster dev, there were errors on load, even though access to Dagster was fine. Upon research, it appears that Dagster is not using the environment variable DBT_PROFILES_DIR when preparing the manifest.

How I Tested These Changes

Fed the profiles_dir argument with the os environment variable, and ran dagster dev. Below is the log without the fix (modified some string values):

dagster dev
2025-01-07 15:22:33 -0800 - dagster - INFO - Launching Dagster services...
2025-01-07 15:22:35 -0800 - dagster.builtin - WARNING - An error was encountered when creating a handle to the dbt adapter in Dagster.
Traceback (most recent call last):
  File "/[email protected]/lib/python3.10/site-packages/dagster_dbt/core/resource.py", line 677, in cli
    adapter = self._initialize_adapter(cli_vars)
  File "/[email protected]/lib/python3.10/site-packages/dagster_dbt/core/resource.py", line 366, in _initialize_adapter
    profile = load_profile(self.project_dir, cli_vars, self.profile, self.target)
  File "/[email protected]/lib/python3.10/site-packages/dbt/config/runtime.py", line 71, in load_profile
    profile = Profile.render(
  File "/[email protected]/lib/python3.10/site-packages/dbt/config/profile.py", line 403, in render
    return cls.from_raw_profiles(
  File "/[email protected]/lib/python3.10/site-packages/dbt/config/profile.py", line 360, in from_raw_profiles
    raise DbtProjectError("Could not find profile named '{}'".format(profile_name))
dbt.exceptions.DbtProjectError: Runtime Error
  Could not find profile named 'custom_profile_name'
2025-01-07 15:22:43 -0800 - dagster.daemon - INFO - Instance is configured with the following daemons: ['AssetDaemon', 'BackfillDaemon', 'SchedulerDaemon', 'SensorDaemon']
2025-01-07 15:22:43 -0800 - dagster-webserver - INFO - Serving dagster-webserver on http://127.0.0.1:3000 in process 4397

And here's the output with the fix:

dagster dev
2025-01-07 15:21:40 -0800 - dagster - INFO - Launching Dagster services...
2025-01-07 15:21:52 -0800 - dagster.daemon - INFO - Instance is configured with the following daemons: ['AssetDaemon', 'BackfillDaemon', 'SchedulerDaemon', 'SensorDaemon']
2025-01-07 15:21:55 -0800 - dagster-webserver - INFO - Serving dagster-webserver on http://127.0.0.1:3000 in process 3339

Changelog

• Add profiles directory in preparing manifest to reduce profile not found error

[maximearmstrong]

Hey @bkimjin - thanks for your contribution. In terms of design, this change will require more that supporting the DBT_PROFILES_DIR env var in _prepare_manifest.

This issue is related to this problem.

I will work on a fix.