dbt-core version update

[alexjbush-mas]

The version of dbt-core used by the project is fairly old (dbt-core~=1.7.0rc1) and dbt-core 1.8 has been available since May. Could you update the dependencies and release a new version of this adapter?

[WillemLiang]

@alexjbush-mas Have you successfully connected to Lakehouse Spark on this version?
I have not tried the old version of dbt-core; I just upgraded its legacy APIs to version 1.8.9, and it has not been successful yet. It failed to connect to the Livy endpoint each time and raised an SSL verify failure. But project dbt-fabric works well when I use the same MS Fabric Tenant and Subscription. It seems this project also needs to fit the Azure server-side API's change.

[alexjbush-mas]

@WillemLiang I have managed to connect successfully, however I have other issues (1. transient timeouts against the Livy API during model build - roughly 5ish fail to build out 15 - and the retry and timeout settings don't seem to work, 2. each dbt command spins up a new Livy session with about 3ish mins startup time - session reuse between dbt commands would be nice).

I don't see any SSL issue - are you behind a corporate proxy? You might need add the CA for your proxy into the CA bundle: https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification

[WillemLiang]

@alexjbush-mas Thanks for sharing! I created a new virtual environment to test the connection of the old version and found that the problem still exists. After checking the local network environment, I found that it was caused by zero-trust software. Once I closed Zscaler, the connection was successful.
Similarly, I also passed the test on a project manually adapted to version 1.8.9 of dbt core. You can refer to my modifications if you urgently need a new version(It's important to keep the same dbt-core version with dbt-fabric in my case). Here are the details:

mkdir dbt-fabricspark && cd dbt-fabricspark

# create a venv for this project
python -m venv --prompt dbt-fabricspark .dbt-fabricspark

# if you're using anaconda
conda deactivate

# activate python venv
source .dbt-fabricspark/bin/activate

# update pip
pip install --upgrade pip

# clone this adapter, do not just `pip install https://github.com/microsoft/dbt-fabricspark/archive/refs/tags/v1.7.0rc1.zip`
git clone https://github.com/microsoft/dbt-fabricspark.git

# get your Python version, for me is python3.12
PYTHON=$(ls .dbt-fabricspark/lib/)

# manually add this package
cp -R ./dbt-fabricspark/dbt/adapters/fabricspark .dbt-fabricspark/lib/${PYTHON}/site-packages/dbt/adapters/fabricspark
cp -R ./dbt-fabricspark/dbt/include/fabricspark .dbt-fabricspark/lib/${PYTHON}/site-packages/dbt/include/fabricspark

# no use anymore, clean up
rm -rf dbt-fabricspark

modify Python code as below:

file	before	after
connections.py	from dbt.contracts.connection import ConnectionState, AdapterResponse	from dbt.adapters.contracts.connection import ConnectionState, AdapterResponse
connections.py	from dbt.contracts.connection import Connection	from dbt.adapters.contracts.connection import Connection
connections.py	from dbt.events import AdapterLogger	from dbt.adapters.events.logging import AdapterLogger
connections.py	from dbt.events.functions import fire_event	from dbt_common.events.functions import fire_event
connections.py	from dbt.events.types import ConnectionUsed, SQLQuery, SQLQueryStatus	from dbt.adapters.events.types import ConnectionUsed, SQLQuery, SQLQueryStatus
livysession.py	from dbt.events import AdapterLogger	from dbt.adapters.events.logging import AdapterLogger
fabric_spark_credentials.py	from dbt.adapters.base import Credentials	from dbt.adapters.contracts.connection import Credentials
shortcuts.py	from dbt.adapters.events.logging import AdapterLogger	from dbt.events import AdapterLogger
connections.py	from dbt.dataclass_schema import StrEnum	from dbt_common.dataclass_schema import StrEnum
relation.py	from dbt.events import AdapterLogger	from dbt.adapters.events.logging import AdapterLogger
column.py	from dbt.dataclass_schema import dbtClassMixin	from dbt_common.dataclass_schema import dbtClassMixin
impl.py	from dbt.clients.agate_helper import DEFAULT_TYPE_TESTER	from dbt_common.clients.agate_helper import DEFAULT_TYPE_TESTER
impl.py	from dbt.contracts.relation import RelationType	from dbt.adapters.contracts.relation import RelationType
impl.py	from dbt.events import AdapterLogger	from dbt.adapters.events.logging import AdapterLogger
impl.py	from dbt.utils import executor, AttrDict	from dbt_common.utils import executor, AttrDict
connections.py	import dbt.exceptions	import dbt.adapters.exceptions
connections.py	dbt.exceptions.FailedToConnectError	dbt.adapters.exceptions.FailedToConnectError