+
+ Core¶
+Package: databricks.bundles.core
Classes¶
+-
+
- +class Resources¶ +
Resources is a collection of resources in a bundle.
+Resources class is returned by ‘load_resources’ function specified in databricks.yml. Each element in +‘python/resources’ list is a fully qualified function name that returns an instance of Resources class. +If there are multiple functions specified in ‘python/resources’ list, the resources from all functions +are combined into a single Resources object.
+Example:
+++experimental: + python: + resources: + - "resources:load_resources" +
load_resources function can be implemented using built-in functions:
+-
+
- +
- +
- +
- +
Programmatic construction of resources is supported using
+add_resource()andadd_job()methods.Example:
++ + ++def load_resources(bundle: Bundle) -> Resources: + resources = Resources() + + for resource_name, config in get_configs(): + job = create_job(config) + + resources.add_job(resource_name, job) + + return resources +
-
+
- +property diagnostics: Diagnostics¶ +
Returns diagnostics. If there are any diagnostic errors, bundle validation fails.
+
-
+
- +add_resource( + + + +) None¶ +
Adds a resource to the collection of resources. Resource name must be unique across all +resources of the same type.
+-
+
- Parameters: +
-
+
resource_name – unique identifier for the resource
+resource – the resource to add
+location – optional location of the resource in the source code
+
+
-
+
- +add_job( + + + +) None¶ +
Adds a job to the collection of resources. Resource name must be unique across all jobs.
+-
+
- Parameters: +
-
+
resource_name – unique identifier for the job
+job – the job to add, can be Job or dict
+location – optional location of the job in the source code
+
+
-
+
- +add_location( + + + +) None¶ +
Associate source code location with a path in the bundle configuration.
+
-
+
- +add_diagnostics( + +
- other: Diagnostics, +
Add diagnostics from another Diagnostics object. +:param other: +:return:
+
-
+
-
+
- +add_diagnostic_error( + + + +) None¶ +
Report a diagnostic error. If there are any diagnostic errors, bundle validation fails.
+-
+
- Parameters: +
-
+
msg – short summary of the error
+detail – optional detailed description of the error
+path – optional path in bundle configuration where the error occurred
+location – optional location in the source code where the error occurred
+
+
-
+
- +add_diagnostic_warning( + + + +) None¶ +
Report a diagnostic warning. Warnings are informational and do not cause bundle validation to fail.
+-
+
- Parameters: +
-
+
msg – short summary of the warning
+detail – optional detailed description of the warning
+path – optional path in bundle configuration where the warning occurred
+location – optional location in the source code where the warning occurred
+
+
-
+
- +class Resource¶ +
Base class for all resources.
+
-
+
- +class ResourceMutator¶ +
Mutators defined within a single Python module are applied in the order they are defined. +The relative order of mutators defined in different modules is not guaranteed.
+ + + + + +
-
+
- +class Bundle¶ +
Bundle contains information about a bundle accessible in functions +loading and mutating resources.
+-
+
- +target: str¶ +
Selected target where the bundle is being loaded. E.g.: ‘development’, ‘staging’, or ‘production’.
+
-
+
- +variables: dict[str, Any]¶ +
Values of bundle variables resolved for selected target. Bundle variables are defined in databricks.yml. +For accessing variables as structured data, use
+resolve_variable().Example:
+++variables: + default_dbr_version: + description: Default version of Databricks Runtime + default: "14.3.x-scala2.12" +
-
+
- +resolve_variable( + + + +) _T¶ +
Resolve a variable to its value.
+If the value is a variable, it will be resolved and returned. +Otherwise, the value will be returned as is.
+
-
+
- +class Variable¶ +
Reference to a bundle variable.
+ + + + + + + +
-
+
- +class Diagnostics¶ +
Diagnostics is a collection of errors and warnings we print to users.
+Each item can have source location or path associated, that is reported in output to +indicate where the error or warning occurred.
+-
+
- +items: tuple[Diagnostic, ...]¶ +
-
+
- +extend( + +
- diagnostics: Self, +
Extend items with another diagnostics. This pattern allows +to accumulate errors and warnings.
+Example:
+++def foo() -> Diagnostics: ... +def bar() -> Diagnostics: ... + +diagnostics = Diagnostics() +diagnostics = diagnostics.extend(foo()) +diagnostics = diagnostics.extend(bar()) +
-
+
-
+
- +extend_tuple( + + + +) tuple[_T, Self]¶ +
Extend items with another diagnostics. This variant is useful when +methods return a pair of value and diagnostics. This pattern allows +to accumulate errors and warnings.
+Example:
+++def foo() -> (int, Diagnostics): ... + +diagnostics = Diagnostics() +value, diagnostics = diagnostics.extend_tuple(foo()) +
-
+
- +classmethod create_error( + + + +) Self¶ +
Create an error diagnostics.
+
-
+
- +classmethod create_warning( + + + +) Self¶ +
Create a warning diagnostics.
+
-
+
- +classmethod from_exception( + +
- exc: Exception, +
- *, +
- summary: str, +
- location: Location | None, +
- path: tuple[str, ...] | None, +
- explanation: str | None, +
Create diagnostics from an exception.
+-
+
- Parameters: +
-
+
exc – exception to create diagnostics from
+summary – short summary of the error
+location – optional location in the source code where the error occurred
+path – optional path to relevant property in databricks.yml
+explanation – optional explanation to add to the details
+
+
-
+
-
+
- +class Diagnostic¶ +
- + + + + + + + + + + + +
Methods¶
+-
+
- +core.load_resources_from_current_package_module() Resources¶ +
Load resources from all submodules of the current package module.
+
-
+
- +core.load_resources_from_module( + +
- module: ModuleType, +
Load resources from the given module.
+For recursive loading of resources from submodules, use load_resources_from_package_module.
+-
+
- Parameters: +
module – module to load resources from
+
+
-
+
-
+
- +core.load_resources_from_modules( + +
- modules: Iterable[ModuleType], +
Load resources from the given modules.
+For recursive loading of resources from submodules, use load_resources_from_package_module.
+-
+
- Parameters: +
modules – list of modules to load resources from
+
+
-
+
-
+
- +core.load_resources_from_package_module( + +
- package_module: ModuleType, +
Load resources from all submodules of the given package module.
+-
+
- Parameters: +
package_module – package module to load resources from
+
+
-
+
Decorators¶
+-
+
- +@job_mutator( + +
- function: Callable, +
Decorator for defining a job mutator. Function should return a new instance of the job with the desired changes, +instead of mutating the input job.
+Example:
+++@job_mutator +def my_job_mutator(bundle: Bundle, job: Job) -> Job: + return replace(job, name="my_job") +
-
+
- Parameters: +
function – Function that mutates a job.
+
+
-
+
-
+
- +@variables( + + + +) type[_T]¶ +
A decorator that initializes each annotated attribute in a class +with
+Variabletype. Variables are initialized with a path +that corresponds to the attribute name. Variables should specify their +type, or else they will be treated asAny. Complex types +like data classes, lists or dictionaries are supported.For example, if your databricks.yml file contains:
+++variables: + warehouse_id: + description: Warehouse ID for SQL tasks + default: ... +
You can define a class with a warehouse_id attribute:
+++@variables +class MyVariables: + warehouse_id: Variable[str] # ${var.warehouse_id} +
And later use it in your code as MyVariables.warehouse_id.
+For accessing bundle variable values, see
+Bundle.resolve_variable().