OpenLineage dbt integration
Wrapper script for automatic metadata collection from dbt
Features
Metadata
- Model run lifecycle
- Model inputs / outputs
Requirements
Right now, openlineage-dbt only supports bigquery, snowflake, spark and redshift dbt adapters.
Installation
$ pip3 install openlineage-dbt
To install from source, run:
$ pip install .
Configuration
HTTP Backend Environment Variables
openlineage-dbt uses the OpenLineage client to push data to the OpenLineage backend.
The OpenLineage client depends on environment variables:
OPENLINEAGE_URL- point to service which will consume OpenLineage eventsOPENLINEAGE_API_KEY- set if consumer of OpenLineage events requiresBearerauthentication keyOPENLINEAGE_NAMESPACE- set if you are using something other than thedefaultnamespace for job namespace.
Logging
In addition to conventional logging approaches, the OpenLineage dbt wrapper script provides an alternative way of configuring its logging behavior. By setting the OPENLINEAGE_DBT_LOGGING environment variable, you can establish the logging level for the openlineage.dbt and its child modules.
You can also set log level of dbtol which is deprecated.
Usage
To begin collecting dbt metadata with OpenLineage, replace dbt run with dbt-ol run.
Additional table and column level metadata will be available if catalog.json, a result of running dbt docs generate, will be found in the target directory.
If you need to send events without running the job you can use the command dbt-ol send-events, it will send the metadata of your last run without running the job.
SPDX-License-Identifier: Apache-2.0
Copyright 2018-2024 contributors to the OpenLineage project
1 maintainer
Metadata
1 maintainer
Release files for openlineage-dbt 1.24.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| openlineage_dbt-1.24.0.tar.gz | 4.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| openlineage_dbt-1.24.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 9.6 kB
Release files / openlineage_dbt-1.24.0.tar.gz
| Download URL | openlineage_dbt-1.24.0.tar.gz |
|---|---|
| Size | 4.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f12446659c4c2412c111ca7d7852cdb313f7a22823008b31708ad58e6d2c656d
|
|
BLAKE2b-256 checksum How to use checksums |
b61dd4fe4890b3e1cc6d97b7225b31acbf34a16da94154099771d862e8eb89fd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.8.20
|
Release files / openlineage_dbt-1.24.0-py3-none-any.whl
| Download URL | openlineage_dbt-1.24.0-py3-none-any.whl |
|---|---|
| Size | 4.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8ce96c411f31385f08963627d983ee7e22c64d9ef3d36582b9a18ce2a3e493e6
|
|
BLAKE2b-256 checksum How to use checksums |
54aee37b5f1b77e6fec3b2b0dab81b4ba4851bc4913f0b2534f2da96cba1842f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.8.20
|
