One-stop-shop for docs and test coverage of dbt projects.

Overview

dbt-coverage

PyPI Downloads GitHub last commit PyPI - Python Version PyPI - Format License: MIT FOSSA Status

One-stop-shop for docs and test coverage of dbt projects.

Why do I need something like this?

dbt-coverage is to dbt what coverage.py and interrogate are to Python.

It is a single CLI tool which checks your dbt project for missing documentation and tests.

Keeping documentation and tests close to the actual SQL code that generates the final model is one of the best design choices of dbt. It ensures documentation is actually useful and tests are actually used. But how do you make adding those a habit in your dbt project?

That is exactly where dbt-coverage comes in. It will

  • Give you a better sense of the level of documentation and test coverage in your project;
  • Help your CI/CD pipeline make sure new changes include documentation and tests;
  • Let you quickly assess the documentation and tests of a new dbt project you get your hands on.

Still not convinced? Here are some more features:

  • zero-config: just install it and run it, there is nothing to set up
  • 🏁 minimal dependences: the only dependencies are click (already installed with dbt) and typer
  • 📦 very small: at ~480 SLOC, you can easily validate it works as advertised

Installation

pip install dbt-coverage

Usage

dbt-coverage comes with two basic commands: compute and compare. The documentation for the individual commands can be shown by using the --help option.

Compute

Compute coverage from target/catalog.json and target/manifest.json files found in a dbt project, e.g. jaffle_shop.

To choose between documentation and test coverage, pass doc or test as the CLI argument.

$ cd jaffle_shop
$ dbt run  # Materialize models
$ dbt docs generate  # Generate catalog.json and manifest.json
$ dbt-coverage compute doc --cov-report coverage-doc.json  # Compute doc coverage, print it and write it to coverage-doc.json file

Coverage report
=====================================================================
jaffle_shop.customers                                  6/7      85.7%
jaffle_shop.orders                                     9/9     100.0%
jaffle_shop.raw_customers                              0/3       0.0%
jaffle_shop.raw_orders                                 0/4       0.0%
jaffle_shop.raw_payments                               0/4       0.0%
jaffle_shop.stg_customers                              0/3       0.0%
jaffle_shop.stg_orders                                 0/4       0.0%
jaffle_shop.stg_payments                               0/4       0.0%
=====================================================================
Total                                                 15/38     39.5%

$ dbt-coverage compute test --cov-report coverage-test.json  # Compute test coverage, print it and write it to coverage-test.json file

Coverage report
=====================================================================
jaffle_shop.customers                                  1/7      14.3%
jaffle_shop.orders                                     8/9      88.9%
jaffle_shop.raw_customers                              0/3       0.0%
jaffle_shop.raw_orders                                 0/4       0.0%
jaffle_shop.raw_payments                               0/4       0.0%
jaffle_shop.stg_customers                              1/3      33.3%
jaffle_shop.stg_orders                                 2/4      50.0%
jaffle_shop.stg_payments                               2/4      50.0%
=====================================================================
Total                                                 14/38     36.8%

Compare

Compare two coverage.json files generated by the compute command. This is useful to ensure that the coverage does not drop while making changes to the project.

$ dbt-coverage compare coverage-after.json coverage-before.json

# Coverage delta summary
              before     after            +/-
=============================================
Coverage      39.47%    38.46%         -1.01%
=============================================
Tables             8         8          +0/+0
Columns           38        39          +1/+0
=============================================
Hits              15        15          +0/+0
Misses            23        24          +1/+0
=============================================

# New misses
=========================================================================
Catalog                         15/38   (39.47%)  ->    15/39   (38.46%) 
=========================================================================
- jaffle_shop.customers          6/7    (85.71%)  ->     6/8    (75.00%) 
-- new_col                       -/-       (-)    ->     0/1     (0.00%) 
=========================================================================

Combined use-case

$ cd my-dbt-project

$ dbt run  # Materialize models
$ dbt docs generate  # Generate catalog.json and manifest.json
$ dbt-coverage compute doc --cov-report before.json --cov-fail-under 0.5  # Fail if coverage is lower than 50%

# Make changes to the dbt project, e.g. add some columns to the DWH, document some columns, etc.

$ dbt run  # Materialize the changed models
$ dbt docs generate  # Generate catalog.json and manifest.json
$ dbt-coverage compute doc --cov-report after.json --cov-fail-compare before.json  # Fail if the current coverage is lower than coverage in before.json
$ dbt-coverage compare after.json before.json  # Generate a detailed coverage delta report

Related packages

License

Licensed under the MIT license (see LICENSE.md file for more details).

FOSSA Status

Comments
  • Allow model filtering

    Allow model filtering

    Issue

    • Closes https://github.com/slidoapp/dbt-coverage/issues/24:

    "Please add options in the CLI to include and exclude models to filter out the checks in some of the models or a path."

    Summary

    I added the ability to perform compute commands on only a subset of tables by adding a --model-path-filter option. This means that a subset of models can be selected based upon their original_file_path value (taken from the manifest.json artifact).

    This functionality means that dbt-coverage can now be used in monolithic dbt projects which contain sub-projects owned by different teams. Before adding model selection functionality, using dbt-coverage would not have been useful/advisable in such a structure because another, unrelated team may decrease the overall coverage, which can then block PR merging (should dbt-coverage have been integrated as part of a CI/CD pipeline for example).

    • See example of added functionality from updated README.md below:
    $ cd jaffle_shop
    $ dbt run  # Materialize models
    $ dbt docs generate  # Generate catalog.json and manifest.json
    $ dbt-coverage compute doc --cov-report coverage-doc.json --model-path-filter models/staging/  # Compute doc coverage for a subset of tables, print it and write it to coverage-doc.json file
    
    Coverage report
    ======================================================
    jaffle_shop.stg_customers              0/3       0.0%
    jaffle_shop.stg_orders                 0/4       0.0%
    jaffle_shop.stg_payments               0/4       0.0%
    ======================================================
    Total                                  0/11      0.0%
    
    $ dbt-coverage compute doc --cov-report coverage-doc.json --model-path-filter models/orders.sql --model-path-filter models/staging/  # Compute doc coverage for a subset of tables, print it and write it to coverage-doc.json file
    
    Coverage report
    ======================================================
    jaffle_shop.orders                     0/9       0.0%
    jaffle_shop.stg_customers              0/3       0.0%
    jaffle_shop.stg_orders                 0/4       0.0%
    jaffle_shop.stg_payments               0/4       0.0%
    ======================================================
    Total                                  0/20      0.0%
    

    Note: this is a relatively 'rough' solution and there are likely many improvements that could be made to my code / far more elegant implementations that would achieve the same functionality. Please, feel free to suggest changes!

    Testing

    • I have tested these changes on dbt's jaffle_shop 'testing project' and have not encountered issues so far.
    opened by followingell 5
  • Custom run artifacts path

    Custom run artifacts path

    Goal

    Give more flexibility to user who have their manifest.json & catalog.json files not stored in /target, which is mainly the case for user using dbt cloud (need to retrieve the file by api call), adding a new parameter to use a custom path allow user not to create an unnecessary targetdir in their cicd pipeline

    Description

    • Add run_artifacts_dir : Path parameter to load_manifest and load_catalog
    • Add optional run_artifacts_dir parameter to cli
    opened by Fszta 5
  • Support multiple coverage doc/test save formats

    Support multiple coverage doc/test save formats

    From my understanding we can only save .json files as a result of dbt-coverage compute.

    It would be nice if we could pick other formats, for example .csv That way we can put those files as seeds and write to the db.

    opened by rloredo 5
  • dbt 1.0

    dbt 1.0

    • Support dbt 1.0 and drop support for dbt<1
    • Warn when using with unsupported manifest.json version

    @mrshu, I acknowledged the changes in the new v4 manifest.json format introduced by dbt 1.0 and by that the support for v3 dropped. Since the difference between the formats is only tiny, do you think it would make sense to support also v3 manifest.json (used in dbt<1) by having 2 methods for parsing the manifest, one for v3 and one for v4?

    Closes #26.

    opened by sweco 3
  • Tabulate doc coverage output using markdown

    Tabulate doc coverage output using markdown

    Resolves #46

    What has changed?

    Introduces a new argument called cov_format that accepts string (default) or markdown for the compute doc command.

    Outputs the results to the terminal in either the default string format, or in a markdown friendly format

    Example

    dbt-coverage compute doc --cov-format markdown
    

    image

    results in:

    image

    opened by vvvito 2
  • Make error more informative if target/manifest.json doesn't exist

    Make error more informative if target/manifest.json doesn't exist

    Planning to submit a PR, just wanted to tee this up first to document the need

    Steps to reproduce:

    pip install dbt-coverage==0.2.2
    git clone https://github.com/dbt-labs/jaffle_shop
    git checkout b0b77aa
    dbt-coverage compute doc
    

    Line of code throwing the error: https://github.com/slidoapp/dbt-coverage/blob/9f97e6c8fc24e83babc51c2b0584afc88e45308c/dbt_coverage/init.py#L507

    Error message:

    INFO:root:Loading catalog and manifest files from project dir: .
    Traceback (most recent call last):
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/bin/dbt-coverage", line 8, in <module>
        sys.exit(app())
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
        return get_command(self)(*args, **kwargs)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/typer/main.py", line 532, in wrapper
        return callback(**use_params)  # type: ignore
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 660, in compute
        return do_compute(project_dir, cov_report, cov_type, cov_fail_under, cov_fail_compare)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 616, in do_compute
        catalog = load_files(project_dir)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 533, in load_files
        catalog = load_catalog(project_dir)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 507, in load_catalog
        with open(project_dir / 'target/catalog.json') as f:
    FileNotFoundError: [Errno 2] No such file or directory: 'target/catalog.json'
    
    opened by bbrewington 2
  • Add support for Manifest v6

    Add support for Manifest v6

    opened by reegs 2
  • Take into account seeds

    Take into account seeds

    It seems that seeds are not considered while reading manifest.json. As a result, the following error pops while running dbt-coverage compute doc

    (venv_dbt) ➜  myCodeFolder git:(master) dbt-coverage compute doc                                
    INFO:root:Loading catalog and manifest files from project dir: .
    INFO:root:Successfully loaded 200 tables from catalog
    Traceback (most recent call last):
      File "/Users/victor/myCodeFolder/venv_dbt/bin/dbt-coverage", line 8, in <module>
        sys.exit(app())
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
        return get_command(self)(*args, **kwargs)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
        return self.main(*args, **kwargs)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1062, in main
        rv = self.invoke(ctx)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 763, in invoke
        return __callback(*args, **kwargs)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/typer/main.py", line 500, in wrapper
        return callback(**use_params)  # type: ignore
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 604, in compute
        do_compute(project_dir, cov_report, cov_type, cov_fail_under, cov_fail_compare)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 562, in do_compute
        catalog = load_files(project_dir)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 486, in load_files
        manifest = load_manifest(project_dir)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 477, in load_manifest
        manifest = Manifest.from_nodes(manifest_nodes)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 121, in from_nodes
        table_name = id_to_table_name[table_id]
    KeyError: 'seed.dbt_project.browsers'
    

    Versions:

    dbt-core==0.21.0
    dbt-coverage==0.1.7
    

    Python 3.8

    opened by vdeb 2
  • KeyError: 'column_name' ?

    KeyError: 'column_name' ?

    # dbt build
    ...
    Done. PASS=23 WARN=0 ERROR=1 SKIP=1 TOTAL=25
    # dbt docs generate
    Running with dbt=0.21.1-rc1
    ...
    $ pip install dbt-coverage
    Successfully installed dbt-coverage-0.1.7 typer-0.4.0
    $ dbt-coverage compute doc 
    INFO:root:Loading catalog and manifest files from project dir: .
    INFO:root:Successfully loaded 17 tables from catalog
    
        catalog = load_files(project_dir)
      File "/root/.local/share/virtualenvs/data-pipelines-kwiw1yve/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 486, in load_files
        manifest = load_manifest(project_dir)
      File "/root/.local/share/virtualenvs/data-pipelines-kwiw1yve/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 477, in load_manifest
        manifest = Manifest.from_nodes(manifest_nodes)
      File "/root/.local/share/virtualenvs/data-pipelines-kwiw1yve/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 122, in from_nodes
        column_name = node['column_name'] or node['test_metadata']['kwargs']['column_name'] \
    KeyError: 'column_name'
    
    opened by chapmanjacobd 2
  • Fix matching of entities between manifest and catalog

    Fix matching of entities between manifest and catalog

    This fixes two issues:

    1. Some systems (e.g. Snowflake) uppercase all table and column names. In accordance with the dbt-docs official source code, we lowercase all table and column names before matching them.
    2. There can be multiple tables with the same name but in different schemas. We therefore use schema_name.table_name to distinguish tables.
    opened by sweco 1
  • Add contents

    Add contents

    What was done

    • Copy contents of dbt_docs_coverage.py verbatim to __init__.py.
    • Add README with basic documentation.
    • Create Python package using poetry.
    • Register dbt_coverage.__init__:app as a CLI script with name dbt-coverage.

    Review objectives

    • Please review README.
    • Please review the poetry files.
    • Ignore __init__.py, it was copied verbatim.
    opened by sweco 1
  • Display coverage doc/test result on gitlab badges

    Display coverage doc/test result on gitlab badges

    Hi team,

    I am wondering whether it is already possible for me to display the coverage doc/test result as a percentage on gitlab badges. Please see similar example for python unit test below image

    Thank you

    opened by kelvinheng92 3
  • Clean up `len(model_path_filter) >= 1` once a Typer issue gets resolved

    Clean up `len(model_path_filter) >= 1` once a Typer issue gets resolved

    Originally I had used is not None as you say, however, this doesn't work due to a known issue with typer: https://github.com/tiangolo/typer/issues/410 whereby [an] "Empty tuple [is] returned for Option instead of expected None when no values passed". As such, I have to use len() since until the issue is resolved model_path_filter will never be None.

    ~~Since you asked the question it's obvious that I could make the reason for my choice clearer, WDYT to the below suggestion?~~

    Originally posted by @followingell in https://github.com/slidoapp/dbt-coverage/pull/45#discussion_r1012694530

    good first issue 
    opened by mrshu 2
  • Support name / identifier format within sources

    Support name / identifier format within sources

    The script does not currently recognize tests applied to a source that is set-up using an identifier rather than just the raw as being associated with the source.

    I think, but am not certain, that this might be related to #30

    opened by reegs 1
  • could you add ability produce a coverage report in cobertura format

    could you add ability produce a coverage report in cobertura format

    To be able to retrieve dbt-coverage output in gitlab, a cobertura format is needed.

    See: https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportscoverage_report

    Could you add a command line argument to choose output format ? (json or xml-cobertura)

    opened by cedric-orange 1
  • support generic tests from dbt_utils when calculating coverage

    support generic tests from dbt_utils when calculating coverage

    Our project is relying on tests defined in dbt_utils - is there a way of adding support for these as well?

    https://github.com/dbt-labs/dbt-utils#unique_combination_of_columns-source

    opened by aa-matthias 5
Releases(v0.3.2)
Owner
Slido
Slido is an audience interaction platform for meetings and events. Public repositories
Slido
A browser login credentials thief for windows and Linux

Thief 🦹🏻 A browser login credentials thief for windows and Linux Python script to decrypt login credentials from browsers in windows or linux Decryp

Ash 1 Dec 13, 2021
Intelligent Employer Profiling Platform.

Intelligent Employer Profiling Platform Setup Instructions Generating Model Data Ensure that Python 3.9+ and pip is installed. Install project depende

Harvey Donnelly 2 Jan 09, 2022
Генератор отчетов на Python с использованием библиотеки docx для работы с word-файлами и запросов к сервису

Генератор отчетов на Python с использованием библиотеки docx для работы с word-файлами и запросов к сервису

Semyon Esaev 2 Jun 24, 2022
Scripts for BGC analysis in large MAGs and results of their application to soil metagenomes within Chernevaya Taiga RSF-funded project

Scripts for BGC analysis in large MAGs and results of their application to soil metagenomes within Chernevaya Taiga RSF-funded project

1 Dec 06, 2021
A log likelihood fit for extracting neutrino oscillation parameters

A-log-likelihood-fit-for-extracting-neutrino-oscillation-parameters Minimised the negative log-likelihood fit to extract neutrino oscillation paramete

Vid Homsak 1 Jan 23, 2022
Creates infinite amount of guilded accounts in seconds.

Guilded Cookie Creator [fuck guilded i quit working on this, they patch like every fucking method after 2/3 days i release shit] Optimizations Asynchr

scripted 7 Feb 28, 2022
Magenta: Music and Art Generation with Machine Intelligence

Magenta is a research project exploring the role of machine learning in the process of creating art and music. Primarily this involves developing new

Magenta 18.1k Jan 05, 2023
A module that can manage you're gtps

Growtopia Private Server Controler Module For Controle Your GTPS | Build in Python3 Creator Information

iFanpS 6 Jan 14, 2022
A simple program to recolour simple png icon-like pictures with just one colour + transparent or white background. Resulting images all have transparent background and a new colour.

A simple program to recolour simple png icon-like pictures with just one colour + transparent or white background. Resulting images all have transparent background and a new colour.

Anna Tůmová 0 Jan 30, 2022
Expense-manager - Expense manager with python

Expense_manager TO-DO Source extractor: Credit Card, Wallet Destination extracto

1 Feb 13, 2022
Class and mathematical functions for quaternion numbers.

Quaternions Class and mathematical functions for quaternion numbers. Installation Python This is a Python 3 module. If you don't have Python installed

3 Nov 08, 2022
Object-oriented programming (OOP) is a method of structuring a program by bundling related properties and behaviors into individual objects. In this tutorial, you’ll learn the basics of object-oriented programming in Python.

06_Python_Object_Class Introduction 👋 Objected oriented programming as a discipline has gained a universal following among developers. Python, an in-

Milaan Parmar / Милан пармар / _米兰 帕尔马 239 Dec 20, 2022
This app converts an pdf file into the audio file.

PDF-to-Audio This app takes an pdf as an input and convert it into audio, and the library text-to-speech starts speaking the preffered page given in t

Ojas Barawal 3 Aug 04, 2021
A totally unrealistic cell growth/reproduction simulation.

A totally unrealistic cell growth/reproduction simulation.

Andrien Wiandyano 1 Oct 24, 2021
Blender-miHoYo-Shaders - Shaders for Blender attempting to replicate the shading of games developed by miHoYo

Blender-miHoYo-Shaders - Shaders for Blender attempting to replicate the shading of games developed by miHoYo

Matsuri 449 Jan 07, 2023
Скрипт позволяет выгрузить участников чатов/каналов(по чату для комментариев) и сообщения в различные форматы файлов.

TG-Parser Парсер участников и сообщений из ТГ-Чатов и чатов для комментариев в ТГ-Каналах Возможности Выгрузка участников групп/каналов(по чату для ко

50 Jan 06, 2023
A Bot that adds YouTube views to your video of choice

YoutubeViews Free Youtube viewer bot A Bot that adds YouTube views to your video of choice Installation git clone https://github.com/davdtheemonk/Yout

ProbablyX 5 Dec 06, 2022
Originally used during Marketplace.tf's open period, this program was used to get the profit of items bought with keys and sold for dollars.

Originally used during Marketplace.tf's open period, this program was used to get the profit of items bought with keys and sold for dollars. Practically useless for me now, but can be used as an exam

BoggoTV 1 Dec 11, 2021
Basic Clojure REPL for Sublime Text

Basic Clojure REPL for Sublime Text Goals: Decomplected: just REPL, nothing more Zero dependencies: works directly with pREPL Compact: Display code ev

Nikita Prokopov 23 Dec 24, 2021
A tool to quickly create codeforces contest directories with templates.

Codeforces Template Tool I created this tool to help me quickly set up codeforces contests/singular problems with templates. Tested for windows, shoul

1 Jun 02, 2022