Meltano: ELT for the DataOps era. Meltano is open source, self-hosted, CLI-first, debuggable, and extensible.

Last update: Jan 02, 2023

Related tags

Overview

Meltano: ELT for the DataOps era

Meltano is open source, self-hosted, CLI-first, debuggable, and extensible.

Pipelines are code, ready to be version controlled, containerized, and deployed continuously. Develop and test locally, then deploy in production along with the built-in Airflow integration, or inside your orchestrator of choice.

Meltano embraces the Singer standard and its community-maintained library of open source extractors and loaders, and leverages dbt for transformation.

Documentation

Check out the "Getting Started" guide or find the full documentation at https://www.meltano.com/docs/.

Contributing to Meltano

We welcome contributions and improvements, please see the contribution guidelines

Responsible Disclosure Policy

Please refer to the responsible disclosure policy on our website.

License

This code is distributed under the MIT license, see the LICENSE file.

Comments

Python-based Plugin Architecture
I think we have a path forward on python plugin architecture (https://gitlab.com/groups/meltano/-/epics/134#note_981098666) which does not require modifying Meltano itself (at least not in the first iterations).

Since pip_url for any repo can include multiple libraries, we can extend any plugins' capabilities by adding a meltano-managed python tool to any plugin that needs capabilities. Then those capabilities just need to registered as CLI-executable and be mapped to commands and capabilities in the Meltano plugin definition.

Example: registering dbt packages in packages.yml

Today we have custom code in Meltano that adds entries to packages.yml for dbt during meltano add transform .... We could replace it as follows:

Create a new python tool called meltano-dbt-helpers with a registered CLI command meltano-dbt-util add package.

Add that package to the pipurl for all dbt plugins. E.g. pip_url: dbt-snowflake meltano-dbt-helpers.

Register commands in the package that leverage the helper. E.g.
commands: add-dep: command: meltano-dbt-util args: add package

Then from the CLI as a user:

meltano run dbt:add-dep dbt-util

Would add dbt-util to the package.

For "smarter" integrations, we just need a contract that pairs capabilities like service with expected commands, like ui, up, and down.

Comparison with other approaches

The best part of this approach is that it doesn't require any code changes in Meltano - we and community members could implement this today with existing features.
roadmap kind/Feature valuestream/Meltano
opened by aaronsteers 33
feat: Add systemdb support for MS SQL Server

Relates to issue #3238.

The main improvement is limiting all String types in the alembic migrations. This allows the possibility to use both MSSQL and MYSQL as a Meltano backend database.

Both databases were manually tested. It might be valuable to add automatic tests to the Github actions.

To use MYSQL run: pip install mysqlclient To use MSSQL run: pip install pyodbc
Community-Contributed PR

opened by JulesHuisman 32
fix: symbolic link creation for meltano.exe when on windows

This fixes the bug #6467 where on Windows meltano upgrade stops because MeltanoInvoker.invoke() is calling subprocess.run() which requires a executable target. Since the symbolic link .meltano\run\bin is targeting meltano not meltano.exe a failure occurs. I updated the Project.activate() function to target meltano.exe when Windows is detected.

This does not fix any previously created symbolic links that might exist in projects.

Close #6467
Community-Contributed PR Windows kind/Bug valuestream/Meltano

opened by BuzzCutNorman 19
Create `cron` utility extension

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3522

Originally created by @alxthm on 2022-05-24 14:54:47

For simple meltano deployments, it would be great to have a cron-based orchestrator using the meltano schedule configuration, without having to set-up an Airflow instance with a db, users, etc.

The meltano UI already makes it possible to manually run meltano schedules, so this would be a nice complement :smile:
kind/Feature valuestream/Meltano migrated from gitlab

opened by MeltyBot 19
Disallow calling a `mapper` by name in `meltano run` and alert to use a `mapping` name
The document misses a step for mapper configuration. Property mapping_name has to be added to the mappers extras, and its value need to match with the name under mappings. It doesn't seem been documented anywhere.

plugins: mappers: - name: transform-field variant: transferwise pip_url: pipelinewise-transform-field executable: transform-field mapping_name: transform-field-hash mappings: - name: transform-field-hash config: transformations: - field_id: Name tap_stream_name: dbo-Test type: MASK-HIDDEN
Documentation Accepting Pull Requests User Support kind/Bug valuestream/Meltano plugin/Singer
opened by lidong-zhao 18
bug: Meltano Upgrade on Windows stops with error "is not a valid Win32 application"

Meltano Version

2.3.0

Python Version

3.9

Bug scope

CLI (options, error messages, logging, etc.)

Operating System

Windows Server 2022 Standard

Description

I have noticed on Windows if the file(symbolic link) .meltano/run/bin is present in your project when you run the upgrade via meltano upgrade or the melatno ui "Upgrade Button" you will get the following WinError.

[WinError 193] %1 is not a valid Win32 application

if you remove the file and run meltano upgrade the upgrade will complete. I am not sure what the long term ramification of deleting the file is.

Code

No response
User Support Windows kind/Bug valuestream/Meltano

opened by BuzzCutNorman 17
New process and guidelines for integration tests (replaces `demo-project` and `homepage_sample_code` tests in GitLab CI

From a call with @tayloramurphy and @DouweM, we started thinking of replacing demo-project with a set of sample Meltano projects in meltano/meltano that can serve as integration tests, perhaps along with specific scripts that would run - such as the invoking of specific plugins.

I'm sure there are better and more sophisticated approaches, but one approach would be a convention have a meltano.yml, a script.sh, and a meltano.after.yml which should match to the results. Another approach would be a set of pytest tests that perform a series of operations against (a copy of?) the source meltano.yml file and confirm the results (or errors), possibly including specific telemetry events that are expected to be sent over the course of those script commands.
kind/Tech Debt valuestream/Meltano

opened by aaronsteers 17
Restore `project_id` from `analytics.json` & improve `analytics.json` handling
Restore project_id from analytics.json if it is not found in meltano.yml.

Overwrite analytics.json with a new one if it is malformed.

Handle more errors when dealing with analytics.json (e.g. JSONDecodeError).

Add additional debug logging.

Do not update meltano.yml from ProjectContext (n.b. this would cause the project_id key to end up inside of the config of the active environment instead of the top-level config).
opened by WillDaSilva 17
ci: add `pytest` support on windows; change: add `meltano elt` guidance for windows users

Closes https://github.com/meltano/meltano/issues/5940

Old MR from Gitlab https://gitlab.com/meltano/meltano/-/merge_requests/2639/diffs

(Note this has been rebased since this MR)

Pipeline is running here https://gitlab.com/vischous/meltano/-/merge_requests/3/pipelines

opened by visch 17
bug: Projects without `environments` nor `default_environment` see error where `dev` environment is not found
Meltano Version

2.5.0

Python Version

NA

Bug scope

Configuration (settings parsing, validation, etc.)

Operating System

NA

Description

Users with projects created from scratch (not using meltano init) or started before Environments where introduced may not be using default_environment nor environments in their meltano.yml.

This causes an error because in 2.5.0 we made default_environment a project setting that defaults to dev. Any project without a dev environment, as those exemplified above, will see an error because Meltano will be looking for dev.

Code

$ meltano install Environment 'dev' was not found

This is reproducible with the following meltano.yml:

plugins: loaders: - name: target-jsonl variant: andyh1203 pip_url: target-jsonl

Workaround

From AJ:

There should be two ways to work around this issue.

Running meltano environment add dev as a one-time fix will resolve the issue, even if no other changes are made and no environment-specific settings are added.

Adding the --no-environment CLI arg will override the default dev environment name.
Accepting Pull Requests kind/Bug urgency/High valuestream/Meltano
opened by edgarrmondragon 16
Give `http://melta.no` troubleshooting help link when plugin fails, with CTA to Slack in web copy

Spec (edited by AJ):

When Meltano fails, we'll print a standard error message:

[Error description text.]

To contact the Meltano team and access troubleshooting help and resources, visit http://melta.no

Orig. issue description (Douwe)

When meltano invoke or meltano run fails, a user is more likely to manage to solve the issue and stick with Meltano if they join Slack than if they don't, as evidenced by the number of people who responded to the survey saying the docs weren't always helpful, but the Slack community was. I imagine a good amount of people that try us out and churn would not have if they had known about Slack. That also lets us reach out to them to learn more about their use case and experience, even if they do end up churning.

So I think we should recommend joining Slack in the product wherever that makes sense.

@tayloramurphy Wdyt? Seems like an easy change with potentially large impact.

cli needs refinement OKR: New User Experience kind/Feature valuestream/Meltano

opened by DouweM 16
bug: Unable to get `--help` for `meltano config ... set` without a project dir context
Meltano Version

2.12.0

Python Version

3.10

Bug scope

API

Operating System

MacOS

Description

Two problems.

Problem 1: It doesn't seem to be possible to get --help on the config set operation.

Problem 2: The error message ('is_default_environment') doesn't help at all.

It may not be possible to solve problem 1 without refactoring that command, but problem 2 should be addressable with improved exception/error handling.

Code

[email protected] ~ % pipx run meltano config set --help ⚠️ meltano is already on your PATH and installed at /Users/aj/.local/bin/meltano. Downloading and running anyway. Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to join our friendly Slack community. 'is_default_environment' [email protected] ~ % pipx run meltano config asdf set --help ⚠️ meltano is already on your PATH and installed at /Users/aj/.local/bin/meltano. Downloading and running anyway. Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to join our friendly Slack community. 'is_default_environment'

Running without project context

The above was executed without a project context, meaning the working directory was not a valid Meltano project. This could be part of the issue, but regardless, I think it could be expected that --help will work even without an active project folder.

Related

https://github.com/meltano/meltano/discussions/7137

cli OKR: New User Experience ux kind/Bug valuestream/Meltano
opened by aaronsteers 0
bug: `meltano config test` returns error with selection criteria
Meltano Version

2.9.0

Python Version

3.9

Bug scope

CLI (options, error messages, logging, etc.)

Operating System

Codespace

Description

I was testing a tap-github connector in GitHub Codespace. When I had a selection criteria to filter to just issues I would get this error:

[email protected]:/workspaces/meltano-codespace-ready# meltano config tap-github test 2022-12-23T21:46:26.678163Z [info ] The default environment 'codespace' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`. Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to join our friendly Slack community. 'ascii' codec can't decode byte 0xe2 in position 3232: ordinal not in range(128)

Removing the selection criteria resolved it.

The Fix

testing configuration should still work even with selection criteria present!

Code

No response
kind/Bug valuestream/Meltano
opened by tayloramurphy 0
bug: nested configs are cleared when using space as delimiter and when using `--interactive`
Update (@aaronsteers)

Per the issue thread below, we've identified that the issue is present in the behavior of the CLI syntax with space as delimiter but not when dot is used. The proposal is to make these two function identically:

meltano config testy-mctest set file_configs.format json # the expected behavior (does not reset sibling config) meltano config testy-mctest set file_configs format json # bug: clears sibling nested config

Original Issue description (@pnadolny13)

When using --interactive to update nested settings it overwrites the entry config set. For example:

- name: tap-file config: file_configs: dataset_name: test_file format: csv path: /my/path/

Then I run meltano config tap-file set --interactive, navigating to the setting file_configs.format (in dot notation) as defined in the plugin definition lock file, updating to json, meltano overwrites the whole nested config with only the new value:

- name: tap-file config: file_configs: format: json

Slack thread with video showing bug https://meltano.slack.com/archives/C03GKHWS0HM/p1671746236351709

meltano, version 2.12.0
kind/Bug valuestream/Meltano
opened by pnadolny13 4
bug: Error: Setting value set via multiple environment Azure connection string (state backend).
Meltano Version

2.12.0

Python Version

3.10

Bug scope

Configuration (settings parsing, validation, etc.)

Operating System

Linux - Ubuntu 20.04

Description

I am running into the same issue as #7059, but with the Azure connection string.

I solved the issue when running the Meltano cli. However, the issue came back when I tried to run the Dagster utility.

Code

Error: Setting value set via multiple environment variables: ['MELTANO_STATE_BACKEND_AZURE_CONNECTION_STRING', 'AZURE_STORAGE_CONNECTION_STRING']
kind/Bug valuestream/Meltano
opened by JulesHuisman 2

Releases(v2.12.0)

v2.12.0(Dec 19, 2022)
v2.12.0 (2022-12-19)

🐛 Fixes

#7084 Display better errors when failing to connect with the Hub API

#7078 Meltano no longer fails when AWS_ACCESS_KEY_ID exists in the host environment

⚡ Performance Improvements

#7091 Compile env var pattern only once

📚 Documentation Improvements

#7089 Fix typo tap-postgres -> postgres -- Thanks @visch!

#7090 Fix config name for poetry -- Thanks @adherr!

Source code(tar.gz)
Source code(zip)
meltano-2.12.0-py3-none-any.whl(5.31 MB)
v2.11.1(Dec 9, 2022)
v2.11.1 (2022-12-09)

🐛 Fixes

#7057 Ensure environment_name_hash is collected

#7064 Permit env under extractors within environments and variant under mappers in meltano.yml schema

Source code(tar.gz)
Source code(zip)
meltano-2.11.1-py3-none-any.whl(5.30 MB)
v2.11.0(Dec 8, 2022)
v2.11.0 (2022-12-08)

✨ New

#7019 Support env vars in plugin pip_url

#6995 meltano install --force

🐛 Fixes

#6873 Correct migration downgrade for 2.7.1 release

#7026 Permit env under plugins in meltano.yml schema

#7022 Pin setuptools==57.5.0 in plugin venvs

#7017 Remove premature deserialization of settings in the CLI

#7010 Re-create plugin venv if Python executable is missing

#6997 Correct meltano config behavior when explicit environment is provided

#6986 List strings instead of enum objects as CLI options

📚 Documentation Improvements

#7046 Fix internal links and meltano config example code block in state backends page

#7041 Expand UI deprecation description -- Thanks @qbatten!

#7044 Add star donation blurb to readme

#7043 Add december campaign banner

#7030 Env var expansion within pip_url

#7035 Update custom extractor Poetry instructions -- Thanks @EMCP!

#7014 Add more project links to PyPI page

#7012 Complete list of commands that run with environment flag

#7002 Add docs for state backend migration

Source code(tar.gz)
Source code(zip)
meltano-2.11.0-py3-none-any.whl(5.30 MB)
v2.10.0(Nov 16, 2022)
v2.10.0 (2022-11-16)

✨ New

#6911 Add support for state backends in S3, Google Cloud Storage, Azure Blob Storage, and local filesystem.

🐛 Fixes

#6971 Correct strict_env_var_mode flag check in plugin configs.

#6935 Persist description and logo_url to plugin lock files -- Thanks @DanielPDWalker!

Source code(tar.gz)
Source code(zip)
meltano-2.10.0-py3-none-any.whl(5.29 MB)
v2.9.0(Nov 8, 2022)
v2.9.0 (2022-11-08)

✨ New

#6964 Add --cwd CLI option to override the current working directory.

#6785 Add support for NO_COLOR env var to disable colored output.

#6628 Make .meltano folder path configurable via new MELTANO_SYS_DIR_ROOT environment variable. -- Thanks @rawwar!

🐛 Fixes

#6938 Fix is_redacted error when a referenced setting is undefined.

#6468 Fix symbolic link creation when running meltano upgrade on Windows -- Thanks @BuzzCutNorman!

📚 Documentation Improvements

#6961 Fix typo "differe" -> "different" -- Thanks @kayakr!

#6951 Reindent code block -- Thanks @jx2lee!

Source code(tar.gz)
Source code(zip)
meltano-2.9.0-py3-none-any.whl(5.28 MB)
v2.8.0(Oct 18, 2022)
v2.8.0 (2022-10-17)

✨ New

#6814 Support meltano init into an empty directory -- Thanks @ReubenFrankel!

🐛 Fixes

#6751 Resolve issue causing Meltano UI to crash when connections to PyPI are blocked. -- Thanks @dingobar!

#6862 The default_environment setting now only applies to commands which execute within the environment: run, invoke, and elt.

#6877 Reverts the default value for the default_environment setting to null instead of 'dev', resolving the related breaking change from v2.5.0.

#6891 Resolve unhandled exception when running interactive config on plugins with no settings.

⚙️ Changes

#6862 The meltano run command will now fail if the Meltano environment is not set and no default environment is configured.

📚 Documentation Improvements

#6703 Update custom extractor tutorial -- Thanks @jared-rimmer!

#6875 Add new Common Issues section to Troubleshooting docs.

#6844 Retroactively update 2.5.0 release notes and changelog to document the now-resolved breaking change.

Source code(tar.gz)
Source code(zip)
meltano-2.8.0-py3-none-any.whl(3.61 MB)
v2.7.2(Oct 3, 2022)
v2.7.2 (2022-10-03)

🐛 Fixes

https://github.com/meltano/meltano/issues/6819 Fix error 'ProjectSettingsService' object has no attribute 'plugin' when running meltano config meltano.

https://github.com/meltano/meltano/issues/6818 Fix issue causing SQLAlchemy deprecation warnings.

📚 Documentation Improvements

https://github.com/meltano/meltano/issues/6800 Simplified installation quick start, with advanced info moved to the detailed install guide.

https://github.com/meltano/meltano/issues/6781 Add parts 2-4 to the tutorial, with quick access from https://docs.meltano.com/.

Source code(tar.gz)
Source code(zip)
meltano-2.7.2-py3-none-any.whl(3.61 MB)
v2.7.1(Sep 27, 2022)
v2.7.1 (2022-09-27)

🐛 Fixes

#6797 Show appropriate UI toasts.

#6795 Correct migration path for state table.

Source code(tar.gz)
Source code(zip)
meltano-2.7.1-py3-none-any.whl(3.61 MB)
v2.7.0(Sep 23, 2022)
v2.7.0 (2022-09-23)

✨ New

#6758 Add support for user-defined state IDs using state_id_suffix in meltano.yml environment definitions and the--state-id-suffix CLI option in meltano run -- Thanks @ReubenFrankel!

🚀 Changes

#6756 Place project_id before environments in meltano.yml in new projects.

🐛 Fixes

#6783 Remove no-op "disable telemetry" toast from the UI.

#6779 Use v1.0.1 of the exit_event Snowplow telemetry schema.

⚙️ Under the Hood

#6742 Refactor to add support for state backends. Meltano will now store state to a new internal state table within systemdb, with support for other state backends coming soon.

#6745 Use random UUID as default project_id when project ID is missing.

#6748 Define MeltanoError exception class and recommend to contributors for improved CLI error messaging.

📚 Documentation Improvements

#6764 Added Part 1 of an ELT tutorial (Link)

#6769 Added a guide for migrating existing dbt projects to Meltano (Link).

#6737 Added a Meltano at a Glance page to the Getting Started section (Link).

#6753 Added a handy list of video tutorials and demos (Link).

#6764 Move installation guide to Getting Started section (Link).

#6709 Added a tutorial for using Meltano with DataHub (Link).

#6739 Add a tutorial for using Meltano with Jupyter (Link).

#6743 Describe how project_id is hashed (Link) .

Source code(tar.gz)
Source code(zip)
meltano-2.7.0-py3-none-any.whl(3.54 MB)
v2.6.0(Sep 12, 2022)
v2.6.0 (2022-09-08)

✨ New

#5892 Add new interactive configuration of plugins with--interactive flag in meltano config set.

#6640 Add rich exception formatting, configurable from logging.yaml.

🐛 Fixes

#6694 Resolves a bug in meltano remove where error states that plugin is missing the settings_with_extras attribute.

#6652 Better handling of permission errors when creating a new project -- Thanks @jakegut!

#6676 Prevent warnings/errors when running meltano config <plugin> test -- Thanks @ReubenFrankel!

📚 Documentation Improvements

#6706 Add troubleshooting page -- Thanks @jared-rimmer!

#6688 Add docs about environment variable expansion.

#6671 Add example meltano projects.

#6674 Add docs for schedule env key.

#6680 Fix feature flag name for strict env var mode.

#6672 Add tutorial on how to debug a custom extractor -- Thanks @jared-rimmer!

Source code(tar.gz)
Source code(zip)
meltano-2.6.0-py3-none-any.whl(3.61 MB)
v2.5.0(Aug 24, 2022)
v2.5.0 (2022-08-24)

✨ New

#6578 Add --no-install option to meltano add -- Thanks, @ReubenFrankel!

#6596 Support glob paths in file plugin update extra -- Thanks, @ReubenFrankel!

#6581 Add new settings for hub_api_root and hub_url_auth -- Thanks, @DanielPDWalker!

⚡ Performance Improvements

#6644 Improve performance when handling large yaml files.

🚀 Changes

#6592 Add validation for CRON expressions in schedules.

#6582 Make default_environment a top-level setting, with default value 'dev'.

NOTE: This was retroactively noted as a breaking change, since it added a requirement that a dev environment name must exist if default_environment was not overridden by the project. This requirement is now removed in the v2.7.3 Release, reverting the breaking change behavior.

🐛 Fixes

#6627 Raise exception when meltano.yml is empty -- Thanks, @rawwar!

#6543 Register legacy DiscoveryFile object in Ruamel YAML instance -- Thanks, @z3z1ma!

#6656 Fixed issue where STRICT_ENV_VAR_MODE feature flag would not raise an error as expected.

#6643 Resolve issue related to alter_column warning for MSSQL.

#6636 Fix help links pointing to docs.meltano.com instead of meltano.com/docs.

#6601 Raise an error if DB migrations fail during upgrade.

#6609 Fix broken --pattern CLI arg in meltano state list.

#6584 Avoid pyproject.toml Poetry parsing issue.

⚙️ Under the Hood

#6621 Make ProjectInitService.init generate/set a project_id.

#6611 Use closing for DB sessions & add docstrings to meltano.core.db.

#6610 Replace StaleJobFailer class with fail_stale_jobs function.

📚 Documentation Improvements

#6648 Expanded logging configuration guide.

#6631 Add "Pytest Best Practices" section to contributing docs.

Source code(tar.gz)
Source code(zip)
meltano-2.5.0-py3-none-any.whl(3.61 MB)
v2.4.0(Aug 5, 2022)
v2.4.0 (2022-08-05)

✨ New

#5983 Add support for environment variables inheritance across configuration layers.

#6167 Add support for MS SQL Server as a systemdb backend -- Thanks @JulesHuisman!

🚀 Changes

#6116 Always use a predictable store location when the default --store=auto setting is applied.

🐛 Fixes

#6532 Resolve issue where Snowplow requests would hang if the Snowplow endpoint was not available.

#6360 meltano lock now updates and locks plugins using the Hub definition if no prior lock file exists.

#6383 Resolve missing systemdb error by initializing backend during meltano install -- Thanks @visch!

#6189 Meltano now validates mapper and mappings names with alerting as necessary -- Thanks @BuzzCutNorman!

#6240 Resolve a locking issue when the enable_uvicorn feature flag is enabled -- Thanks @BuzzCutNorman!

⚙️ Under the hood

#6469 Pin werkzeug library to <2.2.0.

📚 Documentation Improvements

#6107 New reference page for Meltano plugin definition YAML syntax, including plugin containerization options and optional Hub metadata fields.

#6456 Add guidance for using custom python interpreters during pipx install -- Thanks @seajhawk!

#6456 Add Powershell syntax example for setting Meltano environment variables. -- Thanks @seajhawk!

Source code(tar.gz)
Source code(zip)
meltano-2.4.0-py3-none-any.whl(3.61 MB)
v2.3.0(Jul 22, 2022)
✨ New

#6207 Add top-level and plugin-level support for env environment variable declarations in meltano.yml.

#2133 Preserve comments and flow style in meltano.yml when running Meltano CLI commands.

🚀 Changes

#6150 Meltano now surfaces full stderr output for catalog discovery failures.

#3419 Change default meltano config behavior to --no-environment.

🐛 Fixes

#6301 Resolve failures with message object has no attribute 'add_related' when adding new plugins via Meltano UI. -- Thanks, @BuzzCutNorman!

#6188 Resolve meltano run failures when plugins are invoked without providing a subcommand.

#6359 Resolve meltano lock failures by not attempting to lock plugins declared using inherit_from syntax.

#6234 Emit RuntimeWarning if a Meltano setting is provided which does not match expected setting name.

#6380, #3322 Complex extras such as metadata are now honored in inherited plugins and environment inheritance.

📚 Documentation Improvements

#6283 Document how to override capabilities and settings within plugin definitions.

#6136 Clarify job and task behaviors when executed via airflow.

Full Changelog: https://github.com/meltano/meltano/compare/v2.2.0...v2.3.0
Source code(tar.gz)
Source code(zip)
meltano-2.3.0-py3-none-any.whl(3.53 MB)
v2.2.0(Jul 1, 2022)
✨ New

#2830 Add support for Python 3.10.

#3450 Add limited Windows OS support and new installation instructions for Windows. - Thanks, Derek Visch!

#3452 Add copy and move commands to meltano state.

#3135 Improve settings aliases behavior and disallow conflicting values between setting name and aliases.

🐛 Fixes

#5985 Address warning in Airflow plugin version check by using recommended packaging.version.Version.

📝 Documentation Improvements

#3174 We now recommend installing with pipx over manually created virtual environments.

⚙️ Under the Hood

#3367 Windows now covered in CI tests for basic use cases. - Thanks, Derek Visch!

#2035 Rename job table and job_id column in backend systemdb.

Source code(tar.gz)
Source code(zip)
meltano-2.2.0-py3-none-any.whl(3.53 MB)
v2.2.0rc0(Jul 1, 2022)
New

#3452 Add copy and move commands to meltano state

Changes

#3174 We now recommend pipx over manually created virtual environments.

#3450 Windows instllation instructions added

#3367 Windows now covered in CI for some basic cases

#3135 Formalize settings alias behavior and disallow conflicting values between setting name and aliases

#2035 Rename job table and job_id column in backend db

Fixes

#5985 - Address warning in Airflow plugin version check by using recommended packaging.version.Version.

Source code(tar.gz)
Source code(zip)
meltano-2.2.0rc0-py3-none-any.whl(3.53 MB)
v2.1.0(Jun 17, 2022)
New

https://github.com/meltano/meltano/pull/6114 Add a new meltano lock command to lock specific plugin definitions and optionally update existing lock files.

Fixes

https://github.com/meltano/meltano/pull/6227 Resolve a bug that would cause intermittent errors when calling meltano config airflow.

https://github.com/meltano/meltano/pull/6216 Fix bug in Meltano UI that would prevent schedules from being executed in the web interface.

Under the hood

Continued improvements to anonymized usage stats.

Full Changelog: https://github.com/meltano/meltano/compare/v2.0.3...v2.1.0
Source code(tar.gz)
Source code(zip)
v2.0.3(Jun 15, 2022)
What's Changed

Changes

https://github.com/meltano/meltano/pull/6186 Improve CLI startup performance.

Under the Hood

https://github.com/meltano/meltano/pull/6187, https://github.com/meltano/meltano/pull/6212 Various telemetry improvements.

Full Changelog: https://github.com/meltano/meltano/compare/v2.0.2...v2.0.3
Source code(tar.gz)
Source code(zip)
v2.0.2(Jun 15, 2022)
Fixes

#6174 Fix issue where loading schedules could cause Meltano UI to crash.

Docs

#6184 Bugfix in getting-started guide installation instructions - Thanks, Martin Morset (@dingobar)!

Misc

#6149,#6142 Add anonymized telemetry handling for exceptions.

Full Changelog: https://github.com/meltano/meltano/compare/v2.0.0...v2.0.2
Source code(tar.gz)
Source code(zip)
v2.0.1(Jun 9, 2022)
What's Changed

Fixes

#6143 Resolve issue where Meltano UI would not start.

Full Changelog:

https://github.com/meltano/meltano/compare/v2.0.0...v2.0.1
Source code(tar.gz)
Source code(zip)
v2.0.0(Jun 8, 2022)
2.0.0 - (2022-06-08)

Meltano 2.0 is a major update with multiple new features and some breaking changes. See our 2.0 Migration Guide for more information.

New

#3283 Support meltano add directly from Hub.

#2924 Initial meltano job support via new meltano job command and support for referencing jobs in meltano run commands.

#5923 Add support for jobs to schedules and improves general schedule cli UX.

#3348 Add env_var_strict_mode feature flag to raise an exception if unset environment variables are referenced in settings.

#6060 Generate lockfiles when running meltano add to install new plugins (now enabled by default).

#3355 New paradigm of explicit plugin dependencies using requires:, replaces implicitly "related" plugins.

Changes

#5957 Remove the auto-add functionality that applied to transform plugins and plugins with matching namespace.

#6043 Let Meltano settings be stored under environments.

Breaks

#3427 Drop support for the legacy "Dashboards" and "Explore" UX in Meltano UI.

#3427 Drop support for legacy model and dashboard plugin types.

#3048 Drop support for env_aliases feature and remove its references in discovery.yml.

#5957 Drop support for the --include-related CLI flag.

Fixes

#5979 Fixes meltano remove bug where lock files were not removed if they include a variant in the file name.

#6015 Fixes a bug where meltano config meltano set would not always set a value.

Under the Hood

Improved telemetry coverage across the product.

Source code(tar.gz)
Source code(zip)
v1.105.0(May 30, 2022)
1.105.0 - (2022-05-19)

New

#3298 Add adapter-specific dbt plugins for Redshift, Postgres, and BigQuery.

#3482 Expand project-set environment variables in Environment-level env:.

Changes

#3174 - We now recommend pipx over manually created virtual environments.

#3484 Use SettingsService.feature_flag convention for lockfiles.

Fixes

#3483 Fixes a bug where pipx installed meltano can not find executable(.exe) meltano dependency packages. Thanks Dan Norman!

#3500 Fixes bug where lock file was added but not used.

Source code(tar.gz)
Source code(zip)
v1.98.1(May 30, 2022)
1.98.1 - (2022-03-12)

Fixes

#3334 Set the executable of meltano-map-transformer to meltano-map-transform.

Source code(tar.gz)
Source code(zip)
v1.98.0(May 30, 2022)
1.98.0 - (2022-03-10)

Changes

#3316 Improve the meltano init experience with beautified text and cleaner logging

#3317 Improve Snowplow documentation and simplify telemetry notification during meltano init

#3319 Limit Snowplow tracker logs to the command line to ERROR level by default.

#3217 Allow user to configure Sendgrid asm_group_id. Thanks, @davesgonechina!

#3321 Switch Snowplow tracker endpoint to https.

Fixes

#3306 Fix Environments bug where custom configuration values that were not defined either as a setting or custom configuration in the primary plugin definition were not passed to the plugin in the Environment context.

Source code(tar.gz)
Source code(zip)
v1.97.0(May 30, 2022)
1.97.0 - (2022-03-03)

New

#2527, #2528 Add CRON interval mechanism and modal to /pipelines page.

#3017 Add Snowplow Tracking to Meltano.

#3130 Add support for incremental jobs to meltano run.

Changes

#3282 Expand meltano elt failure information to help new users find debug logging. Thanks, Derek Visch!

Fixes

#3289 In target-bigquery default variant adswerve, set the default of the merge_state_messages setting to false so state objects are not merged by the target.

#3301 Fix a typo that causes requests to the /install/batch endpoint to fail.

Source code(tar.gz)
Source code(zip)
v1.13.0(May 30, 2022)
1.13.0 - (2019-12-23)

New

#1269 Add kind: file so single file uploads can be used with extractors (tap-google-analytics's key_file_location is the first user)

#1494 Add LIKE options to Analyze Filter UI so users better understand what filtering patterns are available

Changes

#1399 Log Modal now has a prompt to explain potential factors in required time for pipelines to complete

#1433 Remove /orchestrate route and thus the Airflow iframe as this is overkill for our current target users

Fixes

#1434 Fix Analyze CTAs to only enable if at least one related pipeline has succeeded

#1447 Various fixes around loading and reloading reports to mitigate false positive sqlErrorMessage conditions

#1509 Allow plugin profile config to be set through meltano.yml

Source code(tar.gz)
Source code(zip)
v1.12.0(May 30, 2022)
1.12.0 - (2019-12-16)

New

#1374 Add role revocation for users and roles in the meltano permissions command

#1377 Document cleanup steps after MeltanoData testing

#1438 Add documentation for DNS spoofing error

#1436 Add video walkthrough on how to setup Google Analytics so that the Meltano Extractor can be able to access the Google APIs and the Google Analytics data.

Changes

#1350 Switch to all lower case for Snowflake permission comparisons in the meltano permissions command

#1449 Hide the Marketo Extractor form Meltano UI

#1397 Optimize workflow for MeltanoData setup

#1423 Update sidebar and docs to include Ansible

Source code(tar.gz)
Source code(zip)
v1.7.1(May 30, 2022)

Tag with regression - should be deleted
Source code(tar.gz)
Source code(zip)

Owner

Meltano

ELT for the DataOps era

GitHub Repository https://meltano.com

Shot notebooks resuming the main functions of GeoPandas

Shot notebooks resuming the main functions of GeoPandas, 2 notebooks written as Exercises to apply these functions.

1 Jan 12, 2022

Repositori untuk menyimpan material Long Course STMKGxHMGI tentang Geophysical Python for Seismic Data Analysis

Long Course "Geophysical Python for Seismic Data Analysis" Instruktur: Dr.rer.nat. Wiwit Suryanto, M.Si Dipersiapkan oleh: Anang Sahroni Waktu: Sesi 1

0 Dec 04, 2021

ETL flow framework based on Yaml configs in Python

ETL framework based on Yaml configs in Python A light framework for creating data streams. Setting up streams through configuration in the Yaml file.

18 Jul 06, 2022

Supply a wrapper ``StockDataFrame`` based on the ``pandas.DataFrame`` with inline stock statistics/indicators support.

Stock Statistics/Indicators Calculation Helper VERSION: 0.3.2 Introduction Supply a wrapper StockDataFrame based on the pandas.DataFrame with inline s

1.1k Dec 28, 2022

Python dataset creator to construct datasets composed of OpenFace extracted features and Shimmer3 GSR+ Sensor datas

3 Jul 05, 2022

Pandas and Dask test helper methods with beautiful error messages.

beavis Pandas and Dask test helper methods with beautiful error messages. test helpers These test helper methods are meant to be used in test suites.

18 Nov 28, 2022

PandaPy has the speed of NumPy and the usability of Pandas 10x to 50x faster (by @firmai)

PandaPy "I came across PandaPy last week and have already used it in my current project. It is a fascinating Python library with a lot of potential to

527 Jan 02, 2023

First and foremost, we want dbt documentation to retain a DRY principle. Every time we repeat ourselves, we waste our time. Second, we want to understand column level lineage and automate impact analysis.

dbt-osmosis First and foremost, we want dbt documentation to retain a DRY principle. Every time we repeat ourselves, we waste our time. Second, we wan

150 Jan 06, 2023

Data exploration done quick.

Pandas Tab Implementation of Stata's tabulate command in Pandas for extremely easy to type one-way and two-way tabulations. Support: Python 3.7 and 3.

20 Aug 27, 2022

ped-crash-techvol: Texas Ped Crash Tech Volume Pack

ped-crash-techvol: Texas Ped Crash Tech Volume Pack In conjunction with the Final Report "Identifying Risk Factors that Lead to Increase in Fatal Pede

2 Sep 28, 2022

BAyesian Model-Building Interface (Bambi) in Python.

Bambi BAyesian Model-Building Interface in Python Overview Bambi is a high-level Bayesian model-building interface written in Python. It's built on to

861 Dec 29, 2022

Universal data analysis tools for atmospheric sciences

U_analysis Universal data analysis tools for atmospheric sciences Script written in python 3. This file defines multiple functions that can be used fo

1 Oct 10, 2021

fds is a tool for Data Scientists made by DAGsHub to version control data and code at once.

Fast Data Science, AKA fds, is a CLI for Data Scientists to version control data and code at once, by conveniently wrapping git and dvc

359 Dec 22, 2022

ELFXtract is an automated analysis tool used for enumerating ELF binaries

ELFXtract ELFXtract is an automated analysis tool used for enumerating ELF binaries Powered by Radare2 and r2ghidra This is specially developed for PW

49 Nov 28, 2022

Data-sets from the survey and analysis

bachelor-thesis "Umfragewerte.xlsx" contains the orginal survey results. "umfrage_alle.csv" contains the survey results but one participant is cancele

1 Jan 26, 2022

PyTorch implementation for NCL (Neighborhood-enrighed Contrastive Learning)

NCL (Neighborhood-enrighed Contrastive Learning) This is the official PyTorch implementation for the paper: Zihan Lin*, Changxin Tian*, Yupeng Hou* Wa

73 Jan 03, 2023

Recommendations from Cramer: On the show Mad-Money (CNBC) Jim Cramer picks stocks which he recommends to buy. We will use this data to build a portfolio

Backtesting the "Cramer Effect" & Recommendations from Cramer Recommendations from Cramer: On the show Mad-Money (CNBC) Jim Cramer picks stocks which

12 Aug 30, 2022

Meltano: ELT for the DataOps era. Meltano is open source, self-hosted, CLI-first, debuggable, and extensible.

Related tags

Overview

Meltano: ELT for the DataOps era

Documentation

Contributing to Meltano

Responsible Disclosure Policy

License

Comments

Example: registering dbt packages in packages.yml

Comparison with other approaches

Meltano Version

Python Version

Bug scope

Operating System

Description

Code

Meltano Version

Python Version

Bug scope

Operating System

Description

Code

Workaround

Meltano Version

Python Version

Bug scope

Operating System

Description

Code

Running without project context

Related

Meltano Version

Python Version

Bug scope

Operating System

Description

The Fix

Code

Update (@aaronsteers)

Original Issue description (@pnadolny13)

Meltano Version

Python Version

Bug scope

Operating System

Description

Code

Releases(v2.12.0)

v2.12.0(Dec 19, 2022)

v2.12.0 (2022-12-19)

🐛 Fixes

⚡ Performance Improvements

📚 Documentation Improvements

v2.11.1(Dec 9, 2022)

v2.11.1 (2022-12-09)

🐛 Fixes

v2.11.0(Dec 8, 2022)

v2.11.0 (2022-12-08)

✨ New

🐛 Fixes

📚 Documentation Improvements

v2.10.0(Nov 16, 2022)

v2.10.0 (2022-11-16)

✨ New

🐛 Fixes

v2.9.0(Nov 8, 2022)

v2.9.0 (2022-11-08)

✨ New

🐛 Fixes

📚 Documentation Improvements

v2.8.0(Oct 18, 2022)

v2.8.0 (2022-10-17)

✨ New

🐛 Fixes

⚙️ Changes

📚 Documentation Improvements

v2.7.2(Oct 3, 2022)

v2.7.2 (2022-10-03)

🐛 Fixes

📚 Documentation Improvements

Example: registering dbt packages in `packages.yml`