Hi there--thanks for the great tool!

I'm getting an AttributeError when running dbt-osmosis compose and dbt-osmosis run. See the stacktrace below. My dbt_project.yml is attached, as well.

> dbt-osmosis run -f acdw.dimensions.acdw__dimensions__item 
INFO     🌊 Executing dbt-osmosis                                                                                                                                                                                                                                         main.py:78

INFO     📈 Searching project stucture for required updates and building action plan                                                                                                                                                                                  osmosis.py:907
INFO     ...building project structure mapping in memory                                                                                                                                                                                                              osmosis.py:885
INFO     [('CREATE', '->', WindowsPath('C:/my_project_path/models/acdw/dimensions/schema/acdw__dimensions__item.yml'))]                                                                                                                         osmosis.py:1027
INFO     👷 Executing action plan and conforming projecting schemas to defined structure                                                                                                                                                                              osmosis.py:977
INFO     🚧 Building schema file acdw__dimensions__item.yml                                                                                                                                                                                                           osmosis.py:983
Traceback (most recent call last):
  File "C:\Users\me\Anaconda3\envs\dbt\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\me\Anaconda3\envs\dbt\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\me\Anaconda3\envs\dbt\Scripts\dbt-osmosis.exe\__main__.py", line 7, in <module>
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\dbt_osmosis\main.py", line 89, in run
    if runner.commit_project_restructure_to_disk():
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\dbt_osmosis\core\osmosis.py", line 987, in commit_project_restructure_to_disk
    self.yaml.dump(structure.output, target)
AttributeError: 'DbtYamlManager' object has no attribute 'yaml'

Possibly relatedly, I'm not able to run dbt-osmosis workbench, either. Apparently, dbt_osmosis can't find itself.

> dbt-osmosis workbench
INFO     🌊 Executing dbt-osmosis                                                                                                                                                                                                                                        main.py:406
[...]
2022-10-06 17:41:59.975 Pandas backend loaded 1.4.2
2022-10-06 17:41:59.981 Numpy backend loaded 1.21.5
2022-10-06 17:41:59.982 Pyspark backend NOT loaded
2022-10-06 17:41:59.983 Python backend loaded
2022-10-06 17:42:01.020 Uncaught app exception
Traceback (most recent call last):
  File "C:\Users\me\Anaconda3\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 562, in _run_script
    exec(code, module.__dict__)
  File "C:\Users\me\Anaconda3\envs\dbt\Lib\site-packages\dbt_osmosis\app.py", line 17, in <module>
    from dbt_osmosis.core.osmosis import DEFAULT_PROFILES_DIR, DbtProject
ModuleNotFoundError: No module named 'dbt_osmosis'

dbt_project.zip

Hello, I'm getting following error : AttributeError: 'CompiledSqlNode' object has no attribute 'compiled_sql' Traceback: File "/home/dbt/venv_dbt/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 562, in _run_script exec(code, module.dict) File "/home/dbt/venv_dbt/lib/python3.10/site-packages/dbt_osmosis/app.py", line 327, in if compile_sql(state[RAW_SQL]) != state[COMPILED_SQL]: File "/home/dbt/venv_dbt/lib/python3.10/site-packages/dbt_osmosis/app.py", line 197, in compile_sql return ctx.compile_sql(sql) File "/home/dbt/venv_dbt/lib/python3.10/site-packages/dbt_osmosis/core/osmosis.py", line 367, in compile_sql return self.get_compiled_node(sql).compiled_sql

any ide how to fix that ? 

Good day,

When trying to run the server within docker, I'm getting the error

ERROR:    [Errno 99] error while attempting to bind on address ('::1', 8581, 0, 0): cannot assign requested address

I'm guessing this is because the program was written on IPv6 instead of IPv4. I was wondering if it would be possible to support IPv4 as well.

Thanks!

Hi, I installed the dbt-osmosis and tried to execute the dbt-osmosis workbench command but I get error that No such file or directory: 'streamlit'

here is the full error msg:

INFO     🌊 Executing dbt-osmosis                                                            main.py:390
                                                                                                        
Traceback (most recent call last):
  File "/Users/galpolak/Library/Python/3.9/bin/dbt-osmosis", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/galpolak/Library/Python/3.9/lib/python/site-packages/dbt_osmosis/main.py", line 414, in workbench
    subprocess.run(
  File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'streamlit'

Hello and thanks for looking at my issue. When I click on this button after opening a model sql file in VS Code,

I get this error.

The actual path is c:\\Temp\git\au-nvg-business-intelligence\\dbt\navigate_bi\\dbt_project.yml. Because there are single backslashes in 2 spots (\au-nvg-xxx and \navigate_bi\xxxx), both are being translated to \a and \n. I don't know why a mixture of single and double backslashes are coming in the file name and path.

I've had to put a temp fix in osmosis.py in line 181 to get this working. See project_dir being added with replace calls.

args = PseudoArgs(
            threads=threads,
            target=target,
            profiles_dir=profiles_dir,
            #project_dir=project_dir,
            project_dir=project_dir.replace('\t', '\\t').replace('\n', '\\n').replace('\r', '\\r').replace('\a', '\\a'),
        )

Encountering below page output (and on CLI as well) when executing dbt-osmosis workbench -m unique_model_name. Seems it's getting a none value for a file path, but I'm not sure where or how exactly. This is my first run of dbt-osmosis workbench.

AttributeError: 'NoneType' object has no attribute 'patch_path'
Traceback:
File "C:\Users\myusername\AppData\Local\Programs\Python\Python310\lib\site-packages\streamlit\scriptrunner\script_runner.py", line 554, in _run_script
    exec(code, module.__dict__)
File "C:\Users\myusername\AppData\Local\Programs\Python\Python310\lib\site-packages\dbt_osmosis\app.py", line 345, in <module>
    Path(ctx.project_root) / ctx.get_patch_path(st.session_state[BASE_NODE]),
File "C:\Users\myusername\AppData\Local\Programs\Python\Python310\lib\site-packages\dbt_osmosis\core\osmosis.py", line 179, in get_patch_path
    return Path(node.patch_path.split(FILE_ADAPTER_POSTFIX)[-1])

Lets reduce scope in our workbench to a hyper tailored experience for A) Creating a new model B) Editing an existing model

There should (probably) be two separate but similar layouts for this.

I'm having this error :

    if SCHEMA_FILE.current is None:
AttributeError: 'NoneType' object has no attribute 'current'

Looking at the code in main.py and app.py, I've notices many things.

Here is an extract of the content of the « project » variable (dict) : {'name': 'MYPROJ', 'version': '1.0.1', 'project-root': 'my/path/MYPROJ/dbt', ....}

But in the scipt, to get the name and root path of the project it's done like that : project.project_name instead of project.name project.project_root instead of project.project-root

And SCHEMA_FILE is None because the proj variable, that should contain project.project_name => MYPROJvalue, has : my/path/MYPROJ/dbt. And node.get("package_name") = MYPROJ So the test comparing "node.get("package_name") == proj" is always False so SCHEMA_FILE is logically None.

In our environment people write everything in lower case including both the SQL and the YML. It looks like the SNOWFLAKE adapter consistently returns column names as upper case. Can you provide an option to ignore case in comparisons of column names between the YML version and the DB?

I "fixed" this locally by changing this line to read c.name.lower() in the obvious place: https://github.com/z3z1ma/dbt-osmosis/blob/05ed698a059f855556a0ef59ab3c9cb578ce680e/src/dbt_osmosis/main.py#L438

But you might instead want to keep that case and modify all the places where you compare columns to do a case insensitive comparison. Looks like you're using python sets for the different column lists. I don't know how you'd make them case insensitive, so just adding the lower() maybe as a configuration would be great.

TODO: Extend description

I think our scope does not require us to limit ourselves to "models" and we should attach ourselves to the SQL itself. This simplifies the whole process as can be seen in this PR.

DbtYamlManager.yaml_handler assumed that if a schema.yml file existed, it contained data. This caused a NoneType is not iterable exception when 1) reading an empty file, then 2) checking it for a key.

When a user runs dbt-osmosis diff -m some_model ... they are getting AttributeError: 'DbtOsmosis':

dbt-osmosis diff -m some_model --project-dir /path/to/dbt/transformations --profiles-dir /path/to/dbt/transformations/profiles --target target0

INFO     🌊 Executing dbt-osmosis
                                                                                                                                                                                                             
INFO     Injecting macros, please wait...                                                                                                                                                         
Traceback (most recent call last):
  File "/Users/myuser/.pyenv/versions/3.10.7/bin/dbt-osmosis", line 8, in <module>
    sys.exit(cli())
.....
AttributeError: 'DbtOsmosis' object has no attribute '_get_exec_node'. Did you mean: 'get_ref_node'?

Note

Please note that no error when running the dbt-osmosis workbench - that one runs nicely.

Configuration:

• dbt version: 1.2.0
• python version: Python 3.10.7

The sample streamlit app linked in the README returns an error.

Broken link: https://z3z1ma-dbt-osmosis-srcdbt-osmosisapp-4y67qs.streamlit.app/

Screenshot:

Proposed Behavior

When a user runs dbt-osmosis yaml document it will also pull in and preserve struct level documentation of fields in bigquery projects.

Current behavior

When there are struct fields in the schema.yml or source.yml the process removes any struct documentation that is specified as a field like name: struct.struct_field

Specifications

• dbt version: 1.2.0
• python version: 3.8.10

Symptoms: My dbt installation uses the query-comment config to add labels to BigQuery queries. In our dbt_project.yml we have

query-comment:
  comment: "{{ query_comment(node) }}"
  job-label: true

When trying to run a dbt-osmosis server serve...., the server throws a 500 Could not connect to Database error.

Diagnosis: Trailing things through the code, I reached parse_project in osmosis.py. This inits the DB adapter, loads the manifest, and importantly calls save_macros_to_adapter. BUT... before we get to the save macros, the adapter setter fn calls _verify_connection which tries to query the DB before the adapter is fully initialised...

Now the adapter errors in this chunk in the dbt-bigquery connection

if (
            hasattr(self.profile, "query_comment")
            and self.profile.query_comment
            and self.profile.query_comment.job_label
        ):
            query_comment = self.query_header.comment.query_comment. 
            labels = self._labels_from_query_comment(query_comment)

failing with 'NoneType' object has no attribute 'comment' on line 6 of the snippet. Now this code looks ugly - it's checking the contents of self.profile, then trying to access self.query-header - clearly a massive assumption is made here...

So - we have a conflict. dbt-osmosis is trying to run a query without ensuring the adapter is completely configured - and dbt-bigquery is doing a really wonky test. I'm happy to PR a deferred verify process in dbt-osmosis, or if you consider this to be more of a dbt-bigquery bug, I'll raise it with them.

In this issue @z3z1ma you say that the preferred method of development is with the dbt-server, and in the server docs you imply that you can use the server with dbt-power-user. I've tried to stitch together the solution but am coming up short.

Maybe you could provide an explanation for how to make the integration work and update the docs? Right now dbt-power-user points to this repo for how to set up a dbt REPL, so if you could make that part of the docs more clear I think it could help other people as well.

I think it would be pretty useful to have a command to execute the audit standalone. For example this line dbt-osmosis yaml audit -f development.test would have the following output:

             ✅ Audit Report                                                                                                                                                                                                               
             -------------------------------                                                                                                                                                                                               
                                                                                                                                                                                                                                           
             Database: awsdatacatalog                                                                                                                                                                                                      
             Schema: development                                                                                                                                                                                                 
             Table: test                                                                                                                                                                                                          
                                                                                                                                                                                                                                           
             Total Columns in Database: 3.0
             Total Documentation Coverage: 100.0%        
                                                                                                                                                                                                                                           
             Action Log:                                                                                                                                                                                                                   
             Columns Added to dbt: 2                                                                                                                                                                                                     
             Column Knowledge Inherited: 1                                                                                                                                                                                                 
             Extra Columns Removed: 0   

Then, one would be able to do whatever they want with it: post it as a comment in a PR or send it as a Slack message, etc