A SQL linter and auto-formatter for Humans

Last update: Jan 08, 2023

Overview

The SQL Linter for Humans

SQLFluff is a dialect-flexible and configurable SQL linter. Designed with ELT applications in mind, SQLFluff also works with Jinja templating and dbt. SQLFluff will auto-fix most linting errors, allowing you to focus your time on what matters.

Dialects Supported

Although SQL is reasonably consistent in its implementations, there are several different dialects available with variations of syntax and grammar. SQLFluff currently supports the following SQL dialects (though perhaps not in full):

ANSI SQL - this is the base version and on occasion may not strictly follow the ANSI/ISO SQL definition
BigQuery
Exasol
Hive
MySQL
PostgreSQL (aka Postgres)
Redshift
Snowflake
Spark3
SQLite
Teradata
Transact-SQL (aka T-SQL)

We aim to make it easy to expand on the support of these dialects and also add other, currently unsupported, dialects. Please raise issues (or upvote any existing issues) to let us know of demand for missing support.

Pull requests from those that know the missing syntax or dialects are especially welcomed and are the question way for you to get support added. We are happy to work with any potential contributors on this to help them add this support. Please raise an issue first for any large feature change to ensure it is a good fit for this project before spending time on this work.

Templates Supported

SQL itself does not lend itself well to modularity, so to introduce some flexibility and reusability it is often templated as discussed more in our modularity documentation.

SQLFluff supports the following templates:

Jinja (aka Jinja2)
dbt

Again, please raise issues if you wish to support more templating languages/syntaxes.

Getting Started

To get started, install the package and run sqlfluff lint or sqlfluff fix.

$ pip install sqlfluff
$ echo "  SELECT a  +  b FROM tbl;  " > test.sql
$ sqlfluff lint test.sql
== [test.sql] FAIL
L:   1 | P:   1 | L003 | Single indentation uses a number of spaces not a multiple of 4
L:   1 | P:  14 | L006 | Operators should be surrounded by a single space unless at the start/end of a line
L:   1 | P:  27 | L001 | Unnecessary trailing whitespace

Alternatively, you can use the Official SQLFluff Docker Image or have a play using SQLFluff online.

For full CLI usage and rules reference, see the SQLFluff docs.

Documentation

For full documentation visit docs.sqlfluff.com. This documentation is generated from this repository so please raise issues or pull requests for any additions, corrections, or clarifications.

Releases

SQLFluff is in beta phase - expect the tool to change significantly with potentially non-backwards compatible API and configuration changes in future releases. If you would like to join in please consider contributing.

New releases are made monthly. For more information, visit Releases.

SQLFluff on Slack

We have a fast-growing community on Slack, come and join us!

SQLFluff on Twitter

Contributing

We are grateful to all our contributors. There is a lot to do in this project, and we are just getting started.

If you want to understand more about the architecture of SQLFluff, you can find more here.

If you would like to contribute, check out the open issues on GitHub. You can also see the guide to contributing.

Comments

Result of `fix` changes on subsequent runs (fix is non-deterministic)

Expected Behaviour

Running sqlfluff fix my_query.sql -f modifies query format – but running it again on the same file should not make any further changes.

Observed Behaviour

Summary: The result changes considerably also on the 2nd run. On further runs the result keeps changing with a minor whitespace change.

The behavior is demonstrated in this branch: https://github.com/juhoautio/sqlfluff/commits/non_deterministic_fix_bug

1st run: https://github.com/juhoautio/sqlfluff/commit/bc60c5767245be92b1fc7ff1f5666253e7ea7af5
2nd run: https://github.com/juhoautio/sqlfluff/commit/148c91d81808a845a4e068d864e5a418906d4acd
3rd run: https://github.com/juhoautio/sqlfluff/commit/befff642abeb6a5f4de05ccbce9ea38203e01a9f
4th run: https://github.com/juhoautio/sqlfluff/commit/d9bcbf6c984fa0c7138b99009f5898a38681cbdc

Console output for each command is included below.

1st run of `sqlfluff fix`

$ python -m sqlfluff fix my_query.sql -f
==== finding fixable violations ====
== [my_query.sql] FAIL
L:   3 | P:  20 | L010 | Keywords must be consistently upper case.
L:   4 | P:  20 | L010 | Keywords must be consistently upper case.
L:   5 | P:  23 | L010 | Keywords must be consistently upper case.
L:   7 | P:  16 | L008 | Commas should be followed by a single whitespace unless
                       | followed by a comment.
L:   9 | P:   2 | L003 | Indentation not consistent with line #8
L:   9 | P:   2 | L022 | Blank line expected but not found after CTE closing
                       | bracket.
L:  10 | P:   6 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  11 | P:  10 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  12 | P:  10 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  13 | P:  10 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  13 | P:  10 | L030 | Function names must be consistently lower case.
L:  14 | P:   6 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  14 | P:  30 | L031 | Avoid aliases in from clauses and join conditions.
L:  15 | P:  15 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  15 | P:  15 | L051 | INNER JOIN must be fully qualified.
L:  15 | P:  38 | L031 | Avoid aliases in from clauses and join conditions.
L:  16 | P:  15 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  17 | P:  15 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  17 | P:  15 | L051 | INNER JOIN must be fully qualified.
L:  17 | P:  44 | L031 | Avoid aliases in from clauses and join conditions.
L:  18 | P:  15 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  19 | P:  17 | L003 | Line over-indented compared to line #8
L:  20 | P:   6 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  21 | P:   8 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  22 | P:   8 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  23 | P:   6 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  24 | P:   2 | L003 | Indentation not consistent with line #8
L:  25 | P:   2 | L003 | Indentation not consistent with line #8
L:  25 | P:   2 | L022 | Blank line expected but not found after CTE closing
                       | bracket.
L:  26 | P:   6 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  27 | P:  10 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  27 | P:  23 | L010 | Keywords must be consistently upper case.
L:  28 | P:  10 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  28 | P:  23 | L010 | Keywords must be consistently upper case.
L:  29 | P:  10 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  29 | P:  29 | L006 | Missing whitespace before -
L:  29 | P:  29 | L006 | Missing whitespace after -
L:  29 | P:  42 | L010 | Keywords must be consistently upper case.
L:  30 | P:  10 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  30 | P:  30 | L006 | Missing whitespace before -
L:  30 | P:  30 | L006 | Missing whitespace after -
L:  30 | P:  41 | L006 | Missing whitespace before /
L:  30 | P:  41 | L006 | Missing whitespace after /
L:  30 | P:  52 | L006 | Missing whitespace before *
L:  30 | P:  52 | L006 | Missing whitespace after *
L:  30 | P:  58 | L010 | Keywords must be consistently upper case.
L:  30 | P:  78 | L016 | Line is too long
L:  31 | P:   6 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  31 | P:  26 | L011 | Implicit/explicit aliasing of table.
L:  31 | P:  26 | L031 | Avoid aliases in from clauses and join conditions.
L:  32 | P:   6 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  32 | P:  31 | L011 | Implicit/explicit aliasing of table.
L:  32 | P:  31 | L031 | Avoid aliases in from clauses and join conditions.
L:  33 | P:   6 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  34 | P:   6 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  35 | P:   2 | L003 | Indentation not consistent with line #8
L:  36 | P:   1 | L022 | Blank line expected but not found after CTE closing
                       | bracket.
L:  39 | P:   9 | L030 | Function names must be consistently lower case.
L:  40 | P:  23 | L005 | Commas should not have whitespace directly before them.
L:  40 | P:  33 | L030 | Function names must be consistently lower case.
L:  42 | P:  13 | L030 | Function names must be consistently lower case.
L:  46 | P:   7 | L003 | Indentation not hanging or a multiple of 4 spaces
L:  46 | P:  36 | L010 | Keywords must be consistently upper case.
L:  46 | P:  45 | L048 | Missing whitespace after '2'
L:  55 | P:   1 | L022 | Blank line expected but not found after CTE closing
                       | bracket.
L:  57 | P:   9 | L030 | Function names must be consistently lower case.
L:  64 | P:  13 | L030 | Function names must be consistently lower case.
L:  69 | P:   1 | L022 | Blank line expected but not found after CTE closing
                       | bracket.
==== fixing violations ====
68 fixable linting violations found
FORCE MODE: Attempting fixes...
Persisting Changes...
== [my_query.sql] PASS
Done. Please check your files to confirm.
  [16 unfixable linting violations found]

Resulting diff here: https://github.com/juhoautio/sqlfluff/commit/bc60c5767245be92b1fc7ff1f5666253e7ea7af5

Many formatting fixes, as expected. The result looks good!

2nd run of `sqlfluff fix`

$ python -m sqlfluff fix my_query.sql -f
==== finding fixable violations ====
== [my_query.sql] FAIL
L:  31 | P:  87 | L016 | Line is too long
L:  33 | P: 100 | L016 | Line is too long
L:  51 | P:  47 | L048 | Missing whitespace after '2'
==== fixing violations ====
3 fixable linting violations found
FORCE MODE: Attempting fixes...
Persisting Changes...
== [my_query.sql] PASS
Done. Please check your files to confirm.
  [2 unfixable linting violations found]

Resulting diff here: https://github.com/juhoautio/sqlfluff/commit/148c91d81808a845a4e068d864e5a418906d4acd

The formatting still changes greatly. I'm not sure if this is for better or worse, but that's a matter of taste. The problem is that formatting changes at all.

3rd run of `sqlfluff fix`

$ python -m sqlfluff fix my_query.sql -f
==== finding fixable violations ====
== [my_query.sql] FAIL
L:  57 | P:  47 | L048 | Missing whitespace after '2'
==== fixing violations ====
1 fixable linting violations found
FORCE MODE: Attempting fixes...
Persisting Changes...
== [my_query.sql] PASS
Done. Please check your files to confirm.
  [2 unfixable linting violations found]

Resulting diff here: https://github.com/juhoautio/sqlfluff/commit/befff642abeb6a5f4de05ccbce9ea38203e01a9f

Now formatting doesn't change much but it still does: redundant whitespace is added.

4th run of `sqlfluff fix`

$ python -m sqlfluff fix my_query.sql -f
==== finding fixable violations ====
== [my_query.sql] FAIL
L:  57 | P:  47 | L048 | Missing whitespace after '2'
==== fixing violations ====
1 fixable linting violations found
FORCE MODE: Attempting fixes...
Persisting Changes...
== [my_query.sql] PASS
Done. Please check your files to confirm.
  [2 unfixable linting violations found]

Resulting diff here: https://github.com/juhoautio/sqlfluff/commit/d9bcbf6c984fa0c7138b99009f5898a38681cbdc

Adds yet another redundant space character. I suspect it would keep adding it until infinity if sqlfluff fix is run again.

Steps to Reproduce

Get my_query.sql from here: https://github.com/juhoautio/sqlfluff/commit/2b99acd9bdfd355fb38c8fafe1d25b43cb2c22b3

(This query is an obfuscated version of a query that executes successfully in AWS Athena)

Run python -m sqlfluff fix my_query.sql -f and check the file for changes after each execution.

Dialect

Not specified. Dialects are not important here: the expectation here is that sqlfluff can do generic autoformatting on any sql file and the result shouldn't change after the initial run.

Version

To make sure that these bugs haven't been fixed recently but are not in a release yet, I ran this with local dev version of sqlfluff.

The version I used was the latest commit in main branch: https://github.com/sqlfluff/sqlfluff/commit/7fd452dbfb9ab8537b8ca6e92c9489ff04d6172b

$ tox -e py --devenv .venv
$ source .venv/bin/activate

$ python -m sqlfluff --version
python -m sqlfluff, version 0.9.0

$ python --version
Python 3.7.5

Not using dbt.

Configuration

N/A

bug core

opened by juhoautio 53

dbt.exceptions.InternalException: Asked to compile node, but it has no compiled form

Search before asking

[X] I searched the issues and found no similar issues.

What Happened

While linting any dbt model the error is raised part-way through linting (apparently after all the fail lines have been produced and displayed).

I've tried sqlfluff versions

0.10.1 - no error 0.11.0, 0.13.2, 1.0.0, 1.1.0 - error thrown

Expected Behaviour

sqlfluff reports linting failures.

Observed Behaviour

SQL model materialising as a view. This is really the WEB_USERS model from the fivetran segment dbt package, just renamed to use our naming standard.

with

SOURCE as (
  select
    *
  from
    {{ ref('SG__WEB_USERS__UNION') }}
),

RENAMED as (
  select
    _DBT_SOURCE_RELATION,
    EMAIL,

    {{ sv_utils.get_email_domain('EMAIL') }} as EMAIL_DOMAIN,

    {{ sv_utils.is_valid_email('EMAIL') }} as IS_EMAIL_VALID,

    ID as USER_ID,
    PERSON_ID,
    USERNAME,
    EMAIL_ADDRESS,
    {{ sv_utils.get_email_domain('EMAIL_ADDRESS') }} as EMAIL_ADDRESS_DOMAIN,
    {{ sv_utils.is_valid_email('EMAIL_ADDRESS') }} as IS_EMAIL_ADDRESS_VALID,
    "NAME" as FULL_NAME,
    SHORT_NAME,
    FIRST_NAME,
    INITIALS,
    SURNAME,
    LAST_NAME,
    USER_TYPE,

    CONTEXT_IP as IP,
    -- CONTEXT_LIBRARY_NAME, -- not useful
    SUBSCRIPTIONS,
    CONTEXT_PAGE_SEARCH as PAGE_URL_QUERY,
    CONTEXT_PAGE_URL as PAGE_URL,
    {{ dbt_utils.get_url_host('CONTEXT_PAGE_URL') }} as PAGE_URL_HOST,
    -- CONTEXT_LIBRARY_VERSION, -- not useful
    CONTEXT_PAGE_REFERRER as PAGE_REFERRER,

    replace(
      {{ dbt_utils.get_url_host('CONTEXT_PAGE_REFERRER') }},
      'www.',
      ''
    ) as PAGE_REFERRER_HOST,

    CONTEXT_PAGE_TITLE as PAGE_TITLE,
    CONTEXT_LOCALE as LOCALE,
    CONTEXT_PAGE_PATH as PAGE_PATH,
    CONTEXT_CAMPAIGN_SOURCE as UTM_SOURCE,
    CONTEXT_CAMPAIGN_MEDIUM as UTM_MEDIUM,
    CONTEXT_CAMPAIGN_NAME as UTM_CAMPAIGN,
    BISLR_ID,
    CONTEXT_USER_AGENT as USER_AGENT,
    case
      when lower(context_user_agent) like '%android%' then 'Android'
      else replace(
        {{ dbt_utils.split_part(
          dbt_utils.split_part('CONTEXT_USER_AGENT',"'('",2),"' '",1) 
        }},
        ';','')
    end as device,
    UUID_TS,
    RECEIVED_AT as RECEIVED_AT_DATE_TIME,
    "ROLE" as COMPANY_ROLE,
    AGENCY_ID,
    ORGANISATION_ID,
    COMPANY_ID,
    COMPANY,
    AGENCY_NAME,
    ORGANISATION_NAME,
    COMPANY_NAME,
    ORGANISATION_TRADING_STYLES,
    COMPANY_FINANCIAL_FORCE_ID,
    COMPANY_CURRENCIES,
    COMPANY_RISK_PROFILES,

    AGENCIES,
    HAS_LTD_TERMS,
    AGENCY_ACCOUNT_STATUS,
    PROPOSAL_STATE,
    HAS_LABS_ACCESS,
    HAS_PAYE_TERMS,
    AGENCY_PAYROLL_PROVIDER,
    AML_STATUS,
    AGENCY_SIGN_UP_STATE,
    TIME_SHEETS_AVAILABLE,
    PICTURE_URL,
    AGENCY_HAS_FULL_ACCESS,
    ONLINE_TIMESHEET_PROVIDER,

    ROLES,
    ROLES_IS_SUPER_USER,
    ROLES_IS_CLIENT_AUTHORISER,
    ROLES_IS_SONOVATE_ADMIN,
    ROLES_IS_CANDIDATE,
    PERMISSIONS as USERS_PERMISSIONS,
    PERMISSIONS_IS_CLIENT_AUTHORISER,
    PERMISSIONS_IS_SONOVATE_ADMIN,
    PERMISSIONS_IS_CANDIDATE,

    "PROFILE" as USERS_PROFILE,
    "SESSION_ID" as USERS_SESSION_ID,
    AUTHENTICATED,
    IMAGE_URL,
    AUTHENTICATION_TOKEN,
    TOKEN,
    PERSON_TOKEN,

    LANGUAGE_CODE,
    SETTINGS_DATE_FORMAT,
    SETTINGS_SUBSCRIPTIONS,
    SETTINGS_TIME_ZONE,
    SETTINGS_LANGUAGE,
    SETTINGS_TIME_FORMAT,
    SETTINGS_COUNTRY_CODE,
    SETTINGS_WEEK_START

  from
    SOURCE
),

FINAL as (
  select 
    *,
    case
      when DEVICE = 'iPhone' then 'iPhone'
      when DEVICE = 'Android' then 'Android'
      when DEVICE in ('iPad', 'iPod') then 'Tablet'
      when DEVICE in ('Windows', 'Macintosh', 'X11')
        then 'Desktop'
      else 'Uncategorized' end as DEVICE_CATEGORY
  from
    RENAMED
)

select
  *
from
  FINAL

Example verbose output

fluff.txt

How to reproduce

Just a basic sqlfluff lint operation

sqlfluff lint .\models\base\segment\SG__WEB_USERS.sql

Dialect

snowflake

Version

0.11.0 0.13.2 1.0.1 1.1.0

Configuration

.sqlfluff

# dbt templating does not keep trailing new lines (L009)

[sqlfluff]
verbose = 3
templater = dbt
dialect = snowflake
exclude_rules = L009,L031,L034
ignore_templated_areas = True
sql_file_exts = .sql,.sql.j2,.dml,.ddl,.SQL

[sqlfluff:rules]
tab_space_size = 2

[sqlfluff:rules:L010]
capitalisation_policy = lower

[sqlfluff:rules:L014]
extended_capitalisation_policy = upper

[sqlfluff:rules:L016]
max_line_length = 180

[sqlfluff:rules:L019]
comma_style = leading

[sqlfluff:rules:L030]
extended_capitalisation_policy = lower

[sqlfluff:rules:L040]
capitalisation_policy = lower

[sqlfluff:templater:dbt]
apply_dbt_builtins = True
load_macros_from_path=./dbt_modules/dbt_utils/macros,./macros

[sqlfluff:templater:dbt:macros]
# Macros provided as builtins for dbt projects
dbt_ref = {% macro ref(model_ref) %}{{model_ref}}{% endmacro %}
dbt_source = {% macro source(source_name, table) %}{{source_name}}_{{table}}{% endmacro %}
dbt_config = {% macro config() %}{% for k in kwargs %}{% endfor %}{% endmacro %}
dbt_var = {% macro var(variable) %}item{% endmacro %}
dbt_is_incremental = {% macro is_incremental() %}True{% endmacro %}

[sqlfluff:templater:jinja:context]
num_things=456
tbl_name=my_table
my_list=['a', 'b', 'c']
MY_LIST=("d", "e", "f")

[sqlfluff:templater:jinja]
apply_dbt_builtins = True
load_macros_from_path=./dbt_modules/dbt_utils/macros

[sqlfluff:templater:jinja:macros]
# Macros provided as builtins for dbt projects
dbt_ref = {% macro ref(model_ref) %}{{model_ref}}{% endmacro %}
dbt_source = {% macro source(source_name, table) %}{{source_name}}_{{table}}{% endmacro %}
dbt_config = {% macro config() %}{% for k in kwargs %}{% endfor %}{% endmacro %}
dbt_var = {% macro var(variable) %}item{% endmacro %}
dbt_is_incremental = {% macro is_incremental() %}True{% endmacro %}

.sqlfluffignore

target/
dbt_modules/
macros/

Are you willing to work on and submit a PR to address the issue?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

bug dbt

opened by reubster 50

BigQuery UDFs (#246) and array literals (#337)
Addresses #246 and #337

[x] Add support for complex parameter types, e.g. ARRAY<STRUCT<product_id INT64, rating FLOAT64>>

[x] Add support for [OPTIONS (library = library_array)]

[x] Test this PR against our real-life projects' UDFs
opened by barrywhart 47
Progressbar utility
Fixes #1536

There is my proposition for the progressbar utility.

single file

for the whole directory

multiple paths

It is enabled by default (can be disabled with --disable_progress_bar flag) and rendered for linting and fixing (not for parsing).

The steps are iterated over

paths (if more than 1)

files from path (if more than 1)

rules for each file

also there is one dummy semi-progress bar rendered during parsing, which is just for showing that something happens by printing a label, not a real progress, but it is because of nature of recurrent parsing.

Pull Request checklist

[x] Please confirm you have completed any of the necessary steps below.

Included test cases to demonstrate any code changes, which may be one or more of the following:

.yml rule test cases in test/fixtures/rules/std_rule_cases.

.sql/.yml parser test cases in test/fixtures/dialects (note YML files can be auto generated with python test/generate_parse_fixture_yml.py or by running tox locally).

Full autofix test cases in test/fixtures/linter/autofix.

Other.

Added appropriate documentation for the change.

Created GitHub issues for any relevant followup/future enhancements if appropriate.
opened by adam-tokarski 46
Generate substitution placeholders for Jinja "{{ }}" sections
[x] Fix the "initial blank line before empty substitution" bug

[x] Update TemplateSegment to allow empty source_str, update substitution_end instances to specify blank string

[x] Test where substitution (blank or not) occurs at the beginning and end of a file, also with 1 or 2 blank lines before or after. Add test(s) that rules L050 and L009 work correctly in these cases.

[x] If the template references an undefined variable, then the placeholder gets double added within the unparseable section.

[x] Failing L003 tests -- these are mostly failing because L003 tries to indent/align the placeholder blocks. Needs updating to account for these substitution ones (probably just ignore them).

[ ] Try adding the "empty substitution" segments as a post-processing step. Trying to include them in "source only slices" is affecting position markers and, ultimately, the behavior of Linter.remove_templated_errors(), which is breaking one or more tests (details in this PR comment).

Brief summary of the change made

Fixes #2066

Are there any other side effects of this change that we should be aware of?

Pull Request checklist

[ ] Please confirm you have completed any of the necessary steps below.

Included test cases to demonstrate any code changes, which may be one or more of the following:

.yml rule test cases in test/fixtures/rules/std_rule_cases.

.sql/.yml parser test cases in test/fixtures/dialects (note YML files can be auto generated with python test/generate_parse_fixture_yml.py or by running tox locally).

Full autofix test cases in test/fixtures/linter/autofix.

Other.

Added appropriate documentation for the change.

Created GitHub issues for any relevant followup/future enhancements if appropriate.
opened by barrywhart 31
Parser incorrectly swaps BigQuery struct grammars when `parse_grammar` is not supplied.
Search before asking

[X] I searched the issues and found no similar issues.

What Happened

When a STRUCT is used in conjunction with an AS alias, it's columns are recognised as datatypes, which can trigger L063.

Expected Behaviour

STRUCT column names should not trigger L063.

Observed Behaviour

L: 3 | P: 9 | L063 | Datatypes must be upper case. L: 4 | P: 9 | L063 | Datatypes must be upper case.

How to reproduce

Run sqlfluff lint on

SELECT STRUCT( some_field, some_other_field ) AS col, STRUCT( some_field, some_other_field ) FROM table

Dialect

bigquery

Version

0.13.1

Configuration

[sqlfluff] templater = jinja dialect = bigquery [sqlfluff:rules:L063] # Inconsistent capitalisation of datatypes extended_capitalisation_policy = upper

Are you willing to work on and submit a PR to address the issue?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

bug parser bigquery
opened by dmohns 28

L030: built-in tsql functions should be capitalized by L030 while protecting user defined functions

Search before asking

[X] I searched the issues and found no similar issues.

Description

Looking at my code formatted with sqlfluff so far,

I found that for example month(), day(), year(), sum(), iif() are not recognized as keywords and thus not capitalized yet.

As a result, I have gone through https://docs.microsoft.com/en-us/sql/t-sql/functions/functions?view=sql-server-ver16 and created an initial list of t-sql functions:

ASCII
CHAR
CHARINDEX
CONCAT
CONCAT_WS
DIFFERENCE
FORMAT
LEFT
LEN
LOWER
LTRIM
NCHAR
PATINDEX
QUOTENAME
REPLACE
REPLICATE
REVERSE
RIGHT
RTRIM
SOUNDEX
SPACE
STR
STRING_AGG
STRING_ESCAPE
STRING_SPLIT
STUFF
SUBSTRING
TRANSLATE
TRIM
UNICODE
UPPER

CAST 
CONVERT
PARSE
TRY_CAST
TRY_CONVERT
TRY_PARSE
CURSOR_STATUS

SYSDATETIME 
SYSDATETIMEOFFSET
SYSUTCDATETIME
GETDATE
GETUTCDATE
DATE_BUCKET
DATENAME 
DATEPART 
DAY 
MONTH 
YEAR 
	
DATEFROMPARTS 
DATETIME2FROMPARTS 
DATETIMEFROMPARTS 
DATETIMEOFFSETFROMPARTS 
SMALLDATETIMEFROMPARTS 
TIMEFROMPARTS 
DATEDIFF 
DATEDIFF_BIG 
DATEADD 
EOMONTH 
SWITCHOFFSET 
TODATETIMEOFFSET 
ISDATE 
	
ISJSON
JSON_VALUE
JSON_QUERY
JSON_MODIFY
JSON_PATH_EXISTS

CHOOSE 
IIF 

ABS
ACOS
ASIN
ATAN
ATN2
CEILING
COS
COT
DEGREES
EXP
FLOOR
LOG
LOG10
PI
POWER
RADIANS  
RAND  
ROUND  
SIGN  
SIN  
SQRT  
SQUARE  
TAN

APP_NAME
APPLOCK_MODE
APPLOCK_TEST
ASSEMBLYPROPERTY
COL_LENGTH
COL_NAME
COLUMNPROPERTY
DATABASE_PRINCIPAL_ID
DATABASEPROPERTYEX
DB_ID
DB_NAME
FILE_ID
FILE_IDEX
FILE_NAME
FILEGROUP_ID
FILEGROUP_NAME
FILEGROUPPROPERTY
FILEPROPERTY
FULLTEXTCATALOGPROPERTY
FULLTEXTSERVICEPROPERTY
INDEX_COL  
INDEXKEY_PROPERTY  
INDEXPROPERTY  
NEXT VALUE FOR  
OBJECT_DEFINITION  
OBJECT_ID  
OBJECT_NAME  
OBJECT_SCHEMA_NAME  
OBJECTPROPERTY  
OBJECTPROPERTYEX  
ORIGINAL_DB_NAME  
PARSENAME  
SCHEMA_ID  
SCHEMA_NAME  
SCOPE_IDENTITY  
SERVERPROPERTY  
STATS_DATE  
TYPE_ID  
TYPE_NAME  
TYPEPROPERTY  
VERSION  

CERTENCODED
PWDCOMPARE
CERTPRIVATEKEY
PWDENCRYPT
CURRENT_USER
SCHEMA_ID
DATABASE_PRINCIPAL_ID
SCHEMA_NAME
SESSION_USER
SUSER_ID
SUSER_SID
HAS_PERMS_BY_NAME
SUSER_SNAME
IS_MEMBER
SYSTEM_USER
IS_ROLEMEMBER
SUSER_NAME
IS_SRVROLEMEMBER
USER_ID
LOGINPROPERTY
USER_NAME
ORIGINAL_LOGIN
PERMISSIONS

TEXTPTR

CUME_DIST
CURRENT_TIMESTAMP
DENSE_RANK
FIRST_VALUE
FORMAT
GETDATE
GETUTCDATE
GET_TRANSMISSION_STATUS
LAG
LAST_VALUE
LEAD
MIN_ACTIVE_ROWVERSION
NEWID
NEWSEQUENTIALID
NEXT VALUE FOR
NTILE
PARSENAME
PERCENTILE_CONT
PERCENTILE_DISC
PERCENT_RANK
RAND
RANK
ROW_NUMBER
SUM

Once an initial set of the above is integrated, i can certainly validate it once more and create a PR for all missing items.

Use case

easier to read code.

Dialect

tsql

Are you willing to work on and submit a PR to address the issue?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

enhancement t-sql

opened by fmms 27

Use dbt templater outside of dbt project project root with project_dir config
This PR is my attempt to get round the linked issue - that sqlfluff with dbt templater must be run from the dbt project directory. This is makes it difficult to include sqlfluff in pre-commit hooks when the root of the git repo is not the same as the dbt project root.

This PR is still quite a rough draft but as this is my first time contributing I was keen to get some early feedback.

Current situation

You can specify the dbt project directory in the .sqlfluff config file:

[sqlfluff:templater:dbt] project_dir = <absolute path to dbt project directory>

But when running sqlfluff from outside this directory an error is thrown by the dbt templater module File %s was not found in dbt project.

The dbt templater uses the PathSelectorMethod from dbt source code to extract nodes from the dbt manifest corresponding to the current file being linted.

I believe the root cause of this present issue is that the PathSelectorMethod sets the root of the search to the current working directory.

root = Path.cwd()

So my proposed solution is to create a new subclass which allows the root to be set to the dbt_project directory instead.

Not sure how good an architectural decision this is creating a dbt native class here in SQLFluff - open to alternative suggestions.
opened by pvonglehn 27
371 Optionally use DBT compiler as the TemplaterInterface
Background

We currently process templating for DBT projects the same way we template "plain" Jinja templated files (through Jinja templater). However DBT projects are special because the DBT library itself extends the context of the Jinja rendering, and DBT uses a package system through which the scope of the templating can also be extended.

This leads to a couple of problems loading macros (check out the "Related Issues"), and leads us to extend the logic in the Jinja templater.

Proposal

I would like to add a new Templater specific to DBT.

If the users specify that the templater used should be dbt in their configs, they would need to have the DBT package installed (extra dependency). From there we can be sure that the templated SQL is the same, as we re-use the DBT compiler to compile the specific node.

Related issues

#493 Problem with macro order using Jijnja Templater in DBT projects

#381 Allow loads macros to throw exceptions in DBT projects

#371 Compatibility with DBT utils package

#335 Loading macros from files fail when a macro contains another macro

#333 Support DBT var()

#156 Loads macros from path

#516 config.get not supported

Notes

Currently this is a POC, the logic needs to be refactored, but I wanted to show a working solution (this actually lints properly DBT models that couldn't be linted with the Jinja templater)
dbt
opened by dmateusp 27
Release timeline for 0.4.0.
I'm putting in place some basic project process at the moment. As part of that I think we should have a place for the discussion around which features will and won't go into the 0.4.0 release.

The current project board is here: https://github.com/sqlfluff/sqlfluff/projects/2

As of 6th October, the shortlist looks pretty well populated. I think there are lots of excellent suggestions in there. Before bringing them all in I want to make sure that we have a clear path forward on each one.

The ones that I think more clarity before we can bring them in are:

The concept of a public API.

https://github.com/sqlfluff/sqlfluff/issues/289

This one I'm happy to take hold of and get some clarity on, and I'm already working through some ideas as part of sqluff online: https://github.com/sqlfluff/sqlfluff-online/issues/14

The collection of issues related to macros and dbt libraries:

https://github.com/sqlfluff/sqlfluff/issues/156

https://github.com/sqlfluff/sqlfluff/issues/371

https://github.com/sqlfluff/sqlfluff/issues/156

https://github.com/sqlfluff/sqlfluff/issues/381

I'm going to suggest a date of 24th October for closing the 0.4.0 iteration. By which time we either have to get clear on the second chunk above or we delay it to the 0.5.0 iteration. At that point I'll create the relevant issues and projects for 0.5.0.
release
opened by alanmcruickshank 27
`sqlfluff fix` corrupts Jinja for loop
Expected Behaviour

Output should be semantically equivalent to input, and have valid syntax.

Observed Behaviour

Output seems to be corrupted.

Steps to Reproduce

$ printf 'SELECT\n 1,\n{%% for _ in [1, 2, 3] %%} 2,{%%endfor %%}\n' | sqlfluff fix - Unfixable violations detected. SELECT 1, {% for _ in [1, 2, 3] %} {%endfor %}

Note that the contents of the for loop was completely deleted.

Slightly varying the input causes the output to be corrupted in other ways (e.g. 2 is present but the comma is not, or the 2 is duplicated multiple times).

Dialect

None specified

Version

SQLFluff ce4e5a344526f7ee61a0950adc079e4d3b5af438, Python 3.9.7

Configuration

None
bug jinja fix corrupts sql
opened by CyberShadow 25
Warnings are not output in GitHub annotations
Search before asking

[X] I searched the issues and found no similar issues.

What Happened

Running sqlfluff lint with the --format github-annotation option removes warnings from the output.

Expected Behaviour

I would expect the results to be annotated variously as warnings and/or errors, in line with the standard human-readable output.

Observed Behaviour

Interestingly, the --annotation-level option seems to change the output level for all errors (e.g. --annotation-level warning sets actual errors to warnings) and still silently drops all actual warnings.

How to reproduce

sqlfluff lint --format github-annotation file-with-errors-and-warnings.sql

.sqlfluff with some rules downgraded to warnings e.g.

[sqlfluff] dialect = postgres exclude_rules = L009, L030, L031, L032, L051, L055 warnings = L001, L005, L006, L008, L010, L016, L023, L050

Dialect

postres

Version

sqlfluff 1.4.5 python 3.8

Configuration

[sqlfluff] dialect = postgres exclude_rules = L009, L030, L031, L032, L051, L055 warnings = L001, L005, L006, L008, L010, L016, L023, L050 [sqlfluff:rules] tab_space_size = 4 indent_unit = space max_line_length = 100 aliasing = explicit

Are you willing to work on and submit a PR to address the issue?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

bug
opened by Nate-Wessel 0
2.0.0a1 Auto-migration of deprecated config value `rules:max_line_length` has no effect
Search before asking

[X] I searched the issues and found no similar issues.

What Happened

Running SQLFluff 2.0.0a1 with an existing config file that has rules:max_line_length yields the following warning

deprecated config value `rules:max_line_length`

which suggests that the (new) value will get automatically applied.

However, a run of sqlfluff lint throws a lot of L016 warnings with default value (80)

L: 228 | P: 15 | L016 | Line is too long (106 > 80). L: 229 | P: 13 | L016 | Line is too long (106 > 80). L: 335 | P: 9 | L016 | Line is too long (227 > 80).

Expected Behaviour

see above

Observed Behaviour

see above

How to reproduce

see above

Dialect

all

Version

2.0.0a1

Configuration

[sqlfluff:rules] max_line_length = 800

Are you willing to work on and submit a PR to address the issue?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

bug
opened by dmohns 0
Fix for #4222

This resolves #4222 and resolves #4226. While there isn't a test case for the latter - fixing the first one makes the bigquery/004-templating test a test of the latter.

Cases of untaken indents were being handled improperly when the dedent point covered multiple dedents. This now makes sure that all dedents are untaken or otherwise forces a newline.

opened by alanmcruickshank 2
2.0.0a1 Auto fix produces broken SQL with non-existent Jinja parameter
Search before asking

[X] I searched the issues and found no similar issues.

What Happened

This is a bit of an odd one, but I think we still have to treat this as a bug.

Consider the follow SQL

CREATE OR REPLACE TABLE `{{ params.non_existent }}.{{ params.table_name }}` AS ( SELECT * FROM table2 ) ;

and assume

params = { "table_name": "mytable" }

When running sqlfluff fix to fix the lint error in line 4 SQLFluff removes the first backtick ``` from the table.

CREATE OR REPLACE TABLE {{ params.non_existent }}.{{ params.table_name }}` AS ( SELECT * FROM table2 ) ;

which is broken.

Expected Behaviour

see above

Observed Behaviour

see above

How to reproduce

see above

Dialect

BigQuery

Version

2.0.0a1

Configuration

bigquery

Are you willing to work on and submit a PR to address the issue?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

bug
opened by dmohns 0
2.0.0a1 Multiline `SELECT` are not correctly indented in CTA
Search before asking

[X] I searched the issues and found no similar issues.

What Happened

Consider the following two examples:

CREATE TABLE mytable AS (SELECT id, user_id FROM another_table ) ;

This one ☝️ does not throw any lint errors which is wrong I think. The SELECT should be on a new, indented line.

Compare this to below example that uses only a single SELECT target.

CREATE TABLE mytable AS (SELECT id FROM table ) ;

which gets linted to

CREATE TABLE mytable AS ( SELECT id FROM table ) ;

Expected Behaviour

see above

Observed Behaviour

see above

How to reproduce

see above

Dialect

ansi

Version

2.0.0a1

Configuration

default

Are you willing to work on and submit a PR to address the issue?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

bug
opened by dmohns 0

Releases(2.0.0a1)

2.0.0a1(Dec 28, 2022)
NOTE: This is an alpha release for testing purposes. There are several new features here, and breaking changes to configuration. We welcome testing feedback from the community, but know that this release may feel less polished than usual.

Highlights

This is the first alpha version for 2.0.0. It brings all of the changes to whitespace handing, including a total rewrite of indentation and long line logic (L003 & L016). That brings several breaking changes to the configuration of layout, see the layout docs for more details and familiarise yourself with the new default configuration.

In addition, for the dbt templater, this introduces a large re-write of the codebase, dropping support for dbt versions before 1.0.0. This leverages functionality from dbt-osmosis to reduce the amount of functionality supported directly by SQLFluff, and performance during testing of the new version has been reported as significantly faster.

There will likely be more changes to rule classification before a full release of 2.0.0, so anticipate that configuration files may change slightly again in future alpha releases.

What’s Changed

Fixed False Positive for L037 (#4198) @WillAyd

Fix #4215 (#4217) @alanmcruickshank

don't consider templated whitespace (#4213) @alanmcruickshank

show fatal errors regardless (#4214) @alanmcruickshank

don't pickle the templater (#4208) @alanmcruickshank

MySQL: Support column character set and collation (#4204) @yoichi

Fix some issues with Docker Compose environment (#4201) @barrywhart

Implicit Indents (#4054) @alanmcruickshank

Tweak Coveralls settings (#4199) @barrywhart

In addition to Codecov, also upload to Coveralls (#4197) @barrywhart

Fix: create table default cast returns unparsable section (#4192) @NelsonTorres

Fix JSON parsing issue with diff-quality plugin (#4190) @barrywhart

Codecov migration (#4195) @alanmcruickshank

Stop adding trailing os.sep if ignore file is on the root of the file… (#4182) @baa-ableton

Port dbt-osmosis templater changes to SQLFluff (#3976) @barrywhart

Reflow 4: Long Lines (#4067) @alanmcruickshank

Fix comment bug on reindent (#4179) @alanmcruickshank

Reflow 3: Reindent (#3942) @alanmcruickshank

New Contributors

@baa-ableton made their first contribution in #4182

@WillAyd made their first contribution in #4198

Source code(tar.gz)
Source code(zip)
1.4.5(Dec 19, 2022)
Highlights

This is a bugfix release, primarily for diff-quality. In addition, a new rules for spacing around parenthesis is also included.

This is also the final 1.x.x release. Following releases will be a series of alpha releases for 2.x.x. If you affected by any outstanding bugs or regressions from this release, consider either rolling backward to a previous release without those issues, or failing forward to on an alpha release for 2.x.x (or a full release if that's out). Note that 2.x.x will bring a selection of breaking changes to config file structure, rule categorisation and dbt support.

What’s Changed

Add rule for space around parenthesis #4131 @NelsonTorres

diff-quality plugin: Print invalid JSON on parse failure #4176 @barrywhart

Ensure diff-quality runs the correct SQLFluff #4175 @barrywhart

New Contributors

@NelsonTorres made their first contribution in #4131

Source code(tar.gz)
Source code(zip)
1.4.4(Dec 14, 2022)
Highlights

Bug fix for 1.4.3 which was incorrectly flagging L006 for concat operators (||) and other two-symbol binary operators.

What’s Changed

Prep version 1.4.4 (#4170) @github-actions

Recognise || as an operator to avoid rule L006 flagging it (#4168) @tunetheweb

:bug: Check verbosity level of pytest run before running certain tests (#4167) @pwildenhain

[snowflake] Add support for snowflake select * exclude/replace syntax (#4160) @moreaupascal56

Source code(tar.gz)
Source code(zip)
1.4.3(Dec 14, 2022)
Highlights

Rewrote diff-quality plugin to run SQLFluff as a subprocess. More reliable, easier to switch between diff-quality and running sqlfluff lint directly.

New rule L067 enforces consistent syntax for type casts.

New rule L068 enforces a consistent number of columns in set queries (e.g. UNION).

Initial support for Materialize dialect.

What's Changed

Add flyway variables support via placeholder templater #4026 @srjonemed

Fix Spark comparison parsing #4066 @ms32035

Add errors and fails to pytest summary #4076 @alanmcruickshank

Storage reference segment #4057 @YilangHe

New rule L069: Consistent syntax for sql type casting #3747 @bolajiwahab

Postgres: Views and named notations #4073 @davetapley

Switch reflow buffer from LintFix to LintResult #4083 @alanmcruickshank

Support parallel linting when many individual files specified #4084 @barrywhart

Rule: check number of columns in set operations match #4028 @erevear

Fix syntax in indentation sample code docs #4087 @PBalsdon

Remove "mystery" binary file added in PR #2923 #4088 @barrywhart

Fix mypy issue with regex #4097 @barrywhart

Enable variable reference names to have leading underscore for snowflake dialect #4098 @gavin-tsang

Fix flake8 issue with segment_predicates.py #4101 @barrywhart

Fix bug in example rule plugin #4103 @barrywhart

Fix bug where L034 should ignore INSERT or "CREATE TABLE AS SELECT" with CTE #4108 @barrywhart

Postgres: Alter type rename value #4100 @greg-finley

Bug fix: dbt templater ignores .sqlfluff file encoding on Windows #4109 @barrywhart

Add initial Materialize dialect #4112 @bobbyiliev

L015: Handle COUNT(DISTINCT(col)) #4110 @barrywhart

[Snowflake] format type options extensions for copy_into_location #4129 @YilangHe

Fix tox arguments #4144 @greg-finley

[DB2] Fix parsing of string identifiers #4134 @borchero

BigQuery: Allow double quoted literal in export_option_list #4126 @yoichi

Only upload 3 sets of test results to codecov (possible workaround for hanging builds) #4147 @barrywhart

SparkSQL: ILIKE #4138 @greg-finley

SparkSQL: Mark AS as optional keyword for CTE & CTS #4127 @ulixius9

Fix passenv to work with tox 4 #4154 @tunetheweb

Allow deprecated --disable_progress_bar flag for fix command #4151 @pdebelak

Implement diff_quality_plugin using command-line rather than Python #4148 @barrywhart

L037: insert ASC just after column_reference #4149 @yoichi

New Contributors

@srjonemed made their first contribution in #4026

@ms32035 made their first contribution in #4066

@davetapley made their first contribution in #4073

@PBalsdon made their first contribution in #4087

@gavin-tsang made their first contribution in #4098

@bobbyiliev made their first contribution in #4112

@ulixius9 made their first contribution in #4127

Source code(tar.gz)
Source code(zip)
1.4.2(Nov 13, 2022)
Highlights

This release is less about internals and much more about some quality of life improvements and dialect changes. The most notable are:

The introduction of a sqlfluff render command to preview the results of templated sql.

Linting errors within templated loops should now only appear once in the linting output.

Indentation around jinja {% set %} statements should now be more consistent.

Linting errors around unparsable code are now more appropriately handled (with more to come soon on that front).

Error messages when specified files aren't found are now more specific.

We've also got dialect improvements for Redshift, SOQL & SparkSQL.

What’s Changed

Fix type error in get_rules hook of plugin example (#4060) @Samyak2

L003: Add missing "pragma: no cover" (#4058) @barrywhart

Fix bug in sparksql SELECT statement termination at UNION #4050 (#4052) @anna-azizian

Deduplicate violations in the source space (#4041) @alanmcruickshank

Use "docker compose", not "docker-compose" (#4055) @barrywhart

Allow warnings for specific rules (#4053) @alanmcruickshank

Better file not found error #1023 (#4051) @alanmcruickshank

Filter out issues in unparsable sections (#4032) @alanmcruickshank

Snowflake: ADD and DROP without COLUMN (#4049) @greg-finley

Make render command (#4043) @alanmcruickshank

Bump after_n_builds to 10 (#4046) @greg-finley

Redshift: allows for parenthesis around FROM content (#3962) @adam-tokarski

Update CI to use Python 3.11 (#4038) @greg-finley

Classify self contained set statements as templated (#4034) @alanmcruickshank

Date and Datetime literals in SOQL (#4037) @alanmcruickshank

mypy edits for 0.990 (#4035) @alanmcruickshank

sparksql: support for create/remove widget clause (#4021) @Coola4kov

Redshift CREATE EXTERNAL FUNCTION statement (#4011) @rpr-ableton

Update Redshift bare functions (#4012) @rpr-ableton

Source code(tar.gz)
Source code(zip)
1.4.1(Oct 31, 2022)
Highlights

This is a fix to the configuration migration from 1.4.0. In that release, the configuration of leading/trailing operators would be migrated the wrong way around and precedence between new and old configuration values behaved unexpectedly.

What’s Changed

Prep version 1.4.1 (#4006) @github-actions

Config precedence (#4007) @alanmcruickshank

Redshift CREATE/ATTACH/DETACH/DROP RLS POLICY statements (#4004) @rpr-ableton

Redshift: Add support for APPROXIMATE functions (#3997) @rpr-ableton

hotfix to config migration (#4005) @alanmcruickshank

Source code(tar.gz)
Source code(zip)
1.4.0(Oct 31, 2022)
This release brings several internal changes, and acts as a prelude to 2.0.0 which will be released fairly soon. In particular, the following config values have changed:

sqlfluff:rules:L007:operator_new_lines has been changed to sqlfluff:layout:type:binary_operator:line_position.

sqlfluff:rules:comma_style and sqlfluff:rules:L019:comma_style have both been consolidated into sqlfluff:layout:type:comma:line_position.

If any of these values have been set in your config, they will be automatically translated to the new values at runtime, and a warning will be shown. To silence the warning, update your config file to the new values. For more details on configuring layout (including some changes yet to come in future versions) see the docs.

These changes are driven by underlying centralisation in the routines which control layout. While for this release, no breaking changes are expected - you may find slight differences in how SQLFluff handles edge cases. We believe in the majority of cases these are more consistent, but if you find any which are problematic then do post an issue on GitHub as usual.

Other highlights from this release:

Better dbt supportfor graph nodes and avoiding dependency conflicts.

Numerous dialect improvements to T-SQL, MySQL, SparkSQL, SQLite, Athena Snowflake, Hive, Postgres & Databricks.

There have also been first time contributions from 10 new contributors! 🎉🎉🎉

What’s Changed

Prep version 1.4.0 (#3983) @github-actions

Snowflake partition nonreserved keyword (#3972) @YilangHe

Hive: Add support for EXCHANGE PARTITION in ALTER TABLE (#3991) @nahuelverdugo

Resolve parse error on multiple bracketed statements (#3994) @yoichi

Enable parsing of CLONE keyword in bigquery dialect (#3984) @realLyans

BigQuery: allow nesting of SetExpressionSegment (#3990) @yoichi

feat(clickhouse): Support non-standard CREATE TABLE statement (#3986) @tomasfarias

Fix Windows CI check (#3992) @greg-finley

Snowflake tag reference segment (#3985) @WittierDinosaur

Fix Jinja templater issue where undefined callable threw an exception (#3982) @barrywhart

Reflow Documentation V1 (#3970) @alanmcruickshank

Allow lambda argument columns to be unqualified (#3971) @olagjo

document inline configuration (#3981) @alanmcruickshank

[BUGFIX] Changing cwd temporarily on manifest load as dbt is not using project_dir to read/write target folder (#3979) @barrywhart

Fix type annotation of user_rules in Linter (#3977) @Samyak2

Unpin markupsafe (#3967) @judahrand

Snowflake frame clause variables (#3969) @WittierDinosaur

SparkSQL: added support for : (colon sign) operator (Databricks SQL) (#3956) @karabulute

Athena: Add support for using underscore aliases (#3965) @hectcastro

Snowflake: ALTER TABLE constraint actions (#3959) @erevear

MySQL: Support REPLACE statement (#3964) @yoichi

TSQL: Add support for UPDATE STATISTICS option FULLSCAN (#3950) @hectcastro

ANSI: fixed typos in docstrings and comments (#3953) @karabulute

Postgres: ALTER PROCEDURE (#3949) @krokofant

T-SQL: Allow arbitrary expressions in PARTITION BY clause (#3939) @borchero

Enable dumping of performance information to csv. (#3937) @alanmcruickshank

Consolidate comma style configs (#3945) @alanmcruickshank

Adding missing KeywordSegments for different file types in Athena dialect (#3898) @CommonCrisis

Add templated block uuids (#3936) @alanmcruickshank

Load the full dbt manifest (#3926) @davajm

MySQL: Support optional "IF NOT EXISTS" with CREATE TRIGGER (#3943) @yoichi

T-SQL: Allow to parse SYNONYM statements (#3941) @borchero

Hive: Add support for LATERAL VIEW clause (#3935) @hectcastro

Fix crash in L042 on "UNION" or other "set" queries (#3931) @barrywhart

Refactor Lexer: Split apart elements_to_segments and refine placement of meta segments. (#3925) @alanmcruickshank

Add INSERT INTO <> DEFAULT VALUES to ANSI SQL and T-SQL (#3934) @borchero

Break apart reflow classes and extract methods (#3919) @alanmcruickshank

T-SQL: Fix indendentation of OUTER APPLY (#3932) @borchero

MySQL: Fix create trigger (#3928) @adam-tokarski

SparkSQL: Fixed bug with QUALIFY usage without WHERE clause (applies also for Databricks dialect) (#3930) @karabulute

T-SQL: Allow specifying join hints (#3921) @borchero

SQLite: Add support for table-level CHECK constraint (#3923) @hectcastro

sparksql: added * EXCEPT for SELECT clause (#3922) @adam-tokarski

Map old configs to new configs (#3915) @alanmcruickshank

[issue_3794] allow to use 'usage' as identifier for postgres (#3914) @adam-tokarski

DATABRICKS: Add Support for Delta Live Tables (DLT) Syntax (#3899) @R7L208

Postgres Revoke fix (#3912) @greg-finley

fix: Click output to stderr on errors (#3902) @KingMichaelPark

fix issue with empty enum for postgres (#3910) @adam-tokarski

feat: Optional numerics for postgres arrays (#3903) @KingMichaelPark

fix(test): Return ParseExample namedtuple in get_parse_fixtures (#3911) @tomasfarias

Fix typos (#3901) @kianmeng

provide custom DeprecatedOption (#3904) @adam-tokarski

fix(redshift): Allow keywords in qualified references (#3905) @tomasfarias

Reflow centralisation 2: Rebreak (operators & commas) (#3847) @alanmcruickshank

Bring L008 into reflow work (#3908) @alanmcruickshank

Snowflake: Create network policy ip lists (#3888) @greg-finley

MySQL: Implement (key_part, ...) in index definitions (#3887) @yoichi

Reflow centralisation 1: Scope + Respace (#3824) @alanmcruickshank

Update github badge and add docker badge (#3884) @alanmcruickshank

Source code(tar.gz)
Source code(zip)
1.3.2(Sep 27, 2022)
Highlights

This is primarily a release for dialect fixes and improvements with additions and changes to TSQL, Snowflake, MySQL & Redshift. The other changes of note are:

Support for warnings when users set old removed config values. This supports future change work by allowing a mechanism to warn if they are used.

Improvements to the fix routines for L014 and L042 to handle some trickier cases.

What’s Changed

Add CreateUserSegment for Snowflake dialect (#3880) @Gal40n04ek

raw_segments_with_ancestors (#3878) @alanmcruickshank

Adjust TSQL Operators (#3877) @alanmcruickshank

Refactor path_to (#3875) @alanmcruickshank

Support Removed warning on configs (#3874) @alanmcruickshank

MySQL: Support column-path operator for JSON type (#3864) @yoichi

T-SQL: ALTER FUNCTION/PROCEDURE/VIEW parsing (#3867) @differgroup

MySQL: Support hexadecimal literals and bit value literals (#3869) @yoichi

MySQL: Treat double quotes the same as single quotes (#3871) @yoichi

Snowflake: COMMIT WORK (#3862) @rglbr

Snowflake: AlterShareStatementSegment and CreateDatabaseFromShareStatementSegment (#3858) @moreaupascal56

MySQL: Add CREATE/ALTER VIEW (#3859) @wfelipew

Redshift: CREATE TABLE LIKE enhancements (#3853) @greg-finley

L014 leading underscore capitalization inference fix (#3841) @j-svensmark

MySQL: Add extended DROP TRIGGER functionality (#3846) @yoichi

Allow standalone aliases in L027 (#3848) @olagjo

L042: Enable autofix for some tricky cases (#3700) @barrywhart

Snowflake: CREATE FUNCTION IF NOT EXISTS (#3845) @greg-finley

ignore functions with more than one element ... (#3792) @fmms

MySQL: support remaining constraint parts of CREATE/ALTER TABLE (#3827) @yoichi

New Contributors

@olagjo made their first contribution in #3848

@j-svensmark made their first contribution in #3841

@wfelipew made their first contribution in #3859

@moreaupascal56 made their first contribution in #3858

@rglbr made their first contribution in #3862

@differgroup made their first contribution in #3867

Source code(tar.gz)
Source code(zip)
1.3.1(Sep 9, 2022)
[1.3.1] - 2022-09-09

Highlights

More refactoring of parse structures in preparation for upcoming refactor of formatting/whitespace rules.

Fixes some bugs in L003 (indentation).

New config flag large_file_skip_byte_limit which applies prior to loading the file.

What’s Changed

Snowflake: Fix syntax errors in tests #3834 @JamesRTaylor

Add support for additional magic methods on DummyUndefined #3835 @barrywhart

MySQL: support variable assignments by assignment operator := #3829 @yoichi

MYSQL: improve lexing for single-quoted strings #3831 @mdahlman

MySQL: More support for index definition in CREATE TABLE #3826 @yoichi

Typed matching and ripping out the rest of .name #3819 @alanmcruickshank

sparksql dialect to support lambda expressions (->) #3821 @juhoautio

Fixed broken main branch #3825 @alanmcruickshank

Enable file name logging for multi-files w/ --show-lint-violations flag #3788 @thechopkins

Take database and schema out of Snowflake reserved keywords list #3818 @NiallRees

Remove a chunk of name references #3814 @alanmcruickshank

Fix typo in Snowflake dialect #3813 @Gal40n04ek

Allow the use of libraries in macro definitions #3803 @bjgbeelen

Indentation fixes and rule logging improvements #3808 @alanmcruickshank

Fixes a recursion error in JinjaTemplater handling of undefined values #3809 @barrywhart

Snowflake: extend GRANT syntax #3807 @Gal40n04ek

add warehouse_type in snowflake dialect #3805 @Gal40n04ek

add Create Notification Integration syntax #3801 @Gal40n04ek

T-SQL: fix parsing PARTITION BY NULL in window function #3790 @fmms

SparkSQL: Update L014 rule to not flag Delta Change Data Feed Session & Table Property #3689 @R7L208

Snowflake: OVER (ORDER BY) clause required for first_value (fixes #3797) #3798 @JamesRTaylor

add Alter Pipe syntax for snowflake dialect #3796 @Gal40n04ek

BigQuery: Parse WEEK() in date_part #3787 @yoichi

Postgres: Support setting user properties using intrinsic ON & OFF values #3793 @chris-codaio

extend SF dialect for File Format statements #3774 @Gal40n04ek

Add QUALIFY to SparkSQL dialect #3778 @ThijsKoot

fix regex for S3Path #3782 @Gal40n04ek

Snowflake: add Optional parameter ERROR INTEGRATION for PIPE #3785 @Gal40n04ek

Add a file size check in bytes #3770 @alanmcruickshank

Require importlib_metadata >=1.0.0 #3769 @alanmcruickshank

New Contributors

@Gal40n04ek made their first contribution in #3785

@ThijsKoot made their first contribution in #3778

@chris-codaio made their first contribution in #3793

@JamesRTaylor made their first contribution in #3798

@fmms made their first contribution in #3790

@bjgbeelen made their first contribution in #3803

@thechopkins made their first contribution in #3788

Source code(tar.gz)
Source code(zip)
1.3.0(Aug 21, 2022)
Highlights

This release brings several potentially breaking changes to the underlying parse tree. For users of the cli tool in a linting context you should notice no change. If however your application relies on the structure of the SQLFluff parse tree or the naming of certain elements within the yaml format, then this may not be a drop-in replacement. Specifically:

The addition of a new end_of_file meta segment at the end of the parse structure.

The addition of a template_loop meta segment to signify a jump backward in the source file within a loop structure (e.g. a jinja for loop).

Much more specific types on some raw segments, in particular identifier and literal type segments will now appear in the parse tree with their more specific type (which used to be called name) e.g. naked_identifier, quoted_identifier, numeric_literal etc...

If using the python api, the parent type (such as identifier) will still register if you call .is_type("identifier"), as this function checks all inherited types. However the eventual type returned by .get_type() will now be (in most cases) what used to be accessible at .name. The name attribute will be deprecated in a future release.

Other highlights:

New command-line option --show-lint-violations to show details on unfixable errors when running sqlfluff fix.

Improved consistency of process exit codes.

Short CLI options for many common options.

Jinja templater: When --ignore=templating is enabled, undefined Jinja variables now take on "reasonable" default values rather than blank string (""). This can streamline initial rollout of SQLFluff by reducing or eliminating the need to configure templater variables.

There are also a ton of other features and bug fixes in this release, including first-time contributions from 11 new contributors! 🎉

What’s Changed

T-SQL: ALTER TABLE DROP COLUMN #3749 @greg-finley

Add "# pragma: no cover" to work around sporadic, spurious coverage failure #3767 @barrywhart

Add end_of_file and template_loop markers #3766 @alanmcruickshank

Provide usage examples for new users #3765 @sirlark

SQLite: deferrable in create table statement #3757 @RossOkuno

When ignore=templating and fix_even_unparsable=True, provide defaults for missing vars #3753 @barrywhart

BigQuery: Support Materialized Views #3759 @yoichi

Enhance L062 to ignore blocked words in comments #3754 @tunetheweb

Fix bug where undefined Jinja variable in macro file crashes linter #3751 @barrywhart

Migrate analysis, functional and testing to utils #3743 @alanmcruickshank

Build out rule crawling mechanisms #3717 @alanmcruickshank

Add current_timestamp to Redshift as a bare function #3741 @RossOkuno

BigQuery: Fix parsing parameterized data types #3735 @yoichi

Change MySQL Create Statement Equals Segment to Optional #3730 @keyem4251

SQLite: add parsing of INSERT statement #3734 @imrehg

SPARKSQL: Support Delta Lake Drop Column Clause in ALTER TABLE #3727 @R7L208

Add short versions of several cli options #3732 @alanmcruickshank

Build out type hints in Grammars #3718 @alanmcruickshank

dbt 1.3.0 compatibility #3708 @edgarrmondragon

Revise no cover direction and remove unused code. #3723 @alanmcruickshank

Update broken flattr link #3720 @alanmcruickshank

BigQuery: remove key from unreserved keywords list #3719 @sabrikaragonen

Bigquery reset primary and foreign keys #3714 @sabrikaragonen

Name Deprecation (Part 1) #3701 @alanmcruickshank

Teradata: Add two TdTableConstraints #3690 @greg-finley

Redshift: support expressions in array accessors #3706 @chronitis

Handle logging issues at teardown #3703 @alanmcruickshank

L028, L032: Fix bug where fixes were copying templated table names #3699 @barrywhart

L042: Autofix sometimes results in "fix looping", hitting the linter "loop limit" #3697 @barrywhart

L042: Address corner cases where fix corrupts the SQL #3694 @barrywhart

T-SQL: Properly parse collation names #3686 @borchero

Allow escaping single quotes in single-quoted literal with '' #3682 @pdebelak

T-SQL: Fix indentation after JOIN/APPLY clauses with no ON statement #3684 @borchero

T-SQL: Parse DATEPART date type as date type instead of column name #3681 @borchero

T-SQL: Allow COLLATE clause in JOIN conditions #3680 @borchero

T-SQL: Fix parsing of CREATE VIEW statements with column name syntax #3669 @borchero

Fix typo in github issue template #3674 @alanmcruickshank

Add Athena issue label #3676 @greg-finley

Set issue dialect labels via Github Actions #3666 @greg-finley

Allow configuration of processes from config #3662 @alanmcruickshank

Reposition before recursion in fixes to avoid internal error #3658 @alanmcruickshank

Use UUIDs for matching #3661 @alanmcruickshank

Postgres: Add dialect-specific bare functions #3660 @WittierDinosaur

Postgres: Add CALL Support #3659 @WittierDinosaur

ANSI - Add support for INTERSECT ALL and EXCEPT ALL #3657 @WittierDinosaur

Option to show errors on fix #3610 @chaimt

L042: Fix internal error "Attempted to make a parent marker from multiple files" #3655 @barrywhart

L026: Add support for merge_statement #3654 @barrywhart

Add handling for Redshift CONVERT function data type argument #3653 @pdebelak

Deduplicate files before and during templating #3629 @alanmcruickshank

Rationalise Rule Imports #3631 @alanmcruickshank

Handle Jinja {% call ... %} blocks #3648 @barrywhart

SPARKSQL: Add Delta Lake Constraints syntax to ALTER TABLE #3643 @R7L208

Redshift: syntax for array unnesting with index #3646 @chronitis

Snowflake - ALTER TABLE IF EXISTS and WHEN SYSTEM$STREAM_HAS_DATA() #3641 @chrisalexeev

L057: In BigQuery, allow hyphens by default #3645 @barrywhart

Better messages for partial indentation in L003 #3634 @pdebelak

Add INTEGER to PrimitiveTypeSegment for Sparksql #3624 @ciwassano

Bump version in gettingstarted.rst via the release script #3642 @greg-finley

Improve handling of BigQuery hyphenated table names #3638 @barrywhart

update sqlfluff version in gettingstareted.rst #3639 @keyem4251

L016: Ignore jinja comments if ignore_comment_clauses=True #3637 @barrywhart

Add errors for redundant definitions. #3626 @alanmcruickshank

Object Literals #3620 @alanmcruickshank

Dialect Crumbs #3625 @alanmcruickshank

Consistent return codes #3608 @alanmcruickshank

New Contributors

@keyem4251 made their first contribution in #3639

@ciwassano made their first contribution in #3624

@chronitis made their first contribution in #3646

@chaimt made their first contribution in #3610

@borchero made their first contribution in #3669

@sabrikaragonen made their first contribution in #3714

@edgarrmondragon made their first contribution in #3708

@imrehg made their first contribution in #3734

@yoichi made their first contribution in #3735

@RossOkuno made their first contribution in #3741

@sirlark made their first contribution in #3765

Source code(tar.gz)
Source code(zip)
1.2.1(Jul 15, 2022)
[1.2.1] - 2022-07-15

Highlights

This is primarily a bugfix release to resolve an issue with the 1.2.0 release where the new version of sqlfluff-templater-dbt relied on functionality from the new version of sqlfluff but the package configuration had not been updated. Versions of the two packages are now pinned together.

What’s Changed

Pin sqlfluff-templater-dbt via release script #3613 @greg-finley

Specifying comma delimited is unnecessary #3616 @alanmcruickshank

Handle redshift temporary tables with # at the beginning of name #3615 @pdebelak

Source code(tar.gz)
Source code(zip)
1.2.0(Jul 13, 2022)
Highlights

Major changes include:

Adding AWS Athena as a dialect.

A fix routine for L046 (whitespace in jinja tags), and the mechanisms for more source-only fixes in future.

By default, large files (over 20k characters) are now skipped by sqlfluff. This limit is configurable and disable-able but exists as a sensible default to avoid the performance overhead of linting very large files.

For the dbt templater, fatal compilation errors now lo longer stop linting, and these files are now skipped instead. This enables projects to continue linting beyond the offending file and much better logging information to enable better debugging.

What’s Changed

Add a skip and better logging for fatal dbt issues #3603 @alanmcruickshank

Add large file check #3600 @alanmcruickshank

Oracle: extend support for ALTER TABLE #3596 @davidfuhr

Immutability fixes #3428 @alanmcruickshank

Struct type should be a segment #3591 @alanmcruickshank

Fix Bracketed Struct issue #3590 @alanmcruickshank

Allow spaces and multiple signs for numeric literals #3581 @tunetheweb

Add source fixing capability and fix routines for L046 #3578 @alanmcruickshank

Small grammar cleanup in team rollout docs #3584 @theianrobertson

Postgres: CREATE COLLATION support #3571 @greg-finley

Redshift: Add TOP X to select clause modifiers #3582 @pdebelak

Postgres: Small fixes to COMMENT ON #3566 @greg-finley

Support MySQL system variables #3576 @qgallet

Allow no alias for selects in CTEs with a column list #3580 @pdebelak

New dialect AWS Athena #3551 @cmotta

Split apart fix_string(). #3568 @alanmcruickshank

Fix incorrect L022 with postgres dialect with CTE argument list #3570 @pdebelak

Simplify lint fixing (prep for source fixes) #3567 @alanmcruickshank

Exclude .coverage.py from linting #3564 @zidder

L016: ignore_comment_clauses not working for postgres dialect #3549 @barrywhart

Groundwork for a fix routine for L046 #3552 @alanmcruickshank

Add better handling for SQLFluffUserError when running core cli commands #3431 @alanmcruickshank

New Contributors

@pdebelak made their first contribution in #3570

@cmotta made their first contribution in #3551

@qgallet made their first contribution in #3576

@theianrobertson made their first contribution in #3584

@davidfuhr made their first contribution in #3596

Source code(tar.gz)
Source code(zip)
1.1.0(Jul 3, 2022)
Highlights

Major changes include:

L066 - New rule to allow you to set min/max length requirements for aliases to ensure they are meaningful

L062 - addition of blocked_regex as well as blocked_words

L025 - fix several corner cases where aliases were removed inappropriately

L059 is now disabled by default for Postgres

Many more dialect improvements and bug fixes.

What’s Changed

L025: Derived query requires alias -- also handle UNION, etc. #3548 @barrywhart

L025 should not remove aliases from derived queries #3546 @barrywhart

T-SQL keyword functions should be treated as keywords #3540 @tunetheweb

Fix issue where "--nocolor" is ignored [#3536)(https://github.com/sqlfluff/sqlfluff/pull/3536) @barrywhart

Clickhouse: allow FINAL modifier #3534 @ThomAub

L018 change to just check for newlines rather than alignment #3499 @zidder

SparkSQL: Update terminator grammar for HAVING, WHERE, GROUP BY #3526 @R7L208

Fix L025 false positive for T-SQL VALUES clause #3533 @barrywhart

New rule L066 for enforcing table alias lengths #3384 @f0rk

Redshift: CALL statement #3529 @greg-finley

Core: Compile regexes at init time to avoid overhead #3511 @judahrand

Disable L059 by default for Postgres #3528 @tunetheweb

Core: Add MultiStringParser to match a collection of strings #3510 @judahrand

SQLite: PRIMARY KEY AUTOINCREMENT #3527 @greg-finley

MySQL: Support LOAD DATA #3518 @greg-finley

Redshift: GRANT EXECUTE ON PROCEDURES #3516 @greg-finley

Allow DEFAULT expression in Redshift ALTER TABLE ADD COLUMN statements #3513 @menzenski

BigQuery: Fix parsing of Array creation from full subquery #3502 @judahrand

SparkSQL: Allow dateparts as table aliases #3500 @R7L208

Fix load_macros_from_path to actually support multiple paths #3488 @emancu

Allow linter to apply fixes spanning more than 2 slices #3492 @barrywhart

Fix L022 false positive when the CTE definition has a column list #3490 @barrywhart

SparkSQL: Support for Delta RESTORE statement #3486 @R7L208

Add values function to SET clause #3483 @hgranthorner

SparkSQL: Support for CONVERT TO DELTA command #3482 @R7L208

BigQuery: Remaining procedural statements #3473 @tunetheweb

Postgres: support grouping sets #3477 @tunetheweb

SparkSQL: Support for Delta syntax to create manifest files through the GENERATE command #3478 @R7L208

Add config for optionally indenting contents of ON blocks #3471 @PeterBalsdon

L026: check standalone aliases as well as table aliases #3470 @tunetheweb

L045: Add handling for nested queries and CTEs #3468 @barrywhart

L062: add blocked_regex support #3467 @tunetheweb

SparkSQL: Support for the Delta DESCRIBE DETAIL command #3465 @R7L208

New Contributors

@PeterBalsdon made their first contribution in #3471

@hgranthorner made their first contribution in #3483

@emancu made their first contribution in #3488

@judahrand made their first contribution in #3502

@f0rk made their first contribution in #3384

@zidder made their first contribution in #3499

@ThomAub made their first contribution in #3534

Source code(tar.gz)
Source code(zip)
1.0.0(Jun 17, 2022)
Highlights

This is the first stable release of SQLFluff 🎉🎉🎉.

Does this mean there are no more bugs? No.

Does this mean we're going to stop developing new features? No.

Does this mean that this is a tool that is now broadly usable for many teams? Yes.

We've intentionally chosen to release 1.0.0 at a time of relative stability within SQLFluff and not at a time when new big structural changes are being made. This means that there's a good chance that this release is broadly usable. This also recognises that through the hard work of a huge number of contributors that we've built out this from a fringe tool, to something which gets over 500k downloads a month and over 4k stars on Github.

There's still a lot to do, and some more exciting things on the horizon. If you want to be part of this and join the team of contributors, come and hang out in our slack community or on our twitter account where people can help you get started. If you're a long time user, keep submitting bug reports and inputting on issues on Github.

If you've never used SQLFluff before, or are hesitant about starting to use it in your day to day work, now might be a good time to try it. We have guides on how to get started with the tool, and how to get started with rolling out to a team in our docs.

What’s Changed

Prep version 1.0.0 (#3454) @github-actions

Swap to skip file if not found (#3464) @alanmcruickshank

Postgres: treat GENERATE_SERIES as a value table function (#3463) @tunetheweb

Postgres: Support multiple CONSTRAINTS in CREATE DOMAIN (#3460) @tunetheweb

Redshift: Add ANYELEMENT support (#3458) @tunetheweb

Postgres: Optional select clause elements and better ON CONFLICT support (#3452) @tunetheweb

Add ClickHouse as a dialect (#3448) @tunetheweb

Postgres: allow keywords in qualified column references (#3450) @tunetheweb

Remove Baron Schwatz Dead Link (#3453) @tunetheweb

Postgres: Finish ALTER TYPE (#3451) @greg-finley

SparkSQL: Add Delta Syntax for DESCRIBE HISTORY statement (#3447) @R7L208

Snowflake: Support Stage data file parameters in FROM clauses (#3446) @tunetheweb

Redshift: Support Object unpivoting (#3441) @tunetheweb

Python script to automate release (#3403) @greg-finley

Remove Delta Lake Reference in README.md (#3444) @R7L208

Add databricks dialect as an alias for sparksql dialect (#3440) @R7L208

Make all Postgres identifiers quotable (#3442) @tunetheweb

Update JinjaAnalyzer and JinjaTracer to handle {% block %} (#3436) @barrywhart

SparkSQL: Add support for Delta VACUUM statement (#3439) @R7L208

Warning for parsing errors extended to all dialects (#3411) @mdahlman

Handle templater validation errors more gracefully (#3433) @alanmcruickshank

MYSQL: allow for escaped single quotes (#3424) @mdahlman

L027: Fix false positives by reverting the PR for issue #2992: Check table aliases exist (#3435) @barrywhart

Allow numeric_dollar templater to have curly braces, update dollar + numeric_dollar templater examples in docs (#3432) @menzenski

Allow Redshift IDENTITY column (seed, step) to be optional (#3430) @menzenski

L036: Make wildcard behavior configurable (#3426) @barrywhart

L034: Don't autofix if numeric column references (#3423) @barrywhart

L036: Treat wildcard as multiple select targets (i.e. separate line) (#3422) @barrywhart

Snowflake: IDENTIFIER pseudo-function (#3409) @mdahlman

SNOWFLAKE: Fully referenced object names in clone statements (#3414) @mdahlman

Unpin coverage now issue with 6.3 has been resolved (#3393) @tunetheweb

L045: handle UPDATE statements with CTEs (#3397) @tunetheweb

L027: Add support for ignore_words (#3398) @dmohns

Postgres: Allow CREATE FUNCTION to use Expressions in default values (#3408) @tunetheweb

Fix bug in apply_fixes() with leading/trailing whitespace (#3407) @barrywhart

Redshift: Correct ALTER TABLE syntax (#3395) @tunetheweb

Postgres: Parse index with column sort (#3405) @greg-finley

MySQL: Improve SET Syntax for Variable Assignment (#3394) @mdahlman

Handle Postgres-style type casts when using the colon_nospaces templating style (#3383) @benji-york

Capitalization in help message (#3385) @mdahlman

MySQL: Update keywords (#3381) @mdahlman

Teradata: Database statement and Set Session Database (#3382) @mdahlman

New Contributors

@benji-york made their first contribution in #3383

@menzenski made their first contribution in #3430

Source code(tar.gz)
Source code(zip)
0.13.2(May 20, 2022)
Highlights

Major changes include:

Fix bug causing L003 to report indentation errors for templated code - sorry we know that one's caused many of you some grief :-(

Initial support of SOQL (Salesforce Object Query Language).

Additional Placeholder templating options.

Start of BigQuery procedural language support (starting simple FOR statements and CREATE PROCEDURE statements).

New rule L065 to put set operators onto new lines.

Many more dialect improvements and bug fixes.

What’s Changed

All dialects: Allow RESPECT NULLS/IGNORE NULLS in window functions (#3376) @tunetheweb

Postgres: correct RETURNS TABLE column type (#3379) @tunetheweb

L065: Add rule for set operators surrounded by newlines (#3330) @dmohns

L064: Apply preferred quote-style for partially templated quoted literals (#3300) @dmohns

BigQuery: Support Stored Procedures (#3369) @tunetheweb

MySQL extra Boolean operators (&&, ||, !) (#3359) @mdahlman

Postgres and Redshift: Support LOCK [TABLE] (#3350) @tunetheweb

Placeholder updates: Allow optional braces in dollar placeholders, add colon_nospaces, and cast to string (#3354) @tunetheweb

BigQuery: Basic FOR..IN..DO...END FOR support (#3340) @tunetheweb

L025: exclude VALUES clauses (#3358) @tunetheweb

GitHub Actions: Update existing PR on new runs (#3367) @greg-finley

GitHub Actions: Copy draft release notes to CHANGELOG (#3360) @greg-finley

GitHub Action to set version number (#3347) @greg-finley

Postgres and Redshift: support ALTER SCHEMA (#3346) @mdahlman

MySQL: better SELECT..INTO support (#3351) @tunetheweb

Postgres: support better function calls in CREATE TRIGGER (#3349) @tunetheweb

Misc rule doc updates (#3352) @tunetheweb

Snowflake: Move CASE keyword to Unreserved list (#3353) @tunetheweb

MySQL: Added support for multiple variables in SET statement. (#3328) @cgkoutzigiannis

SOQL: Support date_n_literal (#3344) @greg-finley

Update Docs: Getting Started and Index (#3339) @mdahlman

SOQL: Disable L026 rule (#3338) @greg-finley

Fix critical parse error logged after L003 fix (#3337) @barrywhart

SOQL: Disallow non-SELECT statements (#3329) @greg-finley

ci: bump github actions (#3336) @Fdawgs

Start SOQL dialect (#3312) @greg-finley

Hive: support CLUSTER, DISTRIBUTE, SORT BY (#3304) @barunpuri

Fix typo in Configuration documentation (#3319) @mdahlman

L011: Support MERGE statements (#3292) @tunetheweb

BigQuery: Add workaround to fix false-positves of L063 (#3306) @dmohns

Snowflake: REMOVE statement rework (#3308) @jmc-bbk

Snowflake: PUT statement (#3307) @jmc-bbk

Snowflake: GET statement (#3305) @jmc-bbk

Snowflake: Support ALTER EXTERNAL TABLE (#3302) @jmc-bbk

T-SQL: Fix PIVOT placement (#3298) @jpers36

Cleanup role references (#3287) @tunetheweb

Adding Typeform and videoask into inthewild.rst (#3296) @omonereo-tf

Snowflake: LIST statement enhancement (#3295) @jmc-bbk

MySQL: Support Create User (#3289) @greg-finley

Snowflake: CREATE STAGE grammar enhancement for file formats (#3293) @jmc-bbk

T-SQL: Complete support for delete statement (#3285) @pguyot

MySQL: Support account names (#3286) @greg-finley

L028: In T-SQL dialect, table variables cannot be used to qualify references (#3283) @barrywhart

L007: An operator on a line by itself is okay (#3281) @barrywhart

L046 (spaces around Jinja tags) should check all slices in a segment (#3279) @barrywhart

L003 bug fix: Not ignoring templated newline (#3278) @barrywhart

New Contributors

@omonereo-tf made their first contribution in #3296

@mdahlman made their first contribution in #3319

@cgkoutzigiannis made their first contribution in #3328

Source code(tar.gz)
Source code(zip)
0.13.1(May 6, 2022)
Highlights

Major changes include:

Addition of "rule groups" (currently core and all) to allow ease of turning on and off groups of rules.

Addition of db2 dialect

PRS errors are now highlighted in red.

Many bugs fixes and dialect improvements

What’s Changed

Allow optional AS in MERGE statements using SELECT #3276 @tunetheweb

Add groups each rule is in to Rules documentation #3272 @tunetheweb

BigQuery: Enhanced EXPORT DATA statement #3267 @tunetheweb

BigQuery: CREATE TABLE support for COPY and LIKE #3266 @tunetheweb

Improve error on missing keywords #3268 @greg-finley

MySQL: Add FLUSH support #3269 @greg-finley

Postgres: Add ALTER TYPE support #3265 @greg-finley

Bug fix: L036 handle single-column SELECT with comment on same line as SELECT keyword #3259 @barrywhart

Put working example in the README #3261 @greg-finley

Snowflake: Add CREATE FILE FORMAT Support #3104 @jmc-bbk

Bug fix: Disable L059 in snowflake dialect #3260 @barrywhart

Remove redundant Anything() from match_grammars #3258 @tunetheweb

Postgres: Add DOMAIN support #3257 @tunetheweb

T-SQL: Allow optional brackets with EXECUTE #3255 @pguyot

Add rule groups and a core rules group #3142 @pwildenhain

MySQL: Better UNSIGNED support #3250 @tunetheweb

MySQL (and others): Support DROP TEMPORARY TABLE #3251 @tunetheweb

Add Db2 dialect #3231 @ddresslerlegalplans

BigQuery: Add CREATE EXTERNAL TABLE statement #3241 @dmohns

SQLite: Add support for expressions in CREATE INDEX columns #3240 @tunetheweb

Fix exception in check_still_complete and matching in StartsWith #3236 @tunetheweb

Snowflake: Add Support for DROP Statements #3238 @chrisalexeev

Allow YAML generation script to accept arguments when run through tox #3233 @tunetheweb

Bug fix: Cleanly catch and report errors during load_macros_from_path #3239 @barrywhart

Indent procedure parameters #3234 @fdw

Enhance apply_fixes() to automatically fix violations of can_start_end_non_code #3232 @barrywhart

T-SQL: Fix for xml path identifier #3230 @fdw

SparkSQL: Additional Delta Merge Test Cases #3228 @R7L208

Fix bug where L018 warns inappropriately if CTE definition includes a column list #3227 @barrywhart

BigQuery: Better STRUCT support #3217 @tunetheweb

Fix bug where L003 and L036 fixes caused a parse error #3221 @barrywhart

Make IF EXISTS work with UNION selects #3218 @fdw

Fix bug where the fix_even_unparsable setting was not being respected in .sqlfluff #3220 @barrywhart

BigQuery: Better DELETE table support #3224 @tunetheweb

Snowflake: ALTER MATERIALIZED VIEW statement #3215 @jmc-bbk

BigQuery: recognise DATE, DATETIME and TIME as a date parts for EXTRACT #3209 @tunetheweb

Postgres: enhanced UPDATE statement support #3203 @tunetheweb

Prevent Date Constructors from being changed to double quotes by L064 #3212 @tunetheweb

Postgres: Fix DROP EXTENSION syntax #3213 @tunetheweb

Snowflake: Handle FLATTEN() table function aliases correctly in L025, L027, L028 #3194 @barrywhart

Snowflake: Function LANGUAGE SQL #3202 @WittierDinosaur

Postgres: Enhanced CREATE MATERIALIZED VIEW #3204 @tunetheweb

T-SQL: Support basic FOR XML statements #3193 @fdw

T-SQL: Fix cursor syntax #3192 @fdw

Snowflake: REMOVE statement enhancement #3191 @jmc-bbk

Snowflake: Moved VIEW to unreserved keywords #3190 @WittierDinosaur

BigQuery: Support EXPORT DATA #3177 @tunetheweb

T-SQL: Fix exception when using variable names in FROM clause #3175 @tunetheweb

Fix bug where encoding setting in .sqlfluff file was not being respected #3170 @barrywhart

Highlight PRS errors in red #3168 @OTooleMichael

Remove unnecessary StartsWith and make terminator mandatory when using it #3165 @tunetheweb

Postgres: Support Composite Types #3167 @tunetheweb

T-SQL: Support opening, closing, deallocating and fetching cursors #3166 @fdw

T-SQL: Add declaration of cursors #3164 @fdw

Missed #3151 from CHANGELOG #3163 @tunetheweb

Bug fix: L028 sometimes makes incorrect fix when there are subqueries #3156 @barrywhart

T-SQL: Support OUTPUT INTO #3162 @fdw

T-SQL: Add CREATE TYPE statement #3154 @fdw

Hive: SupportTABLESAMPLE #3159 @barunpuri

Hive: Support back quoted identifier and literal #3158 @barunpuri

T-SQL: Add table hints to INSERT and DELETE #3155 @fdw

New Contributors

@ddresslerlegalplans made their first contribution in #3231

@greg-finley made their first contribution in #3261

Source code(tar.gz)
Source code(zip)
0.13.0(Apr 22, 2022)
Highlights

Major changes include:

New Rule (L064) for preferred quotes for quoted literals

Rule speed improvements and fixing performance regression from 0.12.0

Add configuration option to disallow hanging indents in L003

Add ignore_words_regex configuration option for rules

New GitHub Annotations option

Many bug fixes and dialect improvements

What’s Changed

Redshift: Fix CREATE TABLE column constraints and COPY #3151 @tunetheweb

New Rule L064: Consistent usage of preferred quotes for quoted literals #3118 @dmohns

L025 bug fix: stop incorrectly flagging on nested inner joins #3145 @tunetheweb

T-SQL: Add labels, as well as GRANT/DENY/REVOKE #3149 @tunetheweb

Snowflake: allow bracketless VALUES in FROM clauses #3141 @tunetheweb

T-SQL: Support TRY_CONVERT #3143 @fdw

T-SQL: Support NVARCHAR(MAX) #3130 @fdw

Allow column-less INSERT INTO with bracketed SELECT in ANSI and BigQuery #3139 @tunetheweb

Hive: Support dynamic partition insert #3126 @barunpuri

T-SQL - ALTER TABLE - add support for WITH CHECK ADD CONSTRAINT and CHECK CONSTRAINT #3132 @nevado

TSQL: Support names for transactions #3129 @fdw

Snowflake: StartsWith() in FromExpressionElementSegment caused performance issues for large queries #3128 @WittierDinosaur

Fix parsing of Compound Statements #3121 @jonyscathe

SparkSQL: Update to support all valid Literal Types #3102 @R7L208

TSQL: Exclude non-function-name keywords from function names #3112 @jpers36

ANSI AT TIME ZONE parsing improvements #3115 @tunetheweb

When fixing a file, preserve the input file's permissions #3114 @barrywhart

Bug: L058 (flatten nested CASE) triggers incorrectly (the ELSE contains additional code) #3113 @barrywhart

Bug fix: Handle "lint" human-format file output correctly #3109 @barrywhart

L003: Add configuration option to disallow hanging indents #3063 @dmohns

Add native Github-actions output #3107 @dmohns

Improved signed literal parsing #3108 @tunetheweb

Don't allow fixes to span template blocks #3105 @barrywhart

Add ignore_words_regex configuration option #3098 @dmohns

Redshift: Better AT TIME ZONE support #3087 @tunetheweb

Fix In The Wild typo #3100 @sivaraam

Snowflake: Add Create Storage Integration grammar. #3075 @jmc-bbk

ANSI: Allow indented_using_on in MERGE statements ON #3096 @dmohns

Postgres: Support COLLATE in more clauses #3095 @tunetheweb

BigQuery: Support NORMALIZE function #3086 @tunetheweb

ANSI (and other dialects): Add DROP FUNCTION support #3082 @tunetheweb

Postgres: Support DROP EXTENSION #3083 @tunetheweb

Snowflake: Fix bug in Describe Statement #3076 @jmc-bbk

Update individual rules to take advantage of core rule processing changes #3041 @barrywhart

L003 forgives non misbalanced Jinja tags #3065 @OTooleMichael

Fix tsql dialect EXEC = @Variable StoredProc Failed Parsing Bug (#3070) #3077 @MartynJones87

Snowflake Dialect: Add External Function DDL #3071 @chrisalexeev

SparkSQL: Support for Delta UPDATE statement syntax #3073 @R7L208

SparkSQL: Test cases for Delta DELETE FROM syntax #3072 @R7L208

Postgres: Support quoted LANGUAGE params #3068 @tunetheweb

Fix bug handling Jinja set with multiple vars, e.g.: {% set a, b = 1, 2 %} #3066 @barrywhart

L007 should ignore templated newlines #3067 @barrywhart

Allow aliases to pass L028 #3062 @tunetheweb

Refactor core rule processing for flexibility and speed #3061 @barrywhart

Add editorconfig and precommit for SQL and YML files #3058 @tunetheweb

Rule L003 performance: Cache the line number and last newline position #3060 @barrywhart

Fixed documentation for sql_file_exts example #3059 @KulykDmytro

BigQuery: Support SAFE functions #3048 @tunetheweb

Postgres: Fix UNNEST for L025 #3054 @tunetheweb

Exasol: CREATE/DROP/ALTER USER/ROLE clean up for consistency #3045 @tunetheweb

Postgres add ALTER ROLE/ALTER USER support #3043 @mrf

Add CarePay to SQLFluff in the wild #3038 @pvonglehn

Postgres: Add ON CONFLICT Grammar #3027 @jmc-bbk

Add dialect to Docker test #3033 @tunetheweb

New Contributors

@chrisalexeev made their first contribution in #3071

@MartynJones87 made their first contribution in #3077

@sivaraam made their first contribution in #3100

@jonyscathe made their first contribution in #3121

@barunpuri made their first contribution in #3126

Source code(tar.gz)
Source code(zip)
0.12.0(Apr 7, 2022)
Highlights

Major changes include:

Dialect is now mandatory, either in command line, or in config BREAKING CHANGE

Rename spark3 dialect to sparksql BREAKING CHANGE

L027 now checks tables references exist BREAKING CHANGE

New rule L063 to allow Datatypes to have a different capitalisation policy from L010. BREAKING CHANGE

Refactor and performance improvements of Delimited and L003

Many dialect improvements and fixes

What’s Changed

MySQL: Allow JOINs in UPDATE expressions #3031 @zapion

Fix bug in patch generation for segments made of templated + literal fixes #3030 @barrywhart

Formatters code cleanup #3029 @barrywhart

Postgres better CREATE USER/CREATE ROLE support #3016 @mrf

SparkSQL: Add MERGE syntax #3025 @PhilippLange

Remove Delimited workarounds #3024 @tunetheweb

Add exclude option for Ref grammar #3028 @tunetheweb

SparkSQL: Adding support for Delta Lake table schema updates #3013 @R7L208

L027: Check table aliases exists #2998 @dmohns

Snowflake: Added support for REMOVE statements #3026 @WittierDinosaur

BigQuery: Support WEEK function with days of weeks #3021 @tunetheweb

Sparksql quoted identifier in STRUCT #3023 @PhilippLange

Force user to specify a dialect #2995 @barrywhart

BigQuery: Parse CREATE TABLE with trailing comma #3018 @dmohns

Snowflake: Add IS (NOT) DISTINCT FROM test cases #3014 @kd2718

BigQuery: Add support for column OPTIONS in STRUCT definitions #3017 @dmohns

PostgreSQL: added support for CREATE ROLE and DROP ROLE statements #3010 @dnim

Separate slow CI job to it's own workflow #3012 @tunetheweb

SparkSQL: Test cases for Delta Variation of Writing a table #3009 @R7L208

Snowflake: Added support for CLUSTER BY and other CREATE TABLE improvements #3008 @WittierDinosaur

Support TRIM function parameters #3007 @tunetheweb

BigQuery: Support AT TIME ZONE in EXTRACT #3004 @tunetheweb

BigQuery: Move some keywords to unreserved #3002 @tunetheweb

BigQuery: Allow quoted variable names in DECLARE #3006 @dmohns

T-SQL: allow optional AS keyword in parameters def #3001 @pguyot

T-SQL: add support for RETURNS @var TABLE syntax #3000 @pguyot

T-SQL: add support for parenthesized nested joins #2993 @pguyot

dbt: Read builtins from code #2988 @dmohns

T-SQL: fix table type in DECLARE statements #2999 @pguyot

T-SQL: allow leading GO #2997 @pguyot

T-SQL: add support for assignment operators #2996 @pguyot

BigQuery: Add more MERGE statement variants #2989 @dmohns

L041: Fix bug when there is a newline after the select clause modifier #2981 @jmc-bbk

Rule L045 doesn't recognise CTE usage in a subquery when rule L042 is enabled #2980 @barrywhart

dbt: Make is_incremental() defaults consistent #2985 @dmohns

Rename Grammars for consistency #2986 @tunetheweb

Added support for MySQL UPDATE Statements #2982 @WittierDinosaur

Redshift: Added CREATE EXTERNAL SCHEMA, bugfix in PARTITION BY #2983 @WittierDinosaur

Added ALTER INDEX and REINDEX to Postgres, Some Grammar Cleaning #2979 @WittierDinosaur

T-SQL grammar deduplication #2967 @tunetheweb

L003 Refactor #2884 @OTooleMichael

Delimited Refactor #2831 @WittierDinosaur

SparkSQL: Support for querying snapshots when reading data with Delta Lake #2972 @R7L208

Fix bug in L063 for BigQuery STRUCT params #2975 @tunetheweb

Fix assertion error in dbt templater when file ends with whitespace strip (-%}) #2976 @barrywhart

Pass dbt vars to dbt #2923 @tcholewik

BigQuery: Add support for column OPTIONS #2973 @dmohns

BigQuery: Allow expressions in OPTIONS clauses #2971 @dmohns

Bump black to 22.3.0 on pre-commit #2969 @pguyot

T-SQL: Redefine DatatypeIdentifierSegment #2959 @alanmcruickshank

T-SQL: Add support for WAITFOR statement #2968 @pguyot

T-SQL: Add WHILE statement support #2966 @pguyot

T-SQL: INTO is optional within INSERT statement #2963 @pguyot

Add basic IS (NOT) DISTINCT FROM support in most dialects #2962 @tunetheweb

SparkSQL: Create Table Delta Lake Variant #2954 @R7L208

T-SQL: Add support for CREATE/DROP/DISABLE TRIGGER #2957 @tunetheweb

Bug: L042 modifies parse tree even during "lint" #2955 @barrywhart

Allow multiple post function clauses in Postgres and Redshift #2952 @aviv

Fix bug in L022 for trailing comments in CTE #2946 @tunetheweb

More dialect checking, fixes, inheritance cleanup #2942 @barrywhart

T-SQL: Support OUTPUT Params and GOTO Statements #2949 @tunetheweb

BREAKING CHANGE: change existing dialect name from spark3 to sparksql #2924 @R7L208

Add Symend to SQLFluff In The Wild #2940 @HeyZiko

Simplify segment creation and inheritance in dialects #2933 @barrywhart

Snowflake: Add ALTER STREAM support #2939 @HeyZiko

T-SQL: Handle multiple nested joins #2938 @tunetheweb

Snowflake: Add CREATE STREAM support #2936 @HeyZiko

T-SQL: Support nested joins #2928 @tunetheweb

To replace base dialect segment class, must subclass or provide same stuff #2930 @barrywhart

Add new rule L063 to allow separate capitalisation policy for Datatypes #2931 @tunetheweb

Adds support for column definitions in table alias expressions #2932 @derickl

BigQuery: support numeric aliases in UNPIVOT clauses #2925 @tunetheweb

T-SQL: Supported nested MERGE statements #2926 @tunetheweb

New Contributors

@HeyZiko made their first contribution in #2936

@aviv made their first contribution in #2952

@pguyot made their first contribution in #2963

@dmohns made their first contribution in #2971

@tcholewik made their first contribution in #2923

@jmc-bbk made their first contribution in #2981

@dnim made their first contribution in #3010

@kd2718 made their first contribution in #3014

@mrf made their first contribution in #3016

@zapion made their first contribution in #3031

Source code(tar.gz)
Source code(zip)
0.11.2(Mar 25, 2022)
What’s Changed

Added Support For Snowflake Inline Comments (#2919) @WittierDinosaur

Spark3: Fix bug which did not allow multiple joins (#2917) @tunetheweb

Added Snowflake Alter View Support (#2915) @WittierDinosaur

Adjust L010 to ignore nulls and booleans covered by L040 (#2913) @tunetheweb

Fix bug where L043 corrupts SQL (#2908) @barrywhart

Jinja: Add support for Block Assignments (#2907) @barrywhart

L042 fix fails with missing function get_identifier() on Postgres, Redshift dialects (#2899) @barrywhart

BigQuery: Better Set Operators support (#2901) @tunetheweb

Hive: support for complex types in cast rowtype definition (#2896) @KulykDmytro

Hive: added json type support (#2894) @KulykDmytro

Hive: fix incorrect L027 error for rowtype attribute name (#2893) @KulykDmytro

Hive: Add ARRAY support (#2891) @tunetheweb

Implemented PIVOT/UNPIVOT Redshift + Fixed Snowflake Bug + Standardized PIVOT/UNPIVOT Parsing (#2888) @PLBMR

Fix AssertionError in dbt templater when file ends with multiple newlines (#2887) @barrywhart

Hive: Row typecasting in cast function (#2889) @KulykDmytro

dbt templater should gracefully skip macro files (#2886) @barrywhart

Disable L031 on BigQuery due to complex backtick / project name behavior (#2882) @barrywhart

Documentation: Update dbt templater docs with more detail about pros and cons (#2885) @barrywhart

BigQuery: Better STRUCT Array Support (#2881) @tunetheweb

L042: Detect violations when column is templated (#2879) @barrywhart

Improve parsing of BETWEEN statements (#2878) @MarcAntoineSchmidtQC

Source code(tar.gz)
Source code(zip)
0.11.1(Mar 17, 2022)
Highlights

Major changes include:

A number of changes to fix code to make these more robust

Improvements to templating blocks

generate_parse_fixture_yml options to allow quicker, partial regeneration of YML files

Numerous rule fixes including adding auto fix to L042

Numerous grammar changes

What's Changed

Spark3: Support for SHOW statements #2864 @R7L208

Add Markerr to list of organizations using SQLFluff in the wild #2874 @kdw2126

Refactor JinjaTracer: Split into two classes, break up _slice_template() function #2870 @barrywhart

BigQuery: support Parameterized Numeric Literals #2872 @tunetheweb

L042 autofix #2860 @OTooleMichael

Redshift: transaction statement #2852 @rpr-ableton

JinjaTracer fix for endif/endfor inside "set" or "macro" blocks #2868 @barrywhart

L009: Handle adding newline after {% endif %} at end of file #2862 @barrywhart

Redshift: Add support for AT TIME ZONE #2863 @tunetheweb

L032 bug fix and fix improvement #2859 @OTooleMichael

Refactor JinjaTracer; store lex output as individual strings where possible #2856 @barrywhart

Add ability to regenerate subsets of fixture YAMLs (by dialect, or new only) #2850 @OTooleMichael

Fix bug with Jinja and dbt {% set %} blocks #2849 @barrywhart

Bug fix: ValueError: Position Not Found for lint/parse/fix in JinjaTracer #2846 @barrywhart

Reduce unnecessary setting run ci #2847 @zhongjiajie

Spark3: statements to SET and RESET spark runtime configurations #2839 @R7L208

BigQuery - prevent L006 flagging hyphenated table references #2842 @tunetheweb

TSQL fix CONVERT function definition #2843 @tunetheweb

Change rule test script from bash to python #2840 @OTooleMichael

Spark3: Support DESCRIBE statement #2837 @R7L208

Spark3: Refactor REFRESH statements into one class #2838 @R7L208

Prevent rules incorrectly returning conflicting fixes to same position #2830 @barrywhart

Redshift and BigQuery: Update dateparts values and functions #2829 @rpr-ableton

MySQL add NOW support #2825 @tunetheweb

MySQL DELETE FROM support #2823 @tunetheweb

Rule L059 bug with IF #2824 @tunetheweb

Prevent exceptions when running fix on dialect fixtures #2818 @tunetheweb

Spark3: Support to handle CACHE AND UNCACHE auxiliary statements #2814 @R7L208

Fix L036 error on CREATE VIEW AS SELECT #2816 @tunetheweb

Fixes for the new post-fix parse check #2813 @barrywhart

Add initial MERGE syntax to most dialects #2807 @PhilippLange

Automated tests should fail if a lint fix introduces a parse error #2809 @barrywhart

New Contributors

@kdw2126 made their first contribution in #2874

Source code(tar.gz)
Source code(zip)
0.11.0(Mar 7, 2022)
Highlights

Major changes include:

Changes rule L030 to use extended_capitalisation_policy to support PascalCase BREAKING CHANGE

Fixes dbt error on ephemeral models

Log warnings for fixes that seem to corrupt the parse SQL as may cause incorrect fixes in other rules.

Bug fix to rule L011 for implicit aliases

Bug fix to rule L019 for commas besides templated code

Rule L051 can now optionally be applied to LEFT/RIGHT/OUTER JOINs

Improvements to Test Suite

Many dialect improvements

What's Changed

Exasol: Fix INTERVAL literals / expression #2804 @sti0

Exasol: Add IDLE_TIMEOUT and SNAPSHOT_MODE #2805 @sti0

Exasol: Support value range clause within INSERT statements (7.1+) #2802 @sti0

Exasol: Add lua adapter scripts (7.1+) #2801 @sti0

Exasol: Add openid support for create/alter user (7.1+) #2800 @sti0

Exasol: New consumer group params and unreserved keywords (7.1+) #2799 @sti0

Snowflake: Complete INSERT grammar #2798 @jpy-git

Fix Postgres VALUES, make Spark3 VALUES consistent #2797 @jpy-git

Postgres: INSERT DEFAULT value #2796 @jpy-git

Postgres: Make AS optional in Postgres DELETE #2794 @jpy-git

BigQuery support UNEST aliases #2793 @tunetheweb

Postgres: Add all range operators #2789 @jpy-git

Snowflake: Complete DELETE FROM grammar #2792 @jpy-git

Postgres: Complete DELETE FROM grammar #2791 @jpy-git

Postgres: Add RETURNING grammar to INSERT INTO statement #2790 @jpy-git

Snowflake: Complete PATTERN grammar #2788 @jpy-git

Redshift: add [ALTER/CREATE/DROP] PROCEDURE segments #2774 @rpr-ableton

Spark3: Support for ANALYZE TABLE statement #2780 @R7L208

Snowflake: Add MATCH_RECOGNIZE clause #2781 @jpy-git

Snowflake: Complete LIMIT grammar #2784 @jpy-git

Rough autofix for L028 #2757 @OTooleMichael

Spark3 bug: Create with complex data types (#2761) #2782 @PhilippLange

Snowflake: Complete LIKE grammar #2779 @jpy-git

Spark3: AuxiliaryFILE and JAR statements #2778 @R7L208

Snowflake: Refine SET/UNSET MASKING POLICY grammar #2775 @jpy-git

L049 bug: correct over zealous = --> IS #2760 @OTooleMichael

Make extenstion case insensitive #2773 @tunetheweb

Snowflake: Add dollar quoted string literal #2770 @jpy-git

Bug fix: L036 corrupts SELECT DISTINCT id query #2768 @barrywhart

Snowflake: Add CHANGES clause #2764 @jpy-git

Spark3: Support for EXPLAIN statement #2767 @R7L208

Snowflake: Add CONNECT BY clause #2763 @jpy-git

Spark3: Support for TRANSFORM clause #2762 @R7L208

Snowflake: Fix GROUP BY {CUBE|ROLLUP|GROUPING SETS} parsing #2759 @jpy-git

BigQuery: allow identifiers starting with dash #2756 @tunetheweb

Add ignore_words options to L057 and L059 #2753 @tunetheweb

L012 bug fix for TSQL alternative alias types #2750 @tunetheweb

Spark3: Support for PIVOT clause #2752 @R7L208

Update Redshift reserved keywords list #2751 @rpr-ableton

L007 autofix #2735 @OTooleMichael

L032 fixable in easy cases #2737 @OTooleMichael

Fix dbt templater runtime error in inject_ctes_into_sql() #2748 @barrywhart

L059: Exasol: Allow quotes around passwords in CREATE USER #2744 @sti0

Improve docs for load_macros_from_path #2743 @barrywhart

Make L045 (Query defines a CTE but does not use it) case insensitive #2746 @barrywhart

Add L049 test for T-SQL alternate alias syntax (=) #2745 @barrywhart

BaseSegment.pos_marker is typed as non optional but sometimes set to None #2741 @barrywhart

Support Pascal case for L030 #2739 @tunetheweb

Postgres, Redshift: Support SIMILAR TO pattern matching expressions #2732 @PLBMR

Forgive shorthand cast only / bracket only expressions from L013 #2729 @OTooleMichael

L052: Refactor _eval() into individual functions to improve readability #2733 @barrywhart

L018: Move closing parenthesis to next line #2734 @barrywhart

Improve rule yaml tests: assert that fix_str passes the rule #2624 @juhoautio

Extend rule L051 to LEFT/RIGHT/OUTER JOINs #2719 @rpr-ableton

T-SQL: Allow aliases with = #2727 @fdw

T-SQL: Support table variables #2728 @fdw

Support for checking violations in YAML rule tests #2718 @juhoautio

Roll back PR #2610 #2726 @barrywhart

Redshift: Allow whitespace around cast operators #2721 @PLBMR

Support database links in Oracle #2725 @tunetheweb

Rule L019: Ignore comma placement violations if the adjacent code is templated #2717 @barrywhart

T-SQL: Add drop constraint syntax #2724 @fdw

ANSI: Support optionally bracketed CTE #2716 @OTooleMichael

Spark3: Test cases for CASE clause #2714 @R7L208

Spark3: Support for WINDOW functions #2711 @R7L208

TSQL: Add variables as options for RAISERROR parameters #2709 @jpers36

TSQL: Add OPTION clause to UPDATE #2707 @jpers36

Spark3: Test cases for WHERE clause #2704 @R7L208

Spark3: test cases for Table-Valued Functions #2703 @R7L208

TSQL: Allow for optionally bracketed PARTITION BY elements #2702 @jpers36

TSQL: Fix SET TRANSACTION ISOLATION LEVEL parsing #2701 @jpers36

Migrate tricky L004 tests to python #2681 @juhoautio

Core linter enhancement: Check for successful parse after applying fixes #2657 @barrywhart

Spark3: Support for LATERAL VIEW clause #2687 @R7L208

Document python requirement for tox/mypy & remove basepython from conf #2644 @juhoautio

Fix rule L011 for implicit aliases #2683 @tunetheweb

Pin markupsafe to prevent CI failures #2685 @tunetheweb

Exasol: Allow CROSS joins #2680 @sti0

Exasol: Improve function formatting #2678 @sti0

TSQL: Add indentation for CREATE INDEX/STATISTICS #2679 @jpers36

Spark3: Support for TABLESAMPLE clause #2674 @R7L208

TSQL: Improve RAISERROR functionality #2672 @jpers36

Snowflake dialect update for MERGE INTO predicates #2670 @The-Loud

Assert that fix_str is set #2663 @juhoautio

New Contributors

@The-Loud made their first contribution in #2670

@OTooleMichael made their first contribution in #2716

@PhilippLange made their first contribution in #2782

Source code(tar.gz)
Source code(zip)
0.10.1(Feb 16, 2022)
Highlights

Major changes include:

Improvements to rules L023, L045, L048, L052, L059 to make them more accurate.

If sqlfluff fix cannot find a stable fix after runaway_limit iterations (default 10) then no fixes will be applied.

Addition of --write-output config to command line so prevent errors corrupting output.

Various dialect improvements

What’s Changed

Redshift: Support DATETIME as a valid datatype (#2665) @PLBMR

Support L033 for RedShift (#2661) @tunetheweb

Fix mistakes in rule yaml tests (#2658) @juhoautio

Fix parsing types and add check to test in future (#2652) @tunetheweb

Spark3: Support for SORT BY Clause (#2651) @R7L208

Migrate issue template from markdown to yaml (#2626) @zhongjiajie

L048 - handle more statements and exclude casting operators (#2642) @tunetheweb

MySQL support CURRENT_TIMESTAMP() in CREATE TABLE (#2648) @tunetheweb

Postgres enhanced DELETE FROM syntax (#2643) @tunetheweb

Bug fix: L025 should consider BigQuery QUALIFY clause (#2647) @barrywhart

Bug fix: L025 overlooking JOIN ON clause if join expression in parentheses (#2645) @barrywhart

L045 not reporting unused CTEs if the query uses templating (#2641) @barrywhart

Fix IndexError in L001 (#2640) @barrywhart

L052: If require_final_semicolon is set, ensure semicolon after ALL statements (#2610) @barrywhart

L023 to also fix extra newlines in CTE (#2623) @juhoautio

Spark3: Enhancements for Set Operators (#2622) @R7L208

Doc a better choice for default env (#2630) @juhoautio

Ensure ordering of fix compatible and config in rules docs (#2620) @zhongjiajie

Pin python version for tox -e mypy (#2629) @juhoautio

Hitting the linter loop limit should be treated as an error (#2628) @barrywhart

Allow file output directly from cli (#2625) @alanmcruickshank

BigQuery UNPIVOT and PIVOT fixes (#2619) @tunetheweb

L059 quoted identifiers bug (#2614) @tunetheweb

Snowflake dialect: Adjust snowflake array access (#2621) @alanmcruickshank

Spark3: Test Cases for ORDER BY in SELECT (#2618) @R7L208

Fix typos in 0.10.0 changelog (#2605) @tunetheweb

TSQL: Indent IF clause expression segments (#2615) @jpers36

Spark3: Enhancements for LIMIT Clause (#2612) @R7L208

Allow Bare Functions in column constraints (#2607) @tunetheweb

Add Oracle at and double at sign (execution symbol) (#2608) @r0fls

Spark3: Enhancements to LIKE clause (#2604) @R7L208

Source code(tar.gz)
Source code(zip)
0.10.0(Feb 10, 2022)
Highlights

Major changes include:

Dropping support of DBT < 0.20 ** BREAKING CHANGE **

sqlfluff fix no will no longer fix SQL containing parsing or templating errors ** BREAKING CHANGE **

New rule L062 to allow blocking of list of configurable words (e.g. syntax, or schemas, or tables you do not want people to use)

Lots and lots of docs improvements

Looser requirements for click python package

What’s Changed

L046: Detect Jinja spacing issues where segment begins with literal content (#2603) @barrywhart

MySQL Add BINARY support (#2602) @tunetheweb

Support indenting WINDOWS clauses and (optionally) CTEs (#2601) @tunetheweb

Postgres: Support expressions in arrays (#2599) @tunetheweb

BigQuery support Array of Structs (#2598) @tunetheweb

Support wildcards in triggers (#2597) @tunetheweb

Support CTEs in CREATE VIEW statements (#2596) @tunetheweb

SQLite Support more CREATE TRIGGER options (#2594) @tunetheweb

Snowflake: Support Column Comments in Alter Table statements (#2593) @tunetheweb

Redshift: Add DATETIME as reserved keyword (#2591) @tunetheweb

Support LIMIT and ORDER BY clauses in Values clauses (#2590) @tunetheweb

L016: New option "ignore_comment_clause" to ignore column COMMENTs, etc. (#2589) @barrywhart

Bug fix: L016 ("Line is too long") should consider length of prior fixes (#2587) @barrywhart

Add mysql INSERT ON DUPLICATE KEY (#2494) @rpr-ableton

Snowflake ALTER TABLE: Add multiple columns (#2578) @erevear

MySQL: UNIQUE KEY in CREATE TABLE (#2525) @jpy-git

Spark3: JOIN clause enhancements (#2570) @R7L208

Bug fix: L003 should fix indentation for templated code (#2580) @barrywhart

Exasol: Improve COMMENT and WITH [NO] DATA clause usage. (#2583) @sti0

Exasol: Allow multiple LOCAL keywords in WHERE clause (#2582) @sti0

Exasol: Allow LOCAL keyword within PREFERRING clause (#2579) @sti0

Add/Improve docs for config settings: "ignore", "ignore_templated_areas" (#2574) @barrywhart

Look for .sqlfluffignore in current directory (#2573) @barrywhart

Snowflake: L054 should ignore "WITHIN GROUP" clauses (#2571) @barrywhart

Redshift: Support Redshift SUPER Data Types (#2564) @PLBMR

Capitalization rules (L010, L014, L030, L040) should ignore templated code (#2566) @barrywhart

TSQL: Add Frame clause unreserved keywords (#2562) @jpers36

Simple API: Fix bug where omitted parameters still override .sqlfluff (#2563) @barrywhart

Spark3: Add Direct File Query (#2553) @R7L208

Redshift dialect: replace AnyNumberOf with AnySetOf where it makes sense (#2561) @rpr-ableton

jinja and dbt templaters: More robust handling of whitespace control (#2559) @barrywhart

Improve how "sqlfluff fix" handles templating and parse errors (#2546) @barrywhart

Jinja and dbt templater: Fix "list index out of range" error (#2555) @barrywhart

Fix typo in sqlfluffignore docs (#2551) @tunetheweb

Correct parsing for BigQuery SELECT REPLACE clauses. (#2550) @elyobo

Rules documentation improvements (#2542) @tunetheweb

Remove requirement for Click>=8 (#2547) @tunetheweb

Allow L059 to be configured to always prefer quoted identifiers (#2537) @niconoe-

Adds new rule L062 to allow blocking of certain words (#2540) @tunetheweb

Update to latest Black, drop support for dbt < 0.20 (#2536) @barrywhart

dbt templater: Fix bug where profile wasn't found if DBT_PROFILES_DIR contained uppercase letters (#2539) @barrywhart

Spark3: Added segments & grammar needed for hints (#2528) @R7L208

Spark3: parse some VALUES clauses (#2245) @mcannamela

TSQL: Allow multiple params in SET statements (#2535) @tunetheweb

TSQL: Add indentation for SET statement (#2531) @jpers36

Add additional documentation on dbt-adapter in pre-commit (#2530) @robertdefilippi

TSQL: Add indentation for UPDATE statement (#2532) @jpers36

Fix Snowflake Unordered Select Clause (#2529) @tunetheweb

Fix Quoted Literals for Postgres and Redshift affecting rule L039 (#2526) @tunetheweb

Postgres specific CTEDefinitionSegment (#2524) @jpy-git

New Contributors

@robertdefilippi made their first contribution in https://github.com/sqlfluff/sqlfluff/pull/2530

@niconoe- made their first contribution in https://github.com/sqlfluff/sqlfluff/pull/2537

@elyobo made their first contribution in https://github.com/sqlfluff/sqlfluff/pull/2550

@erevear made their first contribution in https://github.com/sqlfluff/sqlfluff/pull/2578

Source code(tar.gz)
Source code(zip)
0.9.4(Jan 30, 2022)
[0.9.4] - 2022-01-30

Highlights

Major changes include:

dbt performance improvements

Fix click dependency error.

Better datepart versus identifier parsing.

Fix some Jinja errors.

Various grammar fixes and improvements

What’s Changed

Spark3: test cases for HAVING clause in SELECT statement (#2517) @R7L208

Update click version requirement in setup.cfg to match that in requirements.txt (#2518) @barrywhart

Postgres: Implement DO Statements + Refactored Language Clause (#2511) @PLBMR

Spark3: Support for Grouping Sets, CUBE and ROLLUP in GROUP BY clause of SELECT statement (#2505) @R7L208

Refactor date part functions (#2510) @tunetheweb

Postgres: EXPLAIN ANALYSE allows British spelling (#2507) @jpy-git

"noqa": Add support for ignoring template (TMP) and parse (PRS) errors (#2509) @barrywhart

Freeze Black due to incompatibility between 22.1 and flake8-black (#2513) @tunetheweb

Support NATURAL JOINS (#2506) @tunetheweb

dbt Docker environment: Mount the test profiles.yml at ~/.dbt (#2502) @barrywhart

Add dbt_artifacts package to in the wild docs (#2504) @NiallRees

Spark3: Support DISTRIBUTE BY clause in SELECT statement (#2503) @R7L208

dbt templater: For performance reasons, cache the database connection across models (#2498) @barrywhart

Bug fix: Defining and using Jinja macro in the same file causes runtime error (#2499) @barrywhart

Spark3: Support CLUSTER BY clause in SELECT statement (#2491) @R7L208

Grammar: Adds support for COPY statement for Postgres dialect (#2481) @derickl

Add raiserror for T-SQL (#2490) @fdw

Enforce parentheses for function definitions in T-SQL (#2489) @fdw

Add guards to prevent rule crashes (#2488) @barrywhart

New Contributors

@PLBMR made their first contribution in https://github.com/sqlfluff/sqlfluff/pull/2511

Full Changelog: https://github.com/sqlfluff/sqlfluff/compare/0.9.3...0.9.4
Source code(tar.gz)
Source code(zip)
0.9.3(Jan 26, 2022)
[0.9.3] - 2022-01-26

Highlights

Major changes include:

Add ignore_words option for rules L010, L014, L029, L030, L040

Fix some issues in 0.9.2 preventing some queries linting

What’s Changed

Prevent L031 throwing exception on unparsable code (#2486) @tunetheweb

Add linting of fixtures SQL for critical rules errors to tox (#2473) @tunetheweb

Fix L039 for T-SQL comparison operator using space (#2485) @tunetheweb

Fix bug in get_alias causing rule Critical errors for T-SQL (#2479) @tunetheweb

Tweak GitHub templates (#2471) @tunetheweb

Small speed improvement to L054 (#2476) @tunetheweb

L003: Revisit recent change to improve speed (#2474) @barrywhart

Fix select_crawler issue with some Exasol statements (#2470) @tunetheweb

Cleanup date logic by removing DatePartClause and using DatetimeUnitSegment instead (#2464) @tunetheweb

Fix L044 exception when final statement has no SELECT (#2468) @tunetheweb

Support T-SQL system variables (e.g. @@rowcount) (#2463) @tunetheweb

Add base rule to developing rules page (#2462) @tunetheweb

L003: Ignore indentation of lines that only exist in templated space (#2460) @barrywhart

Ignore words for various rules (#2459) @tunetheweb

Support Foreign Key options for MySQL (#2461) @tunetheweb

Exclude WINDOW clauses from L054 (#2455) @tunetheweb

Fix bug with L026 for simple deletes (#2458) @tunetheweb

Spark3: test cases for Common Table Expressions (#2454) @R7L208

Fix T-SQL's IDENTITY_INSERT syntax (#2452) @fdw

T-SQL: Support stored procedures in insert statements (#2451) @fdw

Spark3: Support for LOAD DATA statements (#2450) @R7L208

Source code(tar.gz)
Source code(zip)
0.9.2(Jan 24, 2022)
Highlights

We are pleased to include 110 improvements and fixes in this release, and welcome 7 new contributors to the code.

Major changes include:

Initial Oracle support (note: SQL, but not PL/SQL)

Fix more dbt 1.0.0 connection issues

Improved configuration documentation

New rule (L059) to flag unnecessary quoted identifiers

New rule (L060) to prefer COALESCE instead of IFNULL or NVL

New rule (L061) to prefer != over <>

Many rule fixes

Many dialect improvements

What’s Changed

Prepare version 0.9.2 (#2403) @tunetheweb

Add Postgres DROP PROCEDURE support (#2446) @rpr-ableton

MySQL Alter table ADD/DROP/RENAME INDEX support (#2443) @tunetheweb

Add basic CREATE PROCEDURE support to Postgres (#2441) @tunetheweb

Indent T-SQL DECLARE and EXEC statements (#2439) @tunetheweb

Hive alternative types: INTEGER, DEC, NUMERIC (#2438) @tunetheweb

Implement Snowflake Dateparts (#2437) @tunetheweb

Fix rule L028 for T-SQL for params (#2442) @tunetheweb

Support CREATE UNIQUE INDEX (#2440) @tunetheweb

Make BigQuery typeless STRUCTs Expressions (#2435) @tunetheweb

T-SQL support default params and no RETURN value (#2434) @tunetheweb

"sqlfluff fix" should report any parse errors found (#2423) @barrywhart

Redshift VACUUM support (#2433) @rpr-ableton

Add Oracle PROMPT statement (#2413) @r0fls

Spark3: Support for INSERT OVERWRITE DIRECTORY with Hive Format (#2389) @R7L208

Exasol: Fix escaped identifiers (#2431) @sti0

Exasol: Fix LOCAL.ALIAS Syntax (#2430) @sti0

Exasol: Allow quoted identifier for various statements. (#2428) @sti0

Misc grammar improvements for Snowflake (#2421) @chwiese

New rule L061 to use != over <> (#2409) @sti0

Correct TRANS to TRAN (#2425) @fdw

Remove the "heuristic" slicer, as it was replaced by JinjaTracer (#2422) @barrywhart

L060: More specific description (#2419) @jpy-git

Fix code formatting in Rule docs (#2418) @tunetheweb

Allow UPDATE SET statements in RedShift (#2417) @tunetheweb

Add Redshift cursor DECLARE, FETCH & CLOSE support (#2414) @rpr-ableton

Add Redshift ANALYZE COMPRESSION support (#2412) @rpr-ableton

ANSI Values statement fixes (#2404) @jpy-git

Exasol: Overhaul drop statements (#2407) @sti0

L044, L045: Handle Exasol VALUES clause (#2400) @barrywhart

L060: Use COALESCE instead of IFNULL or NVL. (#2405) @jpy-git

Postgres: Fix Values alias regression (#2401) @jpy-git

Align line length in Python code to 88 characters (#2264) @chwiese

Jinja templater: Allow "load_macros_from_path" to be a comma-separated list of paths (#2387) @barrywhart

Add "TRANS" keyword for TSQL (#2399) @fdw

Docstrings: Replace double backtics with single quote for lint results. (#2386) @jpy-git

Spark3: Support for INSERT OVERWRITE DIRECTORY statements (#2385) @R7L208

Fix unnecessary white underline in doc site (#2383) @tunetheweb

Rolls back some code cleanup that caused coverage report to show gaps (#2384) @barrywhart

Fix "connection already closed" issue with dbt 1.0 and dbt_utils (#2382) @barrywhart

Spark3: Support for INSERT [TABLE] data manipulation statements (#2290) @R7L208

Comment out line in bug report template (#2378) @jpy-git

Postgres: EXPLAIN statement updates (#2374) @jpy-git

Make TABLE a non-reserved word in Postgres (#2377) @tunetheweb

Snowflake COLUMN is not a reserved word (#2376) @tunetheweb

TSQL: Complete ASA Table Index Clause functionality (#2373) @jpers36

Add support for Jinja import and include (#2355) @barrywhart

Add Redshift INTERVAL datatype support (#2366) @rpr-ableton

Whitespace concatenated string literals for MySQL, Postgres and Redshift (#2356) @jpy-git

Fix L026 false positive on "SELECT INTO" statement (#2371) @barrywhart

Exclude EMIT clauses from rule L013 (#2364) @tunetheweb

Functional API: Segments.recursive_crawl (#2369) @jpy-git

Complete Redshift CREATE EXTERNAL TABLE support (#2354) @rpr-ableton

L041: Fix duplicate DISTINCT corruption (#2365) @jpy-git

Bigquery Create View with Options (#2359) @tunetheweb

L026: Handle DML statements and multiple levels of nesting (#2336) @barrywhart

Postgres & MySQL: cleanup AliasExpressionSegment (#2353) @jpy-git

Redefine MySQL Interval segment (#2351) @rpr-ableton

Postgres: INSERT INTO table alias (#2349) @jpy-git

L043: Remove redundant CASE statement replacing NULLS with NULLS (#2346) @jpy-git

Add RedShift DATASHARE support (#2350) @rpr-ableton

Various documentation updates (#2347) @tunetheweb

Snowflake ALTER TABLE: Drop multiple columns (#2348) @jpy-git

Configuration doc: add rule configuration section (#2291) @juhoautio

Redshift: create model, show model & data types (#2338) @rpr-ableton

L059: Unnecessary quoted identifier (#2341) @jpy-git

L043: Use simple replace to apply fixes (#2343) @jpy-git

TSQL: Add functionality to PARTITION BY clause (#2335) @jpers36

L039 casting operator postgres fix (#2334) @jpy-git

AnySetOf grammar (#2326) @jpy-git

Redshift: update CREATE TABLE AS match_grammar (#2333) @rpr-ableton

Redshift CREATE EXTERNAL TABLE: TABLE PROPERTIES (#2330) @jpy-git

Snowflake: Flush out ALTER TABLE's tableColumnAction grammar (#2332) @wong-codaio

Snowflake ALTER TABLE: Add clusteringAction (#2329) @jpy-git

Snowflake ALTER TABLE: Add searchOptimizationAction (#2328) @jpy-git

Fix numeric literal grammar for Postgres/MySQL/Exasol (#2324) @jpy-git

L039: Remove spaces between comparison operators (T-SQL) (#2325) @jpy-git

Enable setting a target of a dbt profile (#2236) @yu-iskw

Snowflake: Add support for column rename (#2327) @wong-codaio

Snowflake: Added AlterTableStatement specific for Snowflake (#2267) @wong-codaio

Full REFERENCES grammar for CREATE TABLE statement (#2315) @jpy-git

Fix Spark numeric literals (#2317) @jpy-git

Change type of Snowflake stage paths to fix issues with L044 (#2320) @chwiese

Add Bytes Quoted Literals to Spark dialect (#2312) @jpy-git

Fix L044 assertion failure with delete stmt & cte (#2321) @barrywhart

L003 should consider only literal leading whitespace (ignore templated) (#2304) @barrywhart

Redshift: update reserved keywords (#2318) @rpr-ableton

docs: Document how to run SQLFluff with local changes to test them (#2316) @kayman-mk

Update redshift unreserved keywords (#2310) @jpy-git

Fix spark and hive quoted literals (#2311) @jpy-git

Oracle Dialect (#2293) @r0fls

Redshift dialect: add COPY and UNLOAD statements (#2307) @rpr-ableton

L052: Fix case where no preceding segments and mulitline (#2279) @jpy-git

Update rule L049 to handle EXEC assignments (#2308) @tunetheweb

Remove DATE, DATETIME and TIME from BigQuery DatePart (#2283) @tunetheweb

Fix #1292: nocolor and verbose can work in config files (#2300) @cympfh

Allow pyproject.toml as extra_config_path (#2305) @jpy-git

L009: Handle adding newline after trailing templated code (#2298) @barrywhart

added missing "t" in doc for Rule_L020 (#2294) @Xilorole

docs: Document configuration keyword for rule L054 (#2288) @tomasfarias

Update L009 to operate in raw, not templated space (#2285) @barrywhart

Redshift CREATE LIBRARY statements (#2277) @rpr-ableton

L025 with 'bigquery' dialect: Correctly interpret calling functions with a table as a parameter (#2278) @barrywhart

Spark3: Coverage for REFRESH auxiliary statements (#2282) @R7L208

Spark3: Coverage for USE DATABASE statement. (#2276) @R7L208

Fix link for editing 'In The Wild' page with new base branch, main (#2280) @barnett

Optionally allow additional configurable characters in L057 (#2274) @tunetheweb

L025 should look at subqueries (#2273) @barrywhart

Add coverage for TRUNCATE statement in Spark3 dialect (#2272) @R7L208

Upgrade click version to 8.0+ to support click.shell_completion (#2271) @wong-codaio

Improve release checklist to make releases easier (#2263) @tunetheweb

Source code(tar.gz)
Source code(zip)
0.9.1(Jan 9, 2022)
Highlights

Fix dbt 1.0.0 connection issue

Fix some SQL corruption issues with templated code

New components to simplify creating rules

Remove support for Python 3.6

What's Changed

Fix delimited identifier parsing for spark3 #2111 @mcannamela

Stop numeric literal from splitting valid naked identifiers. #2114 @jpy-git

Grammar: Add CREATE USER/GROUP statement to Redshift dialect #2115 @jpy-git

Fix mypy type raise in L003 #2127 @barrywhart

Add ability to parse multiple GO/semicolon delimiters #2124 @jpy-git

Allowed array/struct values in default definition of declare #2120 @KulykDmytro

Normalise input newlines #2128 @jpy-git

Clean up all files using the pre-commit hook #2123 @kayman-mk

Refined LintFix API #2133 @jpy-git

Hotfix for LintFix comparisons #2138 @jpy-git

Lint spaces in qualified names #2130 @jpers36

Remove support for Python 3.6 (it's "end of life" December 23, 2021) #2141 @barrywhart

Fully remove python3.6 references #2142 @jpy-git

Fix L022 to not flag CTE column definitions #2139 @jpy-git

docs: set dbt_modules to dbt_packages #2143 @ciklista

Hive: add INTERVAL syntax #2144 @juhoautio

Fix mypy error on python 3.7 #2147 @juhoautio

Update PR template to reference tox generate-fixture-yml command #2148 @jpy-git

Update index.rst notable changes with 0.9.0 details #2132 @jpy-git

Add ALTER USER and ALTER GROUP to redshift dialect #2131 @jpy-git

Add complete DESCRIBE grammar to Snowflake dialect #2149 @jpy-git

Fix bug with BigQuery UNPIVOT #2156 @tunetheweb

Make L057 compatible with BigQuery #2151 @tunetheweb

TSQL: Proper Indentation of ELSE IF #2157 @jpers36

Linter Test Name Duplication #2158 @jpers36

Add struct support for hive and redshift (L026, L028) #2154 @KulykDmytro

Postgres - Support functions prepended with _ and containing $ #2159 @jpy-git

TSQL: function parsing/linting #2155 @jpers36

TSQL: Add THROW statement #2163 @jpers36

Add yamllint to project #2162 @tunetheweb

Fix outdated docstring in dialects_test #2166 @juhoautio

Minor comment fixes #2179 @juhoautio

L010 to apply to date_part (capitalization policy for time units) #2167 @juhoautio

ALTER GROUP fix to accommodate quoted objects #2188 @tdstark

Lexer: add non-breaking spaces to whitespace #2189 @jpers36

Grammar: Add COMMENT statement to Snowflake #2173 @jpy-git

Grammar: Add DISCARD statement to Postgres dialect #2175 @jpy-git

Grammar: Add UNDROP statement to Snowflake dialect #2177 @jpy-git

Grammar: Add UNSET statement to Snowflake dialect #2181 @jpy-git

Grammar: Add RESET statement to Postgres dialect #2182 @jpy-git

Grammar: Add LOAD statement to Postgres dialect #2183 @jpy-git

Grammar: Fix TRUNCATE statement in Snowflake dialect #2184 @jpy-git

Grammar: Add HELP statement to MySQL dialect #2191 @jpy-git

Grammar: Add PURGE BINARY LOGS statement to MySQL dialect #2193 @jpy-git

Grammar: Add RESET MASTER statement to MySQL dialect #2194 @jpy-git

Grammar: Add RENAME TABLE statement to MySQL dialect #2195 @jpy-git

Grammar: Tidy up transaction statements in Snowflake dialect #2196 @jpy-git

Modifying Redshift USER/GROUP Statements To Use ObjectReferenceSegment #2190 @tdstark

Grammar: Fix TRUNCATE statement in Postgres dialect #2185 @jpy-git

Grammar: Add LISTEN, NOTIFY, and UNLISTEN statements to Postgres dialect #2174 @jpy-git

Grammar: Tidy up Snowflake/MySQL/HIVE USE statements #2187 @jpy-git

Make Snowflake keywords unreserved: account, organization, pivot #2172 @jpy-git

Grammar: Add SET sql_log_bin statement to MySQL dialect #2192 @jpy-git

Grammar: Add CALL statement to Snowflake dialect #2176 @jpy-git

L027 Fix self referring column alias edge case #2171 @jpy-git

TSQL: Remove dependency on ANSI keyword lists #2170 @jpers36

Grammar: Add Table Maintenance Statements to MySQL dialect #2198 @jpy-git

Adding CREATE TABLE AS to Redshift #2205 @tdstark

TSQL: Add support for ALTER TABLE ALTER COLUMN #2208 @jpers36

Remove oyaml in favour of pyyaml #2210 @jpy-git

Support Spark CREATE TABLE LIKE syntax #2207 @R7L208

Add override for linguist to include SQL in language statistics #2214 @jpy-git

Add type stubs for appdirs and colorama to improve mypy coverage #2211 @jpy-git

Remove cached-property in favour of stdlib functools implementation #2212 @jpy-git

Restructure CASE segment (extract WHEN and ELSE into their own segment types) #2213 @barrywhart

Add types-regex package for type checking #2216 @jpy-git

Snowflake: Split out CREATE VIEW into its own segment #2217 @wong-codaio

Grammar: Fix multi-character comparison operators #2197 @jpy-git

Snowflake: Support TOP N select clause modifier #2222 @wong-codaio

Fix CLI arguments to allow for autocompletion #2218 @jpy-git

Simplify rule creation by adding a functional API to RuleContext #2126 @barrywhart

Simplify nested cases #2223 @barrywhart

Reword lint message for L058 per review #2226 @barrywhart

Update BaseRule.discard_unsafe_fixes() to avoid touching templated code #2220 @barrywhart

Add L059 - Capitalization on Data Types #2227 @tdstark

TSQL: Table valued functions #2233 @jpers36

Don't allow fixes to COPY code from templated regions #2231 @barrywhart

Fix several small issues with rule docs #2234 @barrywhart

postgres: Add datatypes #2121 @kayman-mk

Combine L059 and L010 #2238 @tdstark

Fix L044 assertion failure: "SELECT *" with no "FROM" clause #2239 @barrywhart

Docs: Make Specific Rules docstring more user friendly #2241 @jpy-git

Fix a bug handling Jinja "{% set %}" blocks with a templated block inside #2240 @barrywhart

Redshift lint create external table statements #2229 @tinder-albertyue

Update tox.ini for best practices #2243 @jpy-git

Docs: Make code blocks consistent #2242 @jpy-git

Add support for nested Jinja macros #2246 @barrywhart

Support DROP DDL statements for Spark3 #2215 @R7L208

Docker Compose environment for SQLFluff developers #2254 @barrywhart

TSQL: Add OFFSET unreserved keyword #2258 @jpers36

Fix connection issue in dbt 1.0.0 #2230 @NiallRees

Redshift CREATE SCHEMA statements #2252 @rpr-ableton

Enhance Snowflake COPY INTO #2250 @chwiese

Coverage for 'REPAIR' Statements for Hive & Spark3 dialect #2256 @R7L208

Source code(tar.gz)
Source code(zip)
0.9.0(Dec 13, 2021)
What’s Changed

This release brings about several great new additions including:

dbt 1.0.0 compatibility.

CLI and Simple API paramaters to provide custom paths to config files.

Refinement to Simple API to return parse output in JSON format rather than as an internal SQLFluff object (BREAKING CHANGE).

An Official SQLFluff Docker Image.

Grammar improvements across various dialects.

A new rule (L057) to check for non-alphanumeric values in identifiers.

There have also been many bug fixes and improvements to the CI and development processes.

🚀 Enhancements

TSQL: Reserved Keyword cleanup #2100 @jpers36

Add wiki links to CONTRIBUTING.md #2106 @tunetheweb

Add snowflake create stage and alter stage statements + RegexParser case fix #2098 @chwiese

Allow for more value types in ALTER TABLE ALTER COLUMN SET DEFAULT statement #2101 @derickl

Grammar: Adds support for ALTER VIEW statement for Postgres dialect #2096 @derickl

Add example for using JSON output of Simple API parse function #2099 @jpy-git

Allow optional keywords in create table unique constraints #2077 @kayman-mk

Grammar: Adds support for ALTER FUNCTION statement for Postgres dialect #2090 @derickl

Grammar: adds support for CREATE/ALTER/DROP DATABASE for Postgres dialect #2081 @derickl

Update parse method of Simple API to output JSON parse tree #2082 @jpy-git

TSQL dialect: add parsing for MERGE statement #2057 @tkachenkomaria244

Simple API config path #2080 @jpy-git

dbt 1.0.0 compatability #2079 @alanmcruickshank

Parse on delete and on update clause for create table constraints #2076 @kayman-mk

Pre-commit: Add hook for doc8 #2074 @jpy-git

Grammar: Fix typo in Alter Table parser in Postgres dialect #2072 @derickl

Grammar: Adds support for materialized views for postgres dialect #2041 @derickl

Add basic pre-commit config #2067 @jpy-git

CLI: Add --ignore-local-config flag #2061 @jpy-git

TSQL: INSERT INTO #2054 @jpers36

Add --disable-noqa option to CLI and config #2043 @jpy-git

TSQL: TRY/CATCH #2044 @jpers36

enabled arrays support in declare and set statements for bigquery dialect #2038 @KulykDmytro

L008 refactor #2004 @jpy-git

Support init.py for library_path #1976 @Tonkonozhenko

L052: Redefine semi-colon newline to multiline newline #2022 @jpy-git

Grammar: Remove hash inline comment from Postgres #2035 @jpy-git

noqa enhancement: Enable glob rule matching for inline comments #2002 @jpy-git

TSQL (ASA): Allow for table identifier in DELETE clause #2031 @jpers36

TSQL (ASA): Fix CTAS with WITH statement #2028 @jpers36

Grammar: Parse multiple grants #2023 @jpy-git

Add tsql nested block comment support and add regex package dependency #2027 @jpy-git

Grammar: Add complete Snowflake datetime units #2026 @jpy-git

Grammar: Add DROP POLICY statement to postgres dialect #2024 @jpy-git

Grammar: Add complete datetime units to postgres dialect #2025 @jpy-git

Grammar: Postgres CREATE POLICY #2021 @jpy-git

Speed up CI #1957 @pwildenhain

Add support for Snowflake create/alter SQL and js UDF #1993 @chwiese

Add encoding CLI argument #1994 @jpy-git

TSQL: Spaces allowed in comparison operators #1965 @jpers36

Add Snowflake schema options #1950 @chwiese

CLI/.sqlfluff enhancement: Rule globs #1972 @jpy-git

Add config CLI argument to lint, fix, and parse #1986 @jpy-git

Add type hints to simple API #1951 @jpy-git

New rule to flag special characters in identifiers #1958 @jpers36

Allow column references in IN statement #1971 @tunetheweb

Remove config.ini in favor of setup.cfg #1966 @jpy-git

Convert sqlfluff-templater-dbt setup.py to setup.cfg #1963 @jpy-git

Official Docker image: Dockerfile and Github Actions workflow #1945 @jpy-git

Move package metadata to setup.cfg #1960 @jpy-git

🐛 Bug Fixes

Fix tsql block comment close #2095 @jpy-git

Fix PlaceholderTemplater slice_type for templated code (substitutions) #2085 @barrywhart

Exasol: Fix UDF script syntax #2083 @sti0

Fix issues with placeholder templating docs #2078 @jpy-git

Update dbt templater docs to clarify that the profiles_dir setting is optional #2070 @barrywhart

Bug fix of L054 for Snowflake and Exasol #2069 @tunetheweb

Fix L043 issue when trying to autofix functions #2059 @jpy-git

Add request for users dbt version in bug_report issue template #2058 @jpy-git

Fix parameters for Snowflake create tasks statement #2037 @chwiese

Linguist: Include test/** in language statistics to better reflect use of SQL #2034 @jpy-git

L044 should handle nested CTEs #1991 @barrywhart

Add dbt adapter install advice to configuration documentation #2011 @jpy-git

Update pre-commit dbt instructions to reference separate dbt package #2005 @jpy-git

Fix config.get for iterable sections #2020 @jpy-git

Fix inline comment interactions with L052 #2019 @jpy-git

Make Snowflake tags DRY #1992 @chwiese

Rename whitelist/blacklist to allowlist/denylist #1989 @jpy-git

Fix issue with inline ignores not respecting comment lines #1985 @jpy-git

Fix L009 FileSegment child + new create_before/create_after edit types #1979 @jpy-git

Adds extra check to L054 to avoid weird error messages #1988 @tunetheweb

BigQuery: Allow keywords in column reference components #1987 @tunetheweb

L027: Remove unnessary crawl in get_select_statement_info #1974 @jpy-git

Add all attributes to init.py files to resolve F401 #1949 @jpy-git

Fix incorrect comment on L055 #1967 @jpy-git

Docs: fix docker hub link to public URL #1964 @kevinmarsh

Fix issue releasing dbt package: tox commands run relative to repo root #1962 @jpy-git

Source code(tar.gz)
Source code(zip)
0.8.2(Nov 22, 2021)
What’s Changed

One of the biggest new features in this release is the support for SQLAlchemy and other "placeholder" templating within SQL queries. Check out the documentation on how to set it up.

This release also adds seven new rules. Get some help with your leading whitespace, semi-colon placement, inconsistent column references in GROUP BY/ORDER BY, and getting rid of RIGHT JOIN's among other useful lints with our new rules! See our rules documentation for more details.

On top of those, we have made loads of grammar improvements across many dialects, improvements to the dbt templater (including issues where sqlfluff fix would corrupt the code :scream:), more fix routines, and lots more improvements.

🚀 Enhancements

[many dialects] Implement generic placeholder templating #1887 @jacopofar

[many dialects] Add support for SQLAlchemy templating #1878 @jacopofar

Add DROP PROCEDURE statement to T-SQL #1921 @jpy-git

TSQL dialect: fix index/tables creation options #1955 @tkachenkomaria244

Add DROP TYPE statement to ANSI dialect #1919 @jpy-git

Add INSERT INTO statements to Redshift Dialect #1896 @tdstark

Added TABLESAMPLE support to Bigquery #1897 @WittierDinosaur

Add [LEFT] ANTI and [LEFT] SEMI joins to the Spark3 dialect #1942 @jpy-git

Parse UPDATE/INSERT within WITH clause #1889 @jpy-git

Add OVERRIDING SYSTEM/USER VALUE to insert statement in postgres dialect #1869 @jpy-git

Add support for DROP SCHEMA [IF EXISTS] name [ CASCADE | RESTRICT ] #1865 @gimmyxd

Add CREATE TABLE Statement To Redshift #1855 @tdstark

Add DROP TYPE statement in postgres dialect #1870 @jpy-git

Add SEQUENCE NAME to postgres sequence options #1866 @jpy-git

Added SET Statement to Postgres #1877 @WittierDinosaur

Postgres: Allow use of quoted identifiers to ALTER TABLE OWNER TO #1856 @markpolyak

Updates to COPY INTO grammar in Snowflake #1884 @WittierDinosaur

Postgres & T-SQL: Drop Function #1924 @WittierDinosaur

Add Expressions to SET syntax #1852 @tunetheweb

Update DbtTemplater to use JinjaTracer #1788 @barrywhart

L043 refinement: Add autofix for common use of CASE to fill NULL values. #1923 @jpy-git

New Rule L050: No leading whitespace #1840 @jpy-git

L050: updating to target jinja templates #1885 @jpy-git

New rule L051 to forbid lone JOIN #1879 @jpy-git

New Rule L052: Semi colon alignment #1902 @jpy-git

New Rule L053: Remove outer brackets from top-level statements. #1916 @jpy-git

New Rule L054: Inconsistent column references in GROUP BY/ORDER BY clauses. #1917 @jpy-git

New Rule L055: Use LEFT JOIN instead of RIGHT JOIN. #1931 @jpy-git

New Rule L056: 'SP_' prefix should not be used for user-defined stored procedures #1930 @jpy-git

Tsql partition by multiple columns #1906 @jpers36

Added bare functions to values clause #1876 @WittierDinosaur

Remove unnecessary context section, from code and the docs #1905 @jacopofar

L036 docstring refinements #1903 @jpy-git

Add exclude_rules option for the Simple API #1850 @tunetheweb

Tox improvements: Streamline development/testing environments. #1860 @jpy-git

Add Tox publish commands #1853 @jpy-git

Documentation: Change inheritance dialect example to Redshift #1900 @chwiese

Remove failing requires.io badge #1898 @jpy-git

[Snowflake] Allow naked AUTOINCREMENT #1883 @gordonhart

Add support for curly brackets in SnowSQL ampersand variables #1901 @chwiese

Add short form help option (-h) #1947 @jpy-git

Remove plaintext API key from benchmark utility #1863 @jpy-git

Add skip_install to static analysis sections of tox.ini #1851 @jpy-git

Move typing_extensions from requirements_dev.txt to requirements.txt #1956 @jpy-git

🐛 Bug Fixes

Fix bug where "sqlfluff fix" deletes dbt "{% snapshot %}" line #1907 @barrywhart

Fix subquery bug in L026 #1948 @jpy-git

Fix bug where L041 was confused by L016's placement of newlines in the parse tree #1904 @barrywhart

Fix progressbar artifacts within linter errors #1873 @adam-tokarski

Correct Snowflake warehouse sizes #1872 @jpy-git

Fixed Delimited() logic, added TSQL grammar #1894 @WittierDinosaur

L036 refinement - FROM clause interaction #1893 @jpy-git

Add missing chardet install in setup.py #1928 @jpy-git

Fix misplaced TableAliasInfo in L031 documentation #1946 @jpy-git

Fix broken link to external SQL style guide #1918 @kevinmarsh

Source code(tar.gz)
Source code(zip)
0.8.1(Nov 7, 2021)
What’s Changed

Fixes missing dependency issue with 0.8.0 for tqdm, plus add a test to ensure this does not happen again.

🐛 Bug Fixes

Fix: add tqdm to setup.py installation requirements #1842 @skykasko

Add test to ensure pip install works #1843 @tunetheweb

Source code(tar.gz)
Source code(zip)

A SQL linter and auto-formatter for Humans

Related tags

Overview

The SQL Linter for Humans

Dialects Supported

Templates Supported

Getting Started

Documentation

Releases

SQLFluff on Slack

SQLFluff on Twitter

Contributing

Comments

Expected Behaviour

Observed Behaviour

1st run of sqlfluff fix

2nd run of sqlfluff fix

3rd run of sqlfluff fix

4th run of sqlfluff fix

Steps to Reproduce

Dialect

Version

Configuration

Search before asking

What Happened

Expected Behaviour

Observed Behaviour

How to reproduce

Dialect

Version

Configuration

Are you willing to work on and submit a PR to address the issue?

Code of Conduct

Pull Request checklist

Brief summary of the change made

Are there any other side effects of this change that we should be aware of?

Pull Request checklist

Search before asking

What Happened

Expected Behaviour

Observed Behaviour

How to reproduce

Dialect

Version

Configuration

Are you willing to work on and submit a PR to address the issue?

Code of Conduct

Search before asking

Description

Use case

Dialect

Are you willing to work on and submit a PR to address the issue?

Code of Conduct

Current situation

Background

Proposal

Related issues

Notes

Expected Behaviour

Observed Behaviour

Steps to Reproduce

Dialect

Version

Configuration

Search before asking

What Happened

Expected Behaviour

Observed Behaviour

How to reproduce

Dialect

Version

Configuration

Are you willing to work on and submit a PR to address the issue?

Code of Conduct

Search before asking

What Happened

Expected Behaviour

Observed Behaviour

How to reproduce

Dialect

1st run of `sqlfluff fix`

2nd run of `sqlfluff fix`

3rd run of `sqlfluff fix`

4th run of `sqlfluff fix`