tfquery: Run SQL queries on your Terraform infrastructure. Query resources and analyze its configuration using a SQL-powered framework.

Overview

🌩️ tfquery 🌩️

Run SQL queries on your Terraform infrastructure. Ask questions that are hard to answer


🚀 What is tfquery?

tfquery is a framework that allows running SQL queries on Terraform code. It's made to analyze your Terraform infrastructure, locate resources, run security compliance checks, spot misconfigured resources, develop CI benchmarks, and much more.

tfquery is made to help in answering questions that are hard to answer about your infrastructure-as-code. It allows querying resources and analyzing its configuration using a SQL-powered framework.

Why?

infrastructure-as-code is the de-facto today for documenting and deploying infrastructure on cloud providers. As the organization grows, it becomes really hard to understand and analyze the deployed infrastructure. Grepping and searching for resources in Terraform state files is not enough. Terraform Modules are automating processes dynamically for infrastructure deployment, so searching for static resources is also not feasible for good visibility.

With tfquery, you can run SQL queries on Terraform state files, and gain the best possible visibility.


💡 Use tfquery to

  • Have full coverage of your infrastructure, without being locked on a specific provider, including Amazon AWS, Microsoft Azure, Google Cloud Platform, Alibaba Cloud, IBM Cloud, Oracle Cloud, and many others.

  • Analyze deployed resources configuration.

  • Develop CI and monitoring checks for cloud infrastructure.

  • Write custom queries to scan Terraform resources.


tfquery vs. Cloud-specific SQL engines?

There are cloud-specific SQL engines that allow you to run SQL queries to understand resources on their infrastructure, both are covered as provided service by the cloud provider, or given as an open-source tool or a product. The main difference tfquery brings:

  • Maintainability: Cloud-specific SQL engines require maintenance in case of new services or breaking changes to existing ones. tfquery make use of Terraform schemas as a standard. tfquery will work on all given services, without the need to continuously update it with new API specs.

  • Coverage: tfquery covers all the cloud providers that Terraform supports out of the box (thanks to Terraform Providers).


📖 Usage

Run SQL query on Terraform states

>>> import tfquery
>>>
>>> result = tfquery.tfstate.run_query("terraform.tfstate", "select count(*) from resources")
>> print(result)
[{'count(*)': 86}]

Parse all resources from a Terraform state file

>>> import tfquery
>>>
>>> resources = tfquery.tfstate.parse_resources("terraform.tfstate")
>>> print(f"[i] Resources Count: {len(resources)}")
[i] Resources Count: 1475

Advanced Usage

Migrate Version 3 to Version 4 Terraform states

This is a parsing library to migrate the older Version 3 Terraform states to a Version 4 state. This is made to add backward compatibility for Terraform states that is made for releases older than Terraform v0.11.

>>> import tfquery
>>>
>>> tfstate_v3 = tfquery.tfstate.load_file("terraform.tfstate")
>>> tfstate_v4 = tfquery.tfstate_v3_migration.upgrade_v3_tfstate(tfstate)

🖲️ Command-Line (tfquery)

TFquery is also available as a CLI tool. It can be used to run SQL queries directly on Terraform states, and for importing resources into persistent storage.

[email protected]$> tfquery -h

usage: tfquery [-h] [--tfstate TFSTATE] [--tfstate-dir TFSTATE_DIR]
                      [--query QUERY] [--db DB_PATH] [--interactive] [--import]

tfquery: Run SQL queries on your Terraform infrastructure.

optional arguments:
  -h, --help            show this help message and exit
  --tfstate TFSTATE     Terraform .tfstate file.
  --tfstate-dir TFSTATE_DIR
                        Directory of Terraform .tfstate files, for running queries on
                        environments.
  --query QUERY, -q QUERY
                        SQL query to execute.
  --db DB_PATH          DB path (optional. default: temporarily-generated database).
  --interactive, -i     Interactive mode.
  --import              Import tfstate into database.

Examples

  • Run SQL query for a directory of multiple Terraform states (for multiple workspaces).:
$ tfquery -q 'select count(*) as count from resources;'  --tfstate-dir /path/to/terraform-states
[i] DB Path: tfstate.db
[+] Imported 4203 resources from ./prod.tfstate.
[i] DB Path: tfstate.db
[+] Imported 3675 resources from ./nonprod.tfstate.
[i] DB Path: tfstate.db
[+] Imported 463 resources from ./qa.tfstate.
  • Import Terraform states into Database.:
$ python3 tfquery --tfstate /path/to/terraform.state --db tfstate.db --import
[i] DB Path: tfstate.db
[+] Imported 386 resources from terraform.tfstate.
  • Run queries on imported resources in a database
$ tfquery --db tfstate.db -q 'select count(*) as count from resources;'
[
    {
        "count": 386
    }
]

💭 Awesome Queries & Scripts

Find all AWS S3 buckets without versioning being enabled

import tfquery, sys
results = tfquery.tfstate.run_query(sys.argv[1], "select * from resources where type = 'aws_s3_bucket'")
for result in results:
    attributes = result["attributes"]
    if 'versioning' not in attributes or len(attributes["versioning"]) == 0:
        # print(result)
        continue
    for versioning in attributes["versioning"]:
        if versioning["enabled"] is False:
            # print(result)
            pass

Find all AWS IAM users, and print their ARNs

import tfquery, sys
results = tfquery.tfstate.run_query(sys.argv[1], "select json_extract(attributes, '$.arn') as arn from resources where type = 'aws_iam_user';")
for result in results:
    print(result["arn"])

or

import tfquery, sys
results = tfquery.tfstate.run_query(sys.argv[1], "select attributes from resources where type = 'aws_iam_user';")
for result in results:
    print(result["attributes"]["arn"])

Find all resources in the environment, and show how many instances were deployed

import tfquery
results = tfquery.tfstate.run_query("terraform.tfstate", "select type, count(*) as count from resources group by type order BY count desc;")
print(results)

Interested in tfquery?

  1. Post a Tweet about the project and tag @mazen160 🙏

  2. 🌟 Star it on Github 🌟

  3. Create a PR for a new awesome feature 💛

  4. Would like to sponsor the project? Contact me on email!


💻 Contribution

Contribution is always welcome! Please feel free to report issues on Github and create PRs for new features.

📌 Ideas to Start on

Would like to contribute to tfquery? Here are some ideas that you may start with:

  • Better documentation: would be great to enhance the documentation with additional examples and queries.

  • CI: Implement CI along with test terraform states for Terragoat.

  • Support dependencies for resources lookup: Create a new table called "dependencies", parse V4 Terraform states, and implement a many-to-one relation for dependencies of resources.

  • More V3 --> V4 migration support: currently V3 resources migrations are supported. Dependencies are not migrated to the new V4 state. It will be great to continue on V3--> V4 support for Terraform states.

  • General validation of Terraform states parser implementation: Validate current implementation of the parser, and enhance it where possible.

  • Connect resources with terraform state base name: For environments with many workspaces, each workspace can have a different name, it would be nice to add a column for terraform state file base name, to help in querying across different workspaces.

  • tfplan parsing: Allow parsing of tfplan files. This can be an opening addition for implementing a new CI security scanner for Terraform deployments.

  • Logo design: a logo design would be great.

  • Web interface representation with coleifer/sqlite-web - Thanks @securityfu for the idea!

  • Add a managed repository of pre-defined queries to enable teams to be able to query Terraform for different compliance and security controls.

As you can see, there are many ways to support. Please help us make the project bigger for everyone!


Installation

[email protected]$> git clone https://github.com/mazen160/tfquery.git
[email protected]$> cd tfquery
[email protected]$> python3 setup.py install

or

[email protected]$> pip install git+https://github.com/mazen160/tfquery

📄 License

The project is licensed under MIT License.

💚 Author

Mazin Ahmed

You might also like...
Using GNU Radio and HackRF One to Receive, Analyze and Send ASK/OOK signals

play_with_ask NIS-8016 Lab A code: Recv.grc/py: Receive signals and match with ASK button using HackRF and GNU radio. I use AM demod block(can also in

Tools for use in DeFi. Impermanent Loss calculations, staking and farming strategies, coingecko and pancakeswap API queries, liquidity pools and more
Tools for use in DeFi. Impermanent Loss calculations, staking and farming strategies, coingecko and pancakeswap API queries, liquidity pools and more

DeFi open source tools Get Started Instalation General Tools Impermanent Loss, simple calculation Compare Buy & Hold with Staking and Farming Complete

Troposphere and shellscript based AWS infrastructure automation creates an awsapigateway lambda with a go backend

Automated-cloudformation-infra Troposphere and shellscript based AWS infrastructure automation. Feel free to clone and edit for personal usage. The en

Infrastructure template and Jupyter notebooks for running RoseTTAFold on AWS Batch.
Infrastructure template and Jupyter notebooks for running RoseTTAFold on AWS Batch.

AWS RoseTTAFold Infrastructure template and Jupyter notebooks for running RoseTTAFold on AWS Batch. Overview Proteins are large biomolecules that play

AWS DeepRacer Free Student Workshop: Run faster by using your custom waypoints

AWS DeepRacer Free Student Workshop: Run faster by using your custom waypoints Reward Function Template for waypoints def reward_function(params):

Draw your telegram bot in draw.io and generate its code
Draw your telegram bot in draw.io and generate its code

README Contents: Draw your bot Install requirements Registering a telegram bot Draw bot Start point Message block Single choice block Functions block

Build better AWS infrastructure

Sceptre About Sceptre is a tool to drive AWS CloudFormation. It automates the mundane, repetitive and error-prone tasks, enabling you to concentrate o

AWS Auto Inventory allows you to quickly and easily generate inventory reports of your AWS resources.
AWS Auto Inventory allows you to quickly and easily generate inventory reports of your AWS resources.

Photo by Denny Müller on Unsplash AWS Automated Inventory ( aws-auto-inventory ) Automates creation of detailed inventories from AWS resources. Table

ANKIT-OS/STYLISH-TEXT is a special repository. Its Is A Telegram Bot Which Can Translate Your Text Into 100+ Language
ANKIT-OS/STYLISH-TEXT is a special repository. Its Is A Telegram Bot Which Can Translate Your Text Into 100+ Language

🔥 ᴳᴼᴼᴳᴸᴱ⁻ᵀᴿᴬᴺᔆᴸᴬᵀᴱᴿ 🔥 The owner would not be responsible for any kind of bans due to the bot. • ⚡ INSTALLING ⚡ • • 🛠️ Lᴀɴɢᴜᴀɢᴇs Aɴᴅ Tᴏᴏʟs 🔰 • If

Comments
  • Invalid terraform state error

    Invalid terraform state error

    Hi 👋 , thanks for doing this.

    I got this error reporting invalid terraform state file:

     9, in validate_tfstate
        raise ValueError("Invalid tfstate file")
    ValueError: Invalid tfstate file
    

    and the terraform state file:

    {
        "version": 3,
        "serial": 1,
        "lineage": "<uuid>",
        "backend": {
            "type": "s3",
            "config": {
                "access_key": null,
                "acl": "private",
                "assume_role_policy": null,
    

    my understanding is the error comes from the validation expecting a key named "terraform_version" instead of "version". However every reference after that seems to be looking for "version", e.g:

    def validate_tfstate(tfstate):
        if "terraform_version" not in tfstate.keys():
            raise ValueError("Invalid tfstate file")
            return False
        if tfstate["version"] < 3:
            raise ValueError("Unsupported tfstate version")
            return False
        return True
    

    Do you think this could be a version issue and should accept both?

    opened by calvernaz 2
  • run_query does not initialise SQLHandler correctly

    run_query does not initialise SQLHandler correctly

    This looks like a really cool project that might help us answer some really interesting questions, thanks for starting it! If we end up using a lot I will endeavour to contribute.

    With tfquery installed from both pypi and git, running the below:

    import tfquery
    
    def main():
        state_path = "terraform.tfstate"
        result = tfquery.tfstate.run_query(state_path, "select count(*) from resources")
        print(result)
    
    if __name__ == "__main__":
        main()
    

    was giving me this error:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/ben/scratch/tfquery/venv/lib/python3.8/site-packages/tfquery/tfstate.py", line 114, in run_query
        s.insert_resources(resources)
      File "/home/ben/scratch/tfquery/venv/lib/python3.8/site-packages/tfquery/sql_handler.py", line 85, in insert_resources
        self.insert_resource(resource)
      File "/home/ben/scratch/tfquery/venv/lib/python3.8/site-packages/tfquery/sql_handler.py", line 99, in insert_resource
        resource["tfstate_file"] = os.path.basename(self.tfstate_file)
      File "/usr/lib/python3.8/posixpath.py", line 142, in basename
        p = os.fspath(p)
    TypeError: expected str, bytes or os.PathLike object, not NoneType
    

    It looks like the line that initialises SQLHandler in run_query should pass the tfstate_file like so:

        s = SQLHandler(in_memory=True, tfstate_file=tfstate_file)
    

    Having made that change locally I can then run queries as expected.

    opened by benabineri 1
  • Tfplan support

    Tfplan support

    This PR introduces support for tfplan parsing. This should allow running SQL queries on Infra changing within Terraform.

    ToDo

    • [x] Update Readme and add examples
    • [x] Add sample data
    opened by mazen160 0
Releases(0.1)
Owner
Mazin Ahmed
Cyber Security Engineer (Offensive Security)
Mazin Ahmed
Aria/qBittorrent Telegram mirror/leech bot.

Missneha Mirror Leech Bot Aria/qBittorrent Telegram mirror/leech bot. missneha Mirror Leech Bot is a multipurpose Telegram Bot written in Python for m

ACHAL 6 Sep 30, 2022
Python Library for Accessing the Cohere API

Cohere Python SDK This package provides functionality developed to simplify interfacing with the Cohere API in Python 3. Documentation See the API's d

cohere.ai 42 Jan 03, 2023
An powerfull telegram group management anime themed bot.

ErzaScarlet Erza Scarlet is the female deuteragonist of the anime/manga series Fairy Tail. She is an S-class Mage from the Guild Fairy Tail. Like most

ꜱōʜᴇʀᴜ ᴋāɴ (AKA) ꜱᴏʜᴀɪʟ ᴋʜᴀɴ 2 May 19, 2022
Probably Overengineered Unimore Booker

POUB Probably Overengineered Unimore Booker A python-powered, actor-based, telegram-facing, timetable-aware booker for unimore (if you know more adjec

Lorenzo Rossi 3 Feb 20, 2022
An hcaptcha-solving discord account generator; capable of randomizing names, profile pictures, and verifying phone numbers.

discord-account-generator An hcaptcha-solving discord account generator; capable of randomizing names, profile pictures, and verifying phone numbers.

Acier 61 Dec 10, 2022
MemeBot - A discord bot that tracks how good people's memes are

MemeBot A discord Meme "Karma" Tracking bot Dependancies Make sure you have pymongo installed and a mongodb cluster setup with two collections. pip in

Uday Sharma 3 Aug 10, 2022
A Python interface between Earth Engine and xarray for processing weather and climate data

wxee What is wxee? wxee was built to make processing gridded, mesoscale time series weather and climate data quick and easy by integrating the data ca

Aaron Zuspan 160 Dec 31, 2022
:electric_plug: Generating short urls with python has never been easier

pyshorteners A simple URL shortening API wrapper Python library. Installing pip install pyshorteners Documentation https://pyshorteners.readthedocs.i

Ellison 351 Jan 03, 2023
A stock information collector and parser for Taiwan and US market. Automatically send LINE message if the pre-defined rules are triggered.

agastock 開發動機 就在海運飆漲的2021年7月,差點跪在地上喜迎財富自由的當下,EPS超高好消息不斷的長榮竟然套在202元一去不回,有圖有真相(哭) 忽然體會到追高殺低不是辦法,魯蛇我得靠邏輯分析也能出頭天,經過三個月無數個不出門的周末,產出簡單的爬蟲和分析工具。 上過金融研訓院的量化交易

Gavin Lee 12 Nov 16, 2022
A Telegram Bot for searching any channel messages from Inline by @AbirHasan2005

Message-Search-Bot A Telegram Bot for searching any channel messages from Inline by @AbirHasan2005. I made this for @AHListBot. You can use this for s

Abir Hasan 44 Dec 27, 2022
NFT Generator - A NFT Generator created using Python

NFT_Generator v1 An NFT Generator created using Python. This NFT Generation tool

3 Dec 02, 2022
a script to bulk check usernames on multiple site. includes proxy & threading support.

linked-bulk-checker bulk checks username availability on multiple sites info people have been selling these so i just made one to release dm my discor

krul 9 Sep 20, 2021
A simple message content sniping Discord bot which you can run yourself! Sniping API pulled from isobot and Arch bot

Discord Snipe Bot This is a bot made with the same message content sniping API from isobot and Arch bot. It's default prefix is -, however you can als

notsniped 5 Aug 11, 2022
bot for hearthstone mercenaries

Hearthstone-Mercenaries-game-bot - prevention: Bot is not ready and now on the development stage estimated release date - 21.10.21 The main idea of th

Andrew Efimov 59 Dec 12, 2022
An incomplete add-on extension to Pyrogram, to create telegram bots a bit more easily

PyStark A star ⭐ from you means a lot An incomplete add-on extension to Pyrogram

Stark Bots 36 Dec 23, 2022
The most fresh and updateable Telegram userbot. By one of the most active contibutors to GeekTG

Installation Script installation: Simply run this command out of root: . (wget -qO- http://gg.gg/get_hikka) Manual installation: apt update && apt in

Dan Gazizullin 150 Jan 04, 2023
Python script for download course from platzi.com

Platzi Downloader Tool Esta es una pequeña herramienta que hace mucho y que te ahorra una gran cantidad de trabajo a la hora de descargar cursos de Pl

Devil64-Dev 21 Sep 22, 2022
A simple python discord bot with commands for moderation and utility.

Discord Bot A simple python discord bot with commands for moderation, utility and fun. Moderation $kick user reason - Kick a user from the server

syn 3 Feb 06, 2022
doi, pubmed, arxiv.org的查询服务API接口,部署于vercel云函数

article-search-service doi, pubmed, arxiv.org的查询服务API接口,部署于vercel云函数 云函数 vercel,国内可能被qiang了。 DOI接口 POST https://article-search-service.vercel.app/api/

HyokaChen 2 Oct 10, 2021
D(HE)ater is a security tool can perform DoS attack by enforcing the DHE key exchange.

D(HE)ater D(HE)ater is an attacking tool based on CPU heating in that it forces the ephemeral variant of Diffie-Hellman key exchange (DHE) in given cr

Balasys 138 Dec 15, 2022