Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Overview

Ethereum ETL

Build Status Join the chat at https://gitter.im/ethereum-eth Telegram Discord

Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Do you just want to query Ethereum data right away? Use the public dataset in BigQuery.

Full documentation available here.

Quickstart

Install Ethereum ETL:

pip3 install ethereum-etl

Export blocks and transactions (Schema, Reference):

> ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c

Export ERC20 and ERC721 transfers (Schema, Reference):

> ethereumetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv

Export traces (Schema, Reference):

> ethereumetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv

Stream blocks, transactions, logs, token_transfers continually to console (Reference):

> pip3 install ethereum-etl[streaming]
> ethereumetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c

Find other commands here.

For the latest version, check out the repo and call

> pip3 install -e . 
> python3 ethereumetl.py

Useful Links

Running Tests

> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> pytest -vv

Running Tox Tests

> pip3 install tox
> tox

Running in Docker

  1. Install Docker https://docs.docker.com/install/

  2. Build a docker image

     > docker build -t ethereum-etl:latest .
     > docker image ls
    
  3. Run a container out of the image

     > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io
     > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://mainnet.infura.io
    
  4. Run streaming to console or Pub/Sub

     > docker build -t ethereum-etl:latest -f Dockerfile .
     > echo "Stream to console"
     > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt
     > echo "Stream to Pub/Sub"
     > docker run -v /path_to_credentials_file/:/ethereum-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/ethereum-etl/credentials_file.json ethereum-etl:latest stream --start-block 500000 --output projects//topics/crypto_ethereum
    

Projects using Ethereum ETL

  • Google - Public BigQuery Ethereum datasets
  • Nansen by D5 - Analytics platform for Ethereum
Comments
  • Add support for internal transactions

    Add support for internal transactions

    Parity provides APIs for retrieving internal transactions. trace_filter API allows retrieving traces for a range of blocks. This method is already supported by web3.py parity module.

    The scope of this task is to

    1. Design the model (the field names from this PR can be used as a basis: https://github.com/blockchain-etl/ethereum-etl/pull/72).
    2. Implement the export job (the job implementation will mostly resemble the flow in ExportTokenTransfersJob.
    3. Write unit tests (an example can be found here)
    4. Add documentation to README (an example can be found here) (5. Add instructions for how to start parity/geth i.e. what options to use).

    =======================

    Remarks:

    • There will be another task to add the BigQuery schema here
    • Geth also provides support for tracing transactions. There will be another task for implementing internal transactions exporter job for Geth. https://github.com/ethereum/go-ethereum/pull/15516, https://www.linkedin.com/pulse/geth-18-iceberg-michael-hunt-jr-
    opened by medvedev1088 21
  • Add Dockerfile

    Add Dockerfile

    The scope for this task is:

    1. Add Dockerfile (this can be used as a basis). Both ipc and rpc should be supported.
    2. Test it.
    3. Update documentation in README with instructions on how to use it.
    help wanted 
    opened by medvedev1088 15
  • Installing Ethereum-ETL

    Installing Ethereum-ETL

    Dear all,

    unfortunately I can't install ethereum-etl based on the instructions in the readme. Running "pip3 install ethereum-etl" as well as cloning the repo and running "pip3 install -e ." always results in:

    Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0b79bd0750>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/ethereum-etl/

    Could not find a version that satisfies the requirement ethereum-etl (from versions: ) No matching distribution found for ethereum-etl

    Does anybody has an idea what the problem is?

    Thanks and greets Peter

    opened by PeHo89 13
  • Add support for geth traces

    Add support for geth traces

    The scope of this task:

    1. Design the model for geth traces (trace.py can be used as a basis).
    2. Implement the export and extract jobs. 2.a The export job should use traceBlockByNumber (https://github.com/ethereum/go-ethereum/pull/15516) and output raw JSON responses. It should use callTracer, an example JS API call: debug.traceBlockByNumber('0x250252', {'tracer': 'callTracer'}); the same is available via JSON RPC. The job should use BatchWeb3Provider similar to export_blocks_job.py. 2.b The extract job should take the file with raw JSON responses, parses them, map to our domain entity and output to CSV file.
    3. Write unit tests (real mainnet responses should be used in test cases, so the same tests can be run against geth node).
    4. Add documentation to README (an example can be found here). Include instructions for how to start geth i.e. what options to use (e.g. geth --gcmode archive)

    ======================= Notes:

    • Parity traces are already supported https://github.com/blockchain-etl/ethereum-etl/pull/104.
    opened by medvedev1088 13
  • web3 upgrade

    web3 upgrade

    Here are the changes to upgrade web3.py for issue #308.

    • One major change was dropping the support for python 3.6 on ethereum etl since web3.py dropped 3.6 support
    • You may want to take a look at these notes on upgrading from web3.py v4 to v5 to see if anything jumps out at you.
    • I did still have the infura test falling locally but I think I needed to just dig in a little more and see why my setup wasn't working. If I have time tomorrow I will look at that.
    • I had to add a few of those mock response files. I copied ones you already had in there and modified them a little.
    • Also, I did not test the docker file but noticed it was using python3.6 so just changed the image it was pulling from to python3.7 hope that works.

    Let me know what you think, happy to tweak this some more if needed.

    opened by dbfreem 11
  • Windows - SyntaxError: invalid syntax

    Windows - SyntaxError: invalid syntax

    C:\Users\XPS\Documents\Python-Ethereum\ethereum-etl-master>python export_blocks_and_transactions.py --start-block 0 --end-block 500000 \ --ipc-path ~/Library/Ethereum/geth.ipc --blocks-output blocks.csv --transactions-output transactions.csv Traceback (most recent call last): File "export_blocks_and_transactions.py", line 4, in from ethereumetl.ipc import IPCWrapper File "C:\Users\XPS\Documents\Python-Ethereum\ethereum-etl-master\ethereumetl\ipc.py", line 5, in from web3.providers.ipc import get_default_ipc_path, PersistantSocket File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3_init_.py", line 8, in from web3.main import Web3 # noqa: E402 File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\main.py", line 26, in from web3.providers.eth_tester import ( File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\providers_init_.py", line 8, in from .websocket import WebsocketProvider # noqa: F401 File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\providers\websocket.py", line 9, in import websockets File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets_init_.py", line 3, in from .client import * File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\client.py", line 20, in from .protocol import WebSocketCommonProtocol File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\protocol.py", line 18, in from .compatibility import asyncio_ensure_future File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\compatibility.py", line 15 asyncio_ensure_future = asyncio.async # Python < 3.5 ^ SyntaxError: invalid syntax

    bug duplicate 
    opened by RaianeQC 11
  • adds support for non-mainnet in etl stream

    adds support for non-mainnet in etl stream

    relates #178

    DEMO on ronin network, an eth sidechain. Notice that latest block can now be read, thanks to middleware injection.

    2021-08-12 19:24:34,321 - ProgressLogger [INFO] - Started work. Items to process: 1.
    2021-08-12 19:24:34,375 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
    2021-08-12 19:24:34,375 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.054600.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Started work.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000169.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Started work.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000143.
    
    opened by numonedad 10
  • Ethereum data export error

    Ethereum data export error

    When i use “ethereumetl export_all -s 0 -e 5999999 -b 100000 -p file://$HOME/Library/Ethereum/geth.ipc -o output”,The following error occurred:“ValueError: Unknown uri scheme /datalake/blockdata/geth.ipc”,Ask what is the reason

    opened by sznariOsmosis 10
  • Add support for extracting Origin Protocol data

    Add support for extracting Origin Protocol data

    This implements a prototype that extracts data generated by Origin Protocol's open source platform for building decentralized marketplaces on Ethereum and IPFS.

    Fixes https://github.com/blockchain-etl/ethereum-etl/issues/215

    opened by franckc 10
  • ValueError: result is None in response (export_all)

    ValueError: result is None in response (export_all)

    I run it on ubuntu

    ethereumetl export_all -s 4900000 -e 4999999 -b 100000 -p https://mainnet.infura.io -o output

    but after n time it has error:

    File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 70, in _export │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 70, in _export total_items=self.end_block - self.start_block + 1 │ total_items=self.end_block - self.start_block + 1 File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 48, in execute │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 48, in execute self.executor.submit(self._fail_safe_execute, work_handler, batch) │ self.executor.submit(self._fail_safe_execute, work_handler, batch) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 31, in submit │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 31, in submit self._check_completed_futures() │ self._check_completed_futures() File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_compl│ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_comp eted_futures │leted_futures future.result() │ future.result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result │ File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() │ return self.__get_result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result │ File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception │ raise self._exception File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run │ File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) │ result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe_│ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe execute │_execute work_handler(batch) │ work_handler(batch) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in _export_batch │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in _export_batch blocks = [self.block_mapper.json_dict_to_block(result) for result in results] │ blocks = [self.block_mapper.json_dict_to_block(result) for result in results] File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in <listcomp> │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in <listcomp> blocks = [self.block_mapper.json_dict_to_block(result) for result in results] │ blocks = [self.block_mapper.json_dict_to_block(result) for result in results] File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 58, in rpc_response_batch_to_results │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 58, in rpc_response_batch_to_results yield rpc_response_to_result(response_item) │ yield rpc_response_to_result(response_item) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 67, in rpc_response_to_result │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 67, in rpc_response_to_result raise ValueError(error_message) │ raise ValueError(error_message) ValueError: result is None in response {'jsonrpc': '2.0', 'id': 0, 'error': {'code': -32603, 'message': 'request proc│ValueError: result is None in response {'jsonrpc': '2.0', 'id': 0, 'error': {'code': -32603, 'message': 'request pro essing failed'}}.

    opened by minhtrih 10
  • Error in exporting traces using Parity

    Error in exporting traces using Parity

    @medvedev1088 We used Parity to export traces. It works perfectly for almost every block. BUT error happens in some block ranges, such as 2,280,000~2,400,000 and 3800000~4,000,000. Error:web3.utils.threads.Timeout: 60 seconds image

    Our server has far more than 8 GB memory. Maybe we could change default time. But we could not figure out the correct command

    opened by cxd1995 10
  • Add decoding log functions here

    Add decoding log functions here

    https://github.com/blockchain-etl/ethereum-etl/blob/6710e6b894789d83ce3c28db79b4ec5bf2ec3845/ethereumetl/jobs/export_receipts_job.py#L78

    Add feature to decode logs in the exportation of receipts

    opened by pauline-thrackle 0
  • [Errno 101]

    [Errno 101]

    Command: ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000
    --blocks-output blocks.csv --transactions-output transactions.csv
    --provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c Error: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mainnet.infura.io', port=443): Max retries exceeded with url: /v3/7aef3f0cd1f64408b163814b22cc643c (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4b8fd63910>: Failed to establish a new connection: [Errno 101] Network is unreachable')) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mainnet.infura.io', port=443): Max retries exceeded with url: /v3/7aef3f0cd1f64408b163814b22cc643c (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4b8fd621d0>: Failed to establish a new connection: [Errno 101] Network is unreachable')) how to solve this problem?

    opened by Facebeside 0
  • M1 mac issue Symbolic Execution not available: No module named 'mythril.ether'

    M1 mac issue Symbolic Execution not available: No module named 'mythril.ether'

    System:

    • M1 Mac
    • Python 3.9.15
    • pip 22.1.2
    • Poetry (version 1.2.2)

    pyproject.toml

    [tool.poetry]
    name = "asd"
    version = "0.1.0"
    description = ""
    authors = ["Your Name <[email protected]>"]
    readme = "README.md"
    packages = [{include = "asd"}]
    
    [tool.poetry.dependencies]
    python = "^3.9"
    ethereum-etl = "^2.1.1"
    pyetherchain = "^0.3.2"
    blake2b-py = { git = "https://github.com/ethereum/blake2b-py.git", tag="v0.1.4" }
    mythril = {extras = ["mythril"], version = "^0.23.12"}
    
    
    
    [build-system]
    requires = ["poetry-core"]
    build-backend = "poetry.core.masonry.api"
    

    reprod script

    from ethereumetl.service.eth_contract_service import EthContractService
    
    bytecode_as_str = '608060405234801561001057600080fd5b506040516111893803806111898339818101604052602081101561003357600080fd5b810190808051906020019092919050505060016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000208190555080600260006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff1602179055506b033b2e3c9fd0803ce8000000600381905550600160048190555050611095806100f46000396000f3fe608060405234801561001057600080fd5b50600436106100b45760003560e01c80636924500911610071578063692450091461020d578063957aa58c146102175780639c52a7f114610235578063bf353dbb14610279578063d9638d36146102d1578063ebecb39d14610346576100b4565b80631504460f146100b95780631a0b287e146100e757806329ae81141461012957806336569e7714610161578063495d32cb146101ab57806365fae35e146101c9575b600080fd5b6100e5600480360360208110156100cf57600080fd5b810190808035906020019092919050505061039e565b005b610127600480360360608110156100fd57600080fd5b810190808035906020019092919080359060200190929190803590602001909291905050506105bd565b005b61015f6004803603604081101561013f57600080fd5b8101908080359060200190929190803590602001909291905050506107d7565b005b6101696109dc565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b6101b3610a02565b6040518082815260200191505060405180910390f35b61020b600480360360208110156101df57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610a08565b005b610215610b36565b005b61021f610c27565b6040518082815260200191505060405180910390f35b6102776004803603602081101561024b57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610c2d565b005b6102bb6004803603602081101561028f57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610d5b565b6040518082815260200191505060405180910390f35b6102fd600480360360208110156102e757600080fd5b8101908080359060200190929190505050610d73565b604051808373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020018281526020019250505060405180910390f35b61039c6004803603606081101561035c57600080fd5b810190808035906020019092919080359060200190929190803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610db7565b005b6000806001600084815260200190815260200160002060000160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff166359e02dd76040518163ffffffff1660e01b81526004016040805180830381600087803b15801561041e57600080fd5b505af1158015610432573d6000803e3d6000fd5b505050506040513d604081101561044857600080fd5b810190808051906020019092919080519060200190929190505050915091506000816104755760006104b1565b6104b061049461048c8560001c633b9aca0061100b565b600354611037565b6001600087815260200190815260200160002060010154611037565b5b9050600260009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16631a0b287e85836040518363ffffffff1660e01b815260040180838152602001807f73706f740000000000000000000000000000000000000000000000000000000081525060200182815260200192505050600060405180830381600087803b15801561055857600080fd5b505af115801561056c573d6000803e3d6000fd5b505050507fdfd7467e425a8107cfd368d159957692c25085aacbcf5228ce08f10f2146486e84848360405180848152602001838152602001828152602001935050505060405180910390a150505050565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610671576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b6001600454146106e9576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260108152602001807f53706f747465722f6e6f742d6c6976650000000000000000000000000000000081525060200191505060405180910390fd5b7f6d617400000000000000000000000000000000000000000000000000000000008214156107315780600160008581526020019081526020016000206001018190555061079f565b6040517f08c379a000000000000000000000000000000000000000000000000000000000815260040180806020018281038252601f8152602001807f53706f747465722f66696c652d756e7265636f676e697a65642d706172616d0081525060200191505060405180910390fd5b5961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a450505050565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020541461088b576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b600160045414610903576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260108152602001807f53706f747465722f6e6f742d6c6976650000000000000000000000000000000081525060200191505060405180910390fd5b7f706172000000000000000000000000000000000000000000000000000000000082141561093757806003819055506109a5565b6040517f08c379a000000000000000000000000000000000000000000000000000000000815260040180806020018281038252601f8152602001807f53706f747465722f66696c652d756e7265636f676e697a65642d706172616d0081525060200191505060405180910390fd5b5961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a4505050565b600260009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1681565b60035481565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610abc576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b60016000808373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020819055505961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a45050565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610bea576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b60006004819055505961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a450565b60045481565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610ce1576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b60008060008373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020819055505961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a45050565b60006020528060005260406000206000915090505481565b60016020528060005260406000206000915090508060000160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff16908060010154905082565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610e6b576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b600160045414610ee3576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260108152602001807f53706f747465722f6e6f742d6c6976650000000000000000000000000000000081525060200191505060405180910390fd5b7f7069700000000000000000000000000000000000000000000000000000000000821415610f6557806001600085815260200190815260200160002060000160006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff160217905550610fd3565b6040517f08c379a000000000000000000000000000000000000000000000000000000000815260040180806020018281038252601f8152602001807f53706f747465722f66696c652d756e7265636f676e697a65642d706172616d0081525060200191505060405180910390fd5b5961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a450505050565b600080821480611028575082828385029250828161102557fe5b04145b61103157600080fd5b92915050565b600081611050846b033b2e3c9fd0803ce800000061100b565b8161105757fe5b0490509291505056fea265627a7a72315820b20103aad12e7c2a10372688d31c070bc7aa268459f8bc3a58d4298c7f6bdf0964736f6c634300050c003200000000000000000000000035d1b3f3d7966a1dfe207aa4514c12a259a0492b'
    
    def main():
        contract_service = EthContractService()
        func_sigs = contract_service.get_function_sighashes(bytecode_as_str)
        print(func_sigs)
    
    
    if __name__ == '__main__':
        main()
    
    opened by 007vasy 0
  • stream command failed to extract contract information

    stream command failed to extract contract information

    Command : ethereumetl stream --start-block 15789281 -e block,transaction,log,token_transfer,contract --log-file log.txt --provider-uri https://mainnet.infura.io/v3/9aa3d95b3bc440fa88ea12eaa4456161

    Error : 2022-11-30 16:07:55,003 - root [ERROR] - An exception occurred while syncing block data. Traceback (most recent call last): File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/blockchainetl/streaming/streamer.py", line 77, in _do_stream synced_blocks = self._sync_cycle() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/blockchainetl/streaming/streamer.py", line 98, in _sync_cycle self.blockchain_streamer_adapter.export_all(self.last_synced_block + 1, target_block) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/streaming/eth_streamer_adapter.py", line 62, in export_all traces = self._export_traces(start_block, end_block) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/streaming/eth_streamer_adapter.py", line 159, in _export_traces job.run() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/blockchainetl/jobs/base_job.py", line 30, in run self._end() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/jobs/export_traces_job.py", line 106, in _end self.batch_work_executor.shutdown() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/batch_work_executor.py", line 97, in shutdown self.executor.shutdown() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/fail_safe_executor.py", line 39, in shutdown self._check_completed_futures() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures future.result() File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 437, in result return self.__get_result() File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result raise self._exception File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/batch_work_executor.py", line 63, in _fail_safe_execute work_handler(batch) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/jobs/export_traces_job.py", line 90, in _export_batch json_traces = self.web3.parity.traceBlock(block_number) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/web3/module.py", line 57, in caller result = w3.manager.request_blocking(method_str, File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/web3/manager.py", line 198, in request_blocking return self.formatted_response(response, File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/web3/manager.py", line 171, in formatted_response raise ValueError(response["error"]) ValueError: {'code': -32601, 'message': 'the method trace_block does not exist/is not available'}

    opened by infocodiste 1
  • can't support proxy contract erc20

    can't support proxy contract erc20

    I looked at the source code and found that bytecode is used to judge whether it is erc20, but many contracts currently use the proxy contract upgrade solution to deploy the contract, which makes it impossible to judge whether it is erc20

    opened by SuperEdison 2
Releases(v2.1.2)
Owner
Blockchain ETL
Facilitating data science on blockchain data. Available in Google BigQuery https://goo.gl/oY5BCQ
Blockchain ETL
A simple Ethereum mining pool

A simple getWork pool for ethereum mining

93 Oct 05, 2022
DIY gravity falls cryptograms made with python

ciphers-cryptograms some diy code to implementing ciphers-cryptograms from gravity falls with python, it's fun tho Algorithm or ciphers list Caesar At

Muhammad Asthi Seta Ari Yuwana 3 Jun 26, 2022
bitcoin-ticker is a E-ink ticker that shows usefull information about bitcoin

bitcoin-ticker bitcoin-ticker is a E-ink ticker that shows usefull information about bitcoin. Due to the limited refresh lifetime, new information is

32 Nov 09, 2022
Simple bitcoin ticker for the Pimorono Inky pHAT Red.

bitcoin-ticker Simple bitcoin ticker for the Pimorono Inky pHAT Red. Equipment Raspberry Pi Zero W v1.1 or Pi 2 model b v1.1 Pimorono Inky pHAT Red (S

2 Mar 15, 2022
A python script for AES Angecryption in Steganography

Angecryption is an encryption or an decryption result from a file to create an other file with the same / or not type.

ISIS 3 Jul 25, 2022
A Python script to implement Hill's Cipher Encryption and Decryption.

Hill_Cipher-Encryption_and_Decryption A Python script to implement Hill's Cipher Encryption and Decryption. Initially in the Encryption part, the Plai

Vishvendra Singh 1 Jan 19, 2022
Cryptocurrency application that displays instant cryptocurrency prices and reads prices with the Google Text-to-Speech library.

📈 Cryptocurrency Price App 💰 ◽ Cryptocurrency application that displays instant cryptocurrency prices and reads prices with the Google Text-to-Speec

Furkan Mert 2 Nov 08, 2021
Simple encryption/decryption utility using Pycryptodome module. Working with AES and RSA algorithms.

EncypherUtil Simple encryption/decryption utility using PyCryptodome module. Working with AES and RSA algorithms. THIS UTILITY IS NOT LICENSED AS CRYP

Egor Yakubovich 0 Jun 14, 2022
SSEPy: Implementation of searchable symmetric encryption in pure Python

SSEPy: Implementation of searchable symmetric encryption in pure Python Searchable symmetric encryption, one of the research hotspots in applied crypt

33 Dec 05, 2022
Amazing CryptoWAF was a CTF challenge for ALLES! CTF 2021

ctf-cryptowaf The AmazingCryptoWAF ™️ is used by the "noter" web app, to offer automagically military encryption for any user data. Even if an attacke

32 Jan 02, 2023
Generate a 2FA Code out of an 2FA Secret(Time-based only!)

2FA-Generator Generate a 2FA Code out of an 2FA Secret(Time-based only!) ❗ Pleas don't share this secrets/generated codes with someone. If someone kno

TheDanniCraft 1 Nov 05, 2021
BETCOIN BET is a digital currency system created with python

BETCOIN BET is a digital currency created with python and flask with features of a centralized bank, wallet system, and open transaction history of al

Ujjwal Kumar 3 Nov 16, 2021
A curated list for getting up to speed on crypto and decentralized networks

crypto reading list A curated list for getting up to speed on crypto and decentralized networks. The content on the toplevel page contains what we con

Jump Crypto 1.1k Jan 07, 2023
Decrypting winrm traffic using password/ntlm hash

Decrypting winrm traffic using password/ntlm hash

Haoxi Tan 9 Jan 05, 2022
En- and decrypting text-messages by creating a key with of the fibonacci-sequence

En- and decrypting text-messages by creating a key with of the fibonacci-sequence. This key helps to create mathematical functions, whose zeros should generates the encrypted message.

Pulsar 1 Feb 05, 2022
GreenDoge is a modern community-centric green cryptocurrency based on a proof-of-space-and-time consensus algorithm.

GreenDoge Blockchain Download GreenDoge blockchain GreenDoge is a modern community-centric green cryptocurrency based on a proof-of-space-and-time con

40 Sep 11, 2022
cryptography is a package designed to expose cryptographic primitives and recipes to Python developers.

pyca/cryptography cryptography is a package which provides cryptographic recipes and primitives to Python developers. Our goal is for it to be your "c

Python Cryptographic Authority 5.2k Dec 30, 2022
Block Chain for RiceSupply Chain and Agriculture Traceability

Block Chain for RiceSupply Chain and Agriculture Traceability Project Under Development Folder: Building a BlockChain Basic blockchain structure using

Chandru S Raghavan 3 Jan 19, 2022
Active github repos of all cryptocurrencies

This repo is to maintain the list of active repositories for all cryptocurrencies that https://codemask.org uses. The active list will be automaticall

CodeMask 5 May 20, 2022
Highly decentralized and censorship-resistant way to store key data

Beacon coin Beacon coin is a Chia singelton coin that can store data that needs to be: always available censorship resistant versioned potentially imm

Sebastjan Trepca 24 Oct 04, 2022