Build capture utility for Linux

Overview

CX-BUILD

Compilation Database alternative

Build

Prerequisite

the CXBUILD uses linux system call trace utility called strace which was customized. So If you want use the cxbuild, you have to build the custom strace(It prints whole path to recognize build commands, https://github.com/damho1104/strace.git)

mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=/usr/local ..
make -j8

When after you build 'cstrace', please copy cstrace binary into '/usr/bin' or '/bin' directory(or add $PATH environement variable) to make the cxbuild freely invokes.

Export single portable binary

Sometimes python and python module requirements is annoying. the pyinstaller can generate everything in single binary. It is really helpful in many cases.

To do this, Required Python 3.x(we tested 3.6.9) and above version. when you ready to build the cxbuild binary, at the source root directory, type below:

python3 -mvenv python
source python/bin/activate
pip install wheel
pip install -r requirements.txt

pip install pyinstaller
pyinstaller build.spec
chmod +x dist/cxbuild
cp dist/cxbuild /usr/local/bin

or Just use cxbuild with python interpreter like this:

source env/bin/activate
pip install -r requirements.txt
python cxbuild.py capture make -j 8

Usage

Capture command

When you ready build your repository, you can use 'capture' command, to make compilation database

cxbuild capture [build command here]

If succeed, "[pwd]/.xdb/compile-commands.json" file will be generated(some intermediate file exist).

Captured command

You can run post process only after build capture completed, It only required when debug something.

The 'captured' command assumes that there exists trace files(full_trace.log) in .xdb directory. and This command run only post process phase, It will generate compile_commands.json file from full_trace.log.

cxbuild captured

Try it

Assumes that 'cstrace' and 'cxbuild' exist in $PATH environment

/bin/cstrace
/bin/cxbuild

gzip

Prepare source build (in Ubuntu OS, apt source command will serve source files to build):

$ apt source gzip
$ls -al
total 1084
drwxr-xr-x  3 minhyuk minhyuk    4096 Nov  1 10:51 .
drwxr-xr-x 12 minhyuk minhyuk    4096 Nov  1 10:51 ..
drwxr-xr-x 11 minhyuk minhyuk    4096 Nov  1 10:52 gzip-1.6
-rw-r--r--  1 minhyuk minhyuk   15604 Jul  8 01:53 gzip_1.6-5ubuntu1.1.debian.tar.xz
-rw-r--r--  1 minhyuk minhyuk    2060 Jul  8 01:53 gzip_1.6-5ubuntu1.1.dsc
-rw-r--r--  1 minhyuk minhyuk 1074924 Aug 21  2013 gzip_1.6.orig.tar.gz

then prepare build tools

$ cd gzip-1.6
$ ./configure

actual build will with cxbuild just type like this:

Making all in tests make[2]: Entering directory '/home/minhyuk/cx/tests/gzip-1.6/tests' make[2]: Nothing to be done for 'all'. make[2]: Leaving directory '/home/minhyuk/cx/tests/gzip-1.6/tests' make[1]: Leaving directory '/home/minhyuk/cx/tests/gzip-1.6' -- [Analyzing build Activities] /home/minhyuk/cx/tests/gzip-1.6/.xdb/xtrace_tree.json > Finished(00:00:09) ">
$ cxbuild capture make -j 8
> eXtension of Compilation Database
-- [Build]
make -j 8
cstrace -s 65535 -e trace=open,openat,execve -v -y -f -o "/home/minhyuk/cx/tests/gzip-1.6/.xdb/full_cstrace.log" -- make -j 8
  GEN      version.c
  GEN      version.h
make  all-recursive
make[1]: Entering directory '/home/minhyuk/cx/tests/gzip-1.6'
Making all in lib
make[2]: Entering directory '/home/minhyuk/cx/tests/gzip-1.6/lib'
  GEN      alloca.h
  GEN      configmake.h
  GEN      c++defs.h
  GEN      arg-nonnull.h
  GEN      warn-on-use.h
  GEN      unused-parameter.h
<...skipped...>
Making all in tests
make[2]: Entering directory '/home/minhyuk/cx/tests/gzip-1.6/tests'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/minhyuk/cx/tests/gzip-1.6/tests'
make[1]: Leaving directory '/home/minhyuk/cx/tests/gzip-1.6'
-- [Analyzing build Activities]
/home/minhyuk/cx/tests/gzip-1.6/.xdb/xtrace_tree.json
> Finished(00:00:09)

Check compile_commands.json:

] ">
$ cat .xdb/compile_commands.json
[
   {
       "directory": "/home/minhyuk/cx/tests/gzip-1.6/lib",
       "command": "gcc -D HAVE_CONFIG_H -I /home/minhyuk/cx/tests/gzip-1.6/lib -g -O2 -MT /home/minhyuk/cx/tests/gzip-1.6/lib/c-strcasecmp.o -MD -MP -MF .deps/c-strcasecmp.Tpo -c -o /home/minhyuk/cx/tests/gzip-1.6/lib/c-strcasecmp.o /home/minhyuk/cx/tests/gzip-1.6/lib/c-strcasecmp.c",
       "file": "c-strcasecmp.c"
   },
   {
       "directory": "/home/minhyuk/cx/tests/gzip-1.6/lib",
       "command": "gcc -D HAVE_CONFIG_H -I /home/minhyuk/cx/tests/gzip-1.6/lib -g -O2 -MT /home/minhyuk/cx/tests/gzip-1.6/lib/c-ctype.o -MD -MP -MF .deps/c-ctype.Tpo -c -o /home/minhyuk/cx/tests/gzip-1.6/lib/c-ctype.o /home/minhyuk/cx/tests/gzip-1.6/lib/c-ctype.c",
       "file": "c-ctype.c"
   },
   {
       "directory": "/home/minhyuk/cx/tests/gzip-1.6/lib",
       "command": "gcc -D HAVE_CONFIG_H -I /home/minhyuk/cx/tests/gzip-1.6/lib -g -O2 -MT /home/minhyuk/cx/tests/gzip-1.6/lib/cloexec.o -MD -MP -MF .deps/cloexec.Tpo -c -o /home/minhyuk/cx/tests/gzip-1.6/lib/cloexec.o /home/minhyuk/cx/tests/gzip-1.6/lib/cloexec.c",
       "file": "cloexec.c"
   },
   {
       "directory": "/home/minhyuk/cx/tests/gzip-1.6/lib",
       "command": "gcc -D HAVE_CONFIG_H -I /home/minhyuk/cx/tests/gzip-1.6/lib -g -O2 -MT /home/minhyuk/cx/tests/gzip-1.6/lib/c-strncasecmp.o -MD -MP -MF .deps/c-strncasecmp.Tpo -c -o /home/minhyuk/cx/tests/gzip-1.6/lib/c-strncasecmp.o /home/minhyuk/cx/tests/gzip-1.6/lib/c-strncasecmp.c",
       "file": "c-strncasecmp.c"
   },
   <...skipped...>
]
You might also like...
MongoDB utility to inflate the contents of small collection to a new larger collection

MongoDB Data Inflater ("data-inflater") The data-inflater tool is a MongoDB utility to automate the creation of a new large database collection using

A utility tool to create .env files

A utility tool to create .env files dump-env takes an .env.template file and some optional environmental variables to create a new .env file from thes

Yet another retry utility in Python

Yet another retry utility in Python, avereno being the Malagasy word for retry.

The git for the Python Story Utility Package library.

SUP The git for the Python Story Utility Package library. Installation: Install SUP by simply running pip install psup in your terminal. Check out our

A utility that makes it easy to work with Python projects containing lots of packages, of which you only want to develop some.

Mixed development source packages on top of stable constraints using pip mxdev [mɪks dɛv] is a utility that makes it easy to work with Python projects

A collection of utility functions to prototype geometry processing research in python

gpytoolbox This repo is a work in progress and contains general utility functions I have needed to code while trying to work on geometry process resea

A morse code encoder and decoder utility.

morsedecode A morse code encoder and decoder utility. Installation Install it via pip: pip install morsedecode Alternatively, you can use pipx to run

Utility to add/remove licenses to/from source files
Utility to add/remove licenses to/from source files

Utility to add/remove licenses to/from source files. Supports processing any combination of globs, files, and directories (recurse). Pruning options allow skipping non-licensing files.

EVE-NG tools, A Utility to make operations with EVE-NG more friendly.
EVE-NG tools, A Utility to make operations with EVE-NG more friendly.

EVE-NG tools, A Utility to make operations with EVE-NG more friendly. Also it support different snapshot operations with same style as Libvirt/KVM

Comments
  • __test_dummy_source__.o is generated at cwd

    __test_dummy_source__.o is generated at cwd

    When you capture a program, __test_dummy_source__.o is generated at cwd.

    $ cxbuild capture gcc -o test test.c
    
    $ ls
    __test_dummy_source__.o test test.c
    
    

    This is not intended behavior. This should be fixed.

    bug 
    opened by HansolChoe 0
  • bug: compile_commands.json may have duplicate data

    bug: compile_commands.json may have duplicate data

    compile_commands.json may have duplicate data(not entirely duplicated because command is slightly different, but compilation target file is same) like below:

    [
        {
            "directory": "/home/workspace/src",
            "command": "gcc -D HAVE_CONFIG_H -I /home/workspace/src -g -O2 -std=gnu99 -Wall -Wextra -Wdeclaration-after-statement -Wpointer-arith -funsigned-char -D _FORTIFY_SOURCE=2 -Wcast-align -Wcast-qual -Wshadow -Wbad-function-cast -Wwrite-strings -Wundef -Wuninitialized -Winit-self -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Waggregate-return -pipe -MT /home/workspace/src/g72x.lo -MD -MP -MF .deps/g72x.Tpo -c -fPIC -D PIC -o /home/workspace/src/.libs/g72x.o -I /usr/lib/gcc/x86_64-linux-gnu/7/include -I /usr/local/include -I /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -I /usr/include/x86_64-linux-gnu -I /usr/include /home/workspace/src/g72x.c",
            "file": "/home/workspace/src/g72x.c"
        },
       ...
        {
            "directory": "/home/workspace/src",
            "command": "gcc -D HAVE_CONFIG_H -I /home/workspace/src -g -O2 -std=gnu99 -Wall -Wextra -Wdeclaration-after-statement -Wpointer-arith -funsigned-char -D _FORTIFY_SOURCE=2 -Wcast-align -Wcast-qual -Wshadow -Wbad-function-cast -Wwrite-strings -Wundef -Wuninitialized -Winit-self -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Waggregate-return -pipe -MT /home/workspace/src/g72x.lo -MD -MP -MF .deps/g72x.Tpo -c -o /home/workspace/src/g72x.o -I /usr/lib/gcc/x86_64-linux-gnu/7/include -I /usr/local/include -I /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -I /usr/include/x86_64-linux-gnu -I /usr/include /home/workspace/src/g72x.c",
            "file": "/home/workspace/src/g72x.c"
        },
      ...
    ]
    

    The command below does not have -fPIC -D PIC but other data are same. This is confusing.

    This symptom can be reproduced by cxbuild captured command at root of this file(fixed.1.zip). It is normal that .xdb/artifacts.zip is empty.

    bug 
    opened by HansolChoe 0
  • Some paths are not captured properly (not absoulte path)

    Some paths are not captured properly (not absoulte path)

    In some cases, paths in projects.json are not recorded in absolute path.

    If you capture the build information of gzip-1.11, you may see logs like below:

    $ ./configure && cxbuild capture make -j10
    ...
    > eXtension of Compilation Database
    -- [Analyzing build Activities]
    -- [Build trace preprocessed]
    -- [project.json written [/home/vagrant/workspace/gzip-1.11/.xdb/project.json]]
    -- [compile_commands.json written [/home/vagrant/workspace/gzip-1.11/.xdb/compile_commands.json]]
      > All dependency should start with '//'. Ignore 'threadlib.c'.
      > All dependency should start with '//'. Ignore 'lock.c'.
    -- [artifacts.zip written [/home/vagrant/workspace/gzip-1.11/.xdb/artifacts.zip]]
    

    threadlib.c and lock.c are not absolute path. In this case, cxbuild is not killed, but it prints warning messages and ignores the two files.

    These files may be okay to ignore, but we need to know when these things happen.

    bug 
    opened by HansolChoe 1
Releases(dpp)
  • dpp(Nov 3, 2022)

    Commits

    • clang-11 no such compiler error (vulcan-action)
    • artifact.zip except for library(.so etc) file (vulcan-action)
    • 6c466f1: Fix get_system_include_list and get_predefined_macro (HansolChoe)
    • 48bf810: Fix _ArtifactBuilder to use the function in cslib (HansolChoe)
    • f7ef04f: Fix _gnu_compiler_tool's compiler setting and pytest (HansolChoe)
    • b975946: Remove is_debug_mode call csutil (HansolChoe)
    • 1e0b592: Fixed _ArtifactBuilder to collect source or header file (Yeongcheol Kim)
    • 51a75a5: Revised to not remove artifacts directory (Yeongcheol Kim)
    • 1eb568b: Revised to check CXBUILD_OUTPUT_DIR environment variable and set as working_dir (Yeongcheol Kim)
    • afd9e0b: cc를 compiler로 인식하도록 패턴 추가 (HansolChoe)
    • 72f8b8e: cc를 command로 인식하도록 추가 (HansolChoe)
    Source code(tar.gz)
    Source code(zip)
    Release-cxbuild-Ubuntu-18.04.tar.gz(10.29 MB)
Owner
GLaDOS (G? L? Automatic Debug Operation System)
GLaDOS (G? L? Automatic Debug Operation System)
Networkx with neo4j back-end

Dump networkx graph into nodes/relations TSV from neo4jnx.tsv import graph_to_tsv g = pklload('indranet_dir_graph.pkl') graph_to_tsv(g, 'docker/nodes.

Benjamin M. Gyori 1 Oct 27, 2021
This repository contains some utilities for playing with PKINIT and certificates.

PKINIT tools This repository contains some utilities for playing with PKINIT and certificates. The tools are built on minikerberos and impacket. Accom

Dirk-jan 395 Dec 27, 2022
Search, generate & deliver Msfvenom payloads in an quick and easy way

Goal Search, generate & deliver payloads in an quick and easy way Be as simple as possible BUT with all msfvenom payloads. Ever lost time searching th

2 Mar 03, 2022
A plugin to simplify creating multi-page Dash apps

Multi-Page Dash App Plugin A plugin to simplify creating multi-page Dash apps. This is a preview of functionality that will of Dash 2.1. Background Th

Plotly 19 Dec 09, 2022
Python module and its web equivalent, to hide text within text by manipulating bits

cacherdutexte.github.io This project contains : Python modules (binary and decimal system 6) with a dedicated tkinter program to use it. A web version

2 Sep 04, 2022
Astvuln is a simple AST scanner which recursively scans a directory, parses each file as AST and runs specified method.

Astvuln Astvuln is a simple AST scanner which recursively scans a directory, parses each file as AST and runs specified method. Some search methods ar

Bitstamp Security 7 May 29, 2022
🍰 ConnectMP - An easy and efficient way to share data between Processes in Python.

ConnectMP - Taking Multi-Process Data Sharing to the moon 🚀 Contribute · Community · Documentation 🎫 Introduction : 🍤 ConnectMP is the easiest and

Aiden Ellis 1 Dec 24, 2021
Course-parsing - Parsing Course Info for NIT Kurukshetra

Parsing Course Info for NIT Kurukshetra Overview This repository houses code for

Saksham Mittal 3 Feb 03, 2022
Python program for analyzing the output files of phonopy.

PhononTools Description Python program to analyze the results generated by phonopy. Using the .yaml and .dat files that phonopy generates one can plot

Harry LaBollita 8 Nov 27, 2022
cpp20.py is a Python script to compile C++20 code using modules.

cpp20.py is a Python script to compile C++20 code using modules. It browses the source files to determine their dependencies. Then, it compiles then in order using the correct flags.

Julien VERNAY 6 Aug 26, 2022
Shut is an opinionated tool to simplify publishing pure Python packages.

Welcome to Shut Shut is an opinionated tool to simplify publishing pure Python packages. What can Shut do for you? Generate setup files (setup.py, MAN

Niklas Rosenstein 6 Nov 18, 2022
A monitor than send discord webhook when a specific monitored product has stock in your nearby pickup stores.

Welcome to Apple In-store Monitor This is a monitor that are not fully scaled, and might still have some bugs.

5 Jun 16, 2022
Daiho Tool is a Script Gathering for Windows/Linux systems written in Python.

Daiho is a Script Developed with Python3. It gathers a total of 22 Discord tools (including a RAT, a Raid Tool, a Nuker Tool, a Token Grabberr, etc). It has a pleasant and intuitive interface to faci

AstraaDev 32 Jan 05, 2023
Blender 2.93 addon for loading Quake II MD2 files

io_mesh_md2 is a Blender 2.93 addon for importing Quake II MD2 files.

Joshua Skelton 11 Aug 31, 2022
Obsidian tools - a Python package for analysing an Obsidian.md vault

obsidiantools is a Python package for getting structured metadata about your Obsidian.md notes and analysing your vault.

Mark Farragher 153 Jan 04, 2023
Local backup made easy, with Python and shutil

KTBackup BETA Local backup made easy, with Python and shutil Features One-command backup and restore Minimalistic (only using stdlib) Convenient direc

kelptaken 1 Dec 27, 2021
Go through a random file in your favourite open source projects!

Random Source Codes Never be bored again! Staring at your screen and just scrolling the great world wide web? Would you rather read through some code

Mridul Seth 1 Nov 03, 2022
A simple language and reference decompiler/compiler for MHW THK Files

Leviathon A simple language and reference decompiler/compiler for MHW THK Files. Project Goals The project aims to define a language specification for

11 Jan 07, 2023
Experimental python optimistic rollup fraud-proof generation

Macula Experimental python optimistic rollup fraud-proof generation tech by @protolambda. Working on a python version for brevity and simplicity. See

Diederik Loerakker 30 Sep 01, 2022
Handy Tool to check the availability of onion site and to extract the title of submitted onion links.

This tool helps is to quickly investigate a huge set of onion sites based by checking its availability which helps to filter out the inactive sites and collect the site title that might helps us to c

Balaji 13 Nov 25, 2022