Is your feature request related to a problem? Please describe. Discussion and planning for syntax revision. Some ideas under discussion:

1. redundant entity type in GET

x = GET process FROM datasource WHERE [process:pid = 123]

Simplified

x = GET process FROM datasource WHERE pid = 123

2. redundant entity type in GET from variable

w = GET process FROM z WHERE [process:pid = 123]

Simplified

w = z WHERE pid = 123

3. expression

<var> [FILTER] [ AGG [ FILTER]] [SORT] [OFFSET] [LIMIT]

may use in DISP and COPY

Describe the bug Hi, I am trying to follow the tutorial from the documentation hub using an ELK stack. However, I am getting a KestrelSyntaxError when querying. I tried it with Python 3.6 and 3.9; both have the same error results.

Details of the bug

• What is the hunt flow/script you are executing? Hunt flow from the tutorial.
• What is the command that failed?

var = GET process FROM stixshifter://host101

• What is the error message?

[ERROR] KestrelSyntaxError: invalid token "" at line 1 column 24. rewrite the failed statement.

To Reproduce Steps to reproduce the behavior:

1. Setup Symon & Elasticsearch
2. Create API key on Elasticsearch for access
3. Test Elasticsearch access using API key
4. Configure environment variables

$ export STIXSHIFTER_HOST101_CONNECTOR=elastic_ecs
$ export STIXSHIFTER_HOST101_CONNECTION='{"host":"REDACTED.elastic-cloud.com", "port":9243, "indices":"winlogbeat-7.14.0-2021.08.04-000001"}'
$ export STIXSHIFTER_HOST101_CONFIG='{"auth":{"id":"REDACTED", "api_key":"REDACTED"}}'

5. Test using stix-shifter:

$ stix-shifter transmit elastic_ecs '{"host":"REDACTED.elastic-cloud.com", "port":9243, "indices":"winlogbeat-7.14.0-2021.08.04-000001"}' '{"auth":{"id":"REDACTED", "api_key":"REDACTED"}}' ping

{

    "success": true,

    "data": "{\n  \"cluster_name\" : \"66a63ad60eae4e2b9fb38f524b8defcc\",\n  \"status\" : \"green\",\n  \"timed_out\" : false,\n  \"number_of_nodes\" : 3,\n  \"number_of_data_nodes\" : 2,\n  \"active_primary_shards\" : 86,\n  \"active_shards\" : 172,\n  \"relocating_shards\" : 0,\n  \"initializing_shards\" : 0,\n  \"unassigned_shards\" : 0,\n  \"delayed_unassigned_shards\" : 0,\n  \"number_of_pending_tasks\" : 0,\n  \"number_of_in_flight_fetch\" : 0,\n  \"task_max_waiting_in_queue_millis\" : 0,\n  \"active_shards_percent_as_number\" : 100.0\n}\n"

}

6. Run jupyter notebook with command

var = GET process FROM stixshifter://host101
[ERROR] KestrelSyntaxError: invalid token "" at line 1 column 24. rewrite the failed statement.

Expected behavior Results from query

Environment (please complete the following information):

• OS: Ubuntu 20.04
• Python version: Python 3.9.5, Python 3.6.9
• Python install environment: Python virtual environment
• STIX-Shifter version: 3.5.0

Describe the bug ValueError when running Hello World Hunt using Python 3.6.9. Installed using pip install kestrel-lang.

Details of the bug

• What is the hunt flow/script you are executing? Hello World Hunt from readme
• What is the command that failed?

$ kestrel helloworld.hf

• What is the error message?

$ kestrel helloworld.hf
Traceback (most recent call last):
  File "/usr/local/bin/kestrel", line 8, in <module>
    runpy.run_module('kestrel', run_name='__main__')
  File "/usr/lib/python3.6/runpy.py", line 208, in run_module
    return _run_code(code, {}, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/kestrel/__main__.py", line 49, in <module>
    logging_setup(None, args.verbose, args.debug)
  File "/usr/local/lib/python3.6/dist-packages/kestrel/__main__.py", line 33, in logging_setup
    force=True,
  File "/usr/lib/python3.6/logging/__init__.py", line 1829, in basicConfig
    raise ValueError('Unrecognised argument(s): %s' % keys)
ValueError: Unrecognised argument(s): force

To Reproduce

1. pip install --upgrade pip setuptools wheel
2. pip install kestrel-lang
3. kestrel helloworld.hf

Expected behavior Output from hunt flow.

Environment (please complete the following information):

• OS: Ubuntu 18.04
• Python version: Python 3.6.9
• Python install environment: Python virtual environment
• STIX-Shifter version: 3.5.0

Describe the bug If one tries to auto-complete a variable name in STIX pattern for a parameterized pattern, it does not work.

Details of the bug This is limited to the STIX pattern parser we current use. Need to upgrade parser.

Describe the bug A parsing error is thrown when a file path has a space in it

Details of the bug GET process FROM file:///a/path/with/a space/in_the_name/bundle.json

Results in:

lark.exceptions.UnexpectedCharacters: No terminal matches 's' in the current parser context, .....
/a/path/with/a space/in_the_name/bundle.json`
                        ^
Expected on of:
                * WHERE

To Reproduce Try to run GET on a file:// bundle with a space in the name

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: macOS 11.6
• Python version: 3.7.7
• Python install environment:
• STIX-Shifter version: latest github develop branch.

Additional context Add any other context about the problem here.

Some analytics (regardless of which interface they use) may call third party APIs, particularly those doing threat intel enrichment. Sometimes those APIs may fail, either due to authentication issues, temporary network problems, etc. There is currently no way for the user to be notified of such problems.

There should be some way for analytics to capture such error information and report it back up. The implementation may differ per analytics interface (e.g. a native python interface where the analytics run under that same python interpreter as the core can probably just raise an exception, while the docker interface may need to write the information to a file).

When applying helloworld.hf file as a parameter to kestrel via cli, the following error message appears:

[[email protected] ~]$ kestrel helloworld.hf --debug 16:19:00 DEBUG kestrel.session Establish session with session_id: None, runtime_dir: None, store_path:None, debug_mode:True 16:19:00 DEBUG kestrel.session Configuration file /kestrel/kestrel.toml does not exist. 16:19:00 DEBUG kestrel.session Configuration file etc/kestrel/kestrel.toml does not exist. 16:19:00 DEBUG kestrel.session Configuration file /home/docker/.local/etc/kestrel/kestrel.toml loaded successfully. 16:19:00 DEBUG kestrel.session Configuration file /home/docker/.config/kestrel/kestrel.toml does not exist. 16:19:00 DEBUG kestrel.session Configuration loaded: {'session': {'local_database_path': 'local.db', 'debug_env_var_name': 'KESTREL_DEBUG'}, 'language': {'default_variable': '_', 'default_sort_order': 'desc'}, 'stixquery': {'timerange_start_offset': -300, 'timerange_stop_offset': 300, 'support_id': False}, 'prefetch': {'get': True, 'find': True, 'process_name_change_timerange_start_offset': -5, 'process_name_change_timerange_stop_offset': 5, 'process_lifespan_start_offset': -10800, 'process_lifespan_stop_offset': 10800}} 16:19:00 DEBUG kestrel.session create new session runtime_directory: /tmp/kestrel-session-212ddaa5-c492-41c7-8c1c-0639a1eb82cd. 16:19:00 DEBUG firepit.sqlitestorage Connection to SQLite DB /tmp/kestrel-session-212ddaa5-c492-41c7-8c1c-0639a1eb82cd/local.db successful 16:19:00 DEBUG firepit.sqlitestorage Executing query: CREATE TABLE IF NOT EXISTS "__symtable" (name TEXT, type TEXT, appdata TEXT); 16:19:00 DEBUG firepit.sqlitestorage Executing query: CREATE TABLE IF NOT EXISTS "__membership" (sco_id TEXT, var TEXT); 16:19:00 DEBUG firepit.sqlitestorage Executing query: CREATE TABLE IF NOT EXISTS "__queries" (sco_id TEXT, query_id TEXT); 16:19:01 DEBUG kestrel.codegen.commands Executing 'new' with statement: {'command': 'new', 'type': 'process', 'data': '[ {"name": "cmd.exe", "pid": "123"}\n , {"name": "explorer.exe", "pid": "99"}\n , {"name": "firefox.exe", "pid": "201"}\n , {"name": "chrome.exe", "pid": "205"}\n ]', 'output': 'proclist'} 16:19:01 DEBUG firepit.splitter _create_table: "CREATE TABLE "process" ("name" TEXT,"pid" TEXT,"type" TEXT,"id" TEXT UNIQUE);" 16:19:01 DEBUG firepit.sqlitestorage Executing query: CREATE TABLE "process" ("name" TEXT,"pid" TEXT,"type" TEXT,"id" TEXT UNIQUE); 16:19:01 DEBUG firepit.sqlitestorage Executing query: CREATE INDEX "process_id" ON "process" ("id"); 16:19:01 DEBUG firepit.sqlstorage _upsert: "INSERT INTO "process" ("name", "pid", "type", "id") VALUES (?, ?, ?, ?) ON CONFLICT (id) DO UPDATE SET "name" = EXCLUDED."name", "pid" = EXCLUDED."pid", "type" = EXCLUDED."type";" Traceback (most recent call last): File "/home/docker/.local/bin/kestrel", line 8, in runpy.run_module('kestrel', run_name='main') File "/usr/lib64/python3.6/runpy.py", line 208, in run_module return _run_code(code, {}, init_globals, run_name, mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/docker/.local/lib/python3.6/site-packages/kestrel/main.py", line 49, in outputs = session.execute(huntflow) File "/home/docker/.local/lib/python3.6/site-packages/kestrel/session.py", line 262, in execute return self._execute_ast(ast) File "/home/docker/.local/lib/python3.6/site-packages/kestrel/session.py", line 437, in _execute_ast output_var_struct, display = execute_cmd(stmt, self) File "/home/docker/.local/lib/python3.6/site-packages/kestrel/codegen/commands.py", line 92, in wrapper return func(stmt, session) File "/home/docker/.local/lib/python3.6/site-packages/kestrel/codegen/commands.py", line 60, in wrapper ret = func(stmt, session) File "/home/docker/.local/lib/python3.6/site-packages/kestrel/codegen/commands.py", line 123, in new stmt["type"] = load_data(session.store, stmt["output"], stmt["data"], stmt["type"]) File "/home/docker/.local/lib/python3.6/site-packages/kestrel/codegen/data.py", line 30, in load_data store.load(output_entity_table, data, entity_type, query_id) File "/home/docker/.local/lib/python3.6/site-packages/firepit/sqlstorage.py", line 294, in load splitter.close() File "/home/docker/.local/lib/python3.6/site-packages/firepit/splitter.py", line 228, in close self.writer.write_records(obj_type, recs, self.schemas[obj_type], self.replace, self.query_id) File "/home/docker/.local/lib/python3.6/site-packages/firepit/splitter.py", line 153, in write_records self.store.upsert(cursor, tablename, obj, query_id) File "/home/docker/.local/lib/python3.6/site-packages/firepit/sqlstorage.py", line 224, in upsert cursor.execute(stmt, values) sqlite3.OperationalError: near "ON": syntax error 16:19:01 DEBUG firepit.sqlitestorage Closing SQLite DB connection

The PR fully addresses #31 and partially addresses #32.

1. implement a new function get_entity_id_attribute() in src/kestrel/codegen/relations.py to compute the appropriate attribute used as identifier attribute for entities.
2. update src/kestrel/codegen/commands.py to use get_entity_id_attribute().
3. replace the or_pattern() for post-prefetch merge in src/kestrel/codegen/commands.py with firepit.merge() (partially address #32).
4. update get_variable_entity_count() in src/kestrel/codegen/summary.py to use get_entity_id_attribute().
5. update _get_variable_query_ids() in src/kestrel/codegen/summary.py since merged variable in firepit do not have __membership records.
6. update gen_variable_summary() in src/kestrel/codegen/summary.py to only give cached records when there is a data source query.

define GitHub Action for automatic package release to pypi https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/

This is a patch from Kestrel side. A upstream patch on stix-shifter will be better preventing the issue: https://github.com/opencybersecurityalliance/stix-shifter/issues/1087

Is your feature request related to a problem? Please describe. The hunter doesn't care if an analytic is using docker or python.

Describe the solution you'd like APPLY my_analytic should work whether it's a python- or docker-based analytic. Similarly FROM my_datasource should work without having to specify stixshifter://my_datasource.

Maybe we have something in config that lists interface preference order? E.g. python,docker so it checks python first, then docker. Stop at first match.

You could still supply the scheme, to force the one you want in case of name collisions.

Describe alternatives you've considered Leave it like it is now.

Additional context N/A

Is your feature request related to a problem? Please describe. In STIX pattern, a property or partial property that has dash - in it needs to be wrapped with single quotes, such as [file:hashes.'SHA-256' = 'xxxxxxxxx...']. This mean in Kestrel, one needs to write GET file WHERE hashes.'SHA-256' = 'xxxxxx...'. This rule may not be expected by most users. Thinking to relax it so users can write GET file WHERE hashes.SHA-256 = 'xxxxxx...' and Kestrel will assemble the STIX pattern with single quotes if needed.

Note that Kestrel is STIX compatible, so if we implement this, it will still allow users to have single quotes like hashes.'SHA-256', in which case Kestrel will not modify the string when assembling the STIX pattern.

Describe the solution you'd like firepit also needs the single quotes. So we can possibly add the single quotes if not there around substrings in attributes with dashes in the parser (transformer).

Describe alternatives you've considered Do the modification in to_stix() and to_firepit() in ECGP.

Additional context Additional consideration is whether this (difference from STIX) makes extra confusion for users who are familiar with STIX. However, since the planned solution supports both (just relaxing the strict single quote requirement), this could be fine.

Description: The autocompletion function doesn't properly address partially complete fields correctly. Fields with the same starter characters as commands also return commands matching last_word as suggestions (which they should not be doing). No error messages are outputted, but the behavior does not match what is expected. Examples included below for clarity. A possible solution mentioned is completely revamping the logic of the docomplete() function, looking at the parsing portion of the existing code in particular.

Environment: I verified that the variable autocompletion error appears in the Tutorial Huntbook environment, so I believe any Kestrel runtime environment through Jupyter Notebook should display this issue. For attribute autocompletion, I was running my Kestrel environment from Jupyter Notebook locally hosted through a Python 3 virtual environment on Windows 11 WSL Ubuntu 20.04.5 LTS. (I have no idea if the grammar there was correct, sorry...)

Details & How to Reproduce: Huntflow taken directly from the Kestrel tutorial (0. Hello World Hunt). In the code block below, <tab> represents hitting the tab button, which calls the autocomplete function (do_complete()) linked above.

proclist = NEW process [ {"name": "cmd.exe", "pid": "123"}
                       , {"name": "explorer.exe", "pid": "99"}
                       , {"name": "firefox.exe", "pid": "201"}
                       , {"name": "chrome.exe", "pid": "205"}
                       ]
browsers = GET process FROM proclist WHERE name IN ('firefox.exe', 'chrome.exe')

# --  scenario 1
DISP <tab>                       # case 1
DISP b<tab>                      # case 2
DISP browsers<tab>               # case 3
DISP browsers <tab>              # case 4

# -- add and run this as a new block before calling DISP
abc = browsers

# --  scenario 2
DISP a<tab>                      # case 2

Expected Behavior: For scenario 1, all cases behave as expected for autocompletion. (suggestions is the list returned by the do_complete() function)

1. suggestions = ['TIMESTAMPED', '_', 'browsers', 'proclist']
2. suggestions = ['browsers']
3. suggestions = ['']
4. suggestions = ['APPLY', 'ATTR', 'DISP', 'FIND', 'GET', 'GROUP', 'INFO', 'JOIN', 'LIMIT', 'LOAD', 'NEW', 'OFFSET', 'SAVE', 'SORT', 'TIMESTAMPED', 'WHERE', '_', 'browsers', 'proclist']

(My question: Why are browsers and proclist considered valid suggestions for case 4?)

Scenario 2 can be generalized to all scenarios where a variable shares the starter characters for autocompletion as a command. Most cases behave as expected, EXCEPT Case 2 which returns suggestions = ['bc', 'pply', 'ttr']. The expected behavior would be suggestions = ['bc'].

The following details are in regards to how this issue relates to another open issue (https://github.com/opencybersecurityalliance/kestrel-lang/issues/79), which details expanding the autocompletion feature to support attributes.

# attribute autocompletion
DISP browsers ATTR <tab>         # case 1
DISP browsers ATTR n<tab>        # case 2
DISP browsers ATTR name<tab>     # case 3
DISP browsers ATTR name <tab>    # case 4

My implementation of the attribute autocompletion feature can be found here. For case 2, the parser treats 'n' as a completed attribute field, but also as the value of last_word when searching for suggestions for the next field. As such, we end up with suggestions = ['ew'], which is wrong (and confusingly weird). The other cases behave as expected, though it might just be coincidental for case 3 in particular (the same applies for variable autocompletion). It seems that this behavior is the same as variable partial completion, so this issue must be addressed before progress can be made on the other.

Describe the bug

procs = GET process
        FROM file:///tmp/lab101.json
        WHERE parent_ref.name = 'svchost.exe'
        START 2021-04-03T00:00:00Z STOP 2021-04-03T02:00:00Z
        
procs_grps = GROUP procs BY binary_ref.name WITH COUNT(pid) AS number_of_procs

APPLY python://attribute-plot ON procs_grps WITH XPARAM=binary_ref.name, YPARAM=number_of_procs

error:

[ERROR] KestrelSyntaxError: invalid token "'binary_ref.name'" at line 6 column 29, expects one of ['BIN', 'ATTRIBUTE']
rewrite the failed statement.

Kestrel version: v1.5.1

Is your feature request related to a problem? Please describe. Currently Kestrel is supported on Linux and macOS. It could be useful to explore deployment on Microsoft Windows, writing doc on how to set it up (if special instruction is needed similar to the macOS requirement), and fixing issues in code if needed.

Describe the solution you'd like A first step is to test/support Kestrel running in Windows Subsystem for Linux. The second step is to test/support Kestrel running as a native Windows application (with Python environment installed).