Since issue 372 was closed, I open a new issue for multiline support. See https://github.com/ait-aecid/logdata-anomaly-miner/issues/372

As I mentioned in the issue, it would be good to have an optional EOL parameter in the config to support simple multiline logs that are clearly separable, e.g., by \n\n that otherwise does not occur. We could also think about supporting more advanced multiline logs, in particular, json formatted logs where each json object spans over several lines rather than a single line. This could be solved by counting brackets, i.e., the ByteStreamAtomizer increases a counter (initially set to 0) for every "{" and decreases it for every "}" (or any other user-defined characters), and passes a log_atom to the parser every time this counter reaches 0.

allowlisted_paths in ECD should be named blocklisted_paths, since these paths are not considered for detection.

allowlisted_paths should also exist, but does the oppsite: Only when one of the paths in the logatom match dictionary contains one of the allowlisted_paths, analysis should be carried out.

The attribute paths should overrule these lists.

This feature should be available for all detectors that may be analyzing all available parser matches, such as the VTD.

/usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from spec or package, falling back on name and path

return f(*args, **kwds)

should not occur, when running the aminer.

When using the %z in the parsing model (see slow.txt), I get around 50 lines per second. Without it I get around 1000 lines per second (see fast.txt). There is something wrong with parsing %z in the DateTimeModelElement.

fast.txt slow.txt train.log config.py.txt

Make sure these boxes are signed before submitting your Pull Request -- thank you.

Must haves

• [x] I have read and followed the contributing guide lines at https://github.com/ait-aecid/logdata-anomaly-miner/wiki/Git-development-workflow
• [x] Issues exist for this PR
• [x] I added related issues using the "Fixes #"-notations
• [x] This Pull-Requests merges into the "development"-branch

Fixes #1061 Fixes #1074

Submission specific

• [ ] This PR introduces breaking changes
• [ ] My change requires a change to the documentation
• [ ] I have updated the documentation accordingly
• [ ] I have added tests to cover my changes
• [ ] All new and existing tests passed

Describe changes:

There should be a parameter for the command line that backups the persistency in regular intervals. Also, there should be a command for the remote control that saves the persistency when executed.

The persistency should be copied into a directory /var/lib/aminer/backup/yyyy-mm-dd-hh-mm-ss/...

There should also be the possibility to restore configs, by remote control, config settings, etc.

My log file contains tabulators (e.g. System name:\tTESTNAME). However, the byte strings in the parsing models cannot interpret these tabulators (\t): FixedDataModelElement('fixed1', b'System name:\t'),

How can I make it possible for the tabs to be interpreted correctly?

There should be a way to write everything that the AMiner outputs in a file. For example, in the beginning of the config, a parameter StandardOutput: "/etc/aminer/output.txt" can be set, where all the output (anomalies, errors, etc) is written to in addition to the usual output components. By default, it should be None and not write anything.

It is possible to define two detectors of the same type that will end up persisting in the same file - this can especially happen by accident, when the "Default" name is used. We should not prevent it completely, but at least print a warning when two or more detectors persist on the same file.

There should be a way to use a MatchRule so that only logs that match are forwarded to a specific detector, using the AtomFilterMatchAction. This can be done in python configs, but not in yaml configs. Also, tests and documentation is missing.

I have this sample data:

[email protected]:/home/ubuntu# cat file3.log 
{"a": ["success", "a.png"]}
{"a": ["success", "b.png"]}
{"a": ["fail", "c.png"]}
{"a": ["success", "c.png"]}

The values in the list should be detected with a value detector. They should not be mixed, i.e., the first and second element in the list are independent.

I use the following config to parse the file:

LearnMode: True

LogResourceList:
  - "file:///home/ubuntu/file3.log"

Parser:  
       - id: x
         type: VariableByteDataModelElement
         name: 'x'
         args: '.abcdefghijklmnopqrstuvwxyz1234567890ABCDEFGGHIJKLMNOPQRSTUVWXYZ'

       - id: json
         start: True
         type: JsonModelElement
         name: 'model'
         key_parser_dict:
           "a": 
             - x

Input:
        timestamp_paths: None
        verbose: True
        json_format: True

Analysis:
        - id: vd
          type: NewMatchPathValueDetector
          paths:
              - '/model/x'
          learn_mode: true
          persistence_id: test

EventHandlers:
        - id: stpe
          json: true
          type: StreamPrinterEventHandler

Note that I use a value detector on the list. The result is as follows:

[email protected]:/home/ubuntu# cat /var/lib/aminer/NewMatchPathValueDetector/test 
["bytes:a.png", "bytes:c.png", "bytes:b.png"]

Only the last value has been learned, but I also want to learn the first element in the array.

I propose to model all elements of the lists as their own elements, so that the parser looks like this:

Parser:
       - id: y
         type: FixedWordlistDataModelElement
         name: 'y'
         args:
           - 'success'
           - 'fail'
             
       - id: x
         type: VariableByteDataModelElement
         name: 'x'
         args: '.abcdefghijklmnopqrstuvwxyz1234567890ABCDEFGGHIJKLMNOPQRSTUVWXYZ'

       - id: json
         start: True
         type: JsonModelElement
         name: 'model'
         key_parser_dict:
           "a": 
             - y
             - x

and the analysis could look like this, where each element can be addressed individually by an analysis component:

Analysis:
        - id: vd
          type: NewMatchPathValueDetector
          paths:
              - '/model/x'
          learn_mode: true
          persistence_id: test

        - id: vd
          type: NewMatchPathValueDetector
          paths:
              - '/model/y'
          learn_mode: true
          persistence_id: test

The current implementation uses a single element to model all elements of the list. This can also be convenient and should be possible by introducing a new element called ListOfElements. It should parse any number of elements in the list with the specified parsing model element. For example, the list of elements here is a list of variable byte elements:

Parser:
       - id: loe
         type: ListOfElements
         name: 'loe'
         args: z
             
       - id: z
         type: VariableByteDataModelElement
         name: 'z'
         args: '.abcdefghijklmnopqrstuvwxyz1234567890ABCDEFGGHIJKLMNOPQRSTUVWXYZ'

       - id: json
         start: True
         type: JsonModelElement
         name: 'model'
         key_parser_dict:
           "a": 
             - loe

The ListOfElements element should then assign the index of the element in the JSON list at the end of the path. For example, the following paths can be used in the analysis section:

Analysis:
        - id: vd
          type: NewMatchPathValueDetector
          paths:
              - '/model/loe/0'
          learn_mode: true
          persistence_id: test

        - id: vd
          type: NewMatchPathValueDetector
          paths:
              - '/model/loe/1'
          learn_mode: true
          persistence_id: test

Make sure these boxes are signed before submitting your Pull Request -- thank you.

Must haves

• [x] I have read and followed the contributing guide lines at https://github.com/ait-aecid/logdata-anomaly-miner/wiki/Git-development-workflow
• [x] Issues exist for this PR
• [x] I added related issues using the "Fixes #"-notations
• [x] This Pull-Requests merges into the "development"-branch

Fixes #1008 Fixes #1009

Submission specific

• [ ] This PR introduces breaking changes
• [ ] My change requires a change to the documentation
• [ ] I have updated the documentation accordingly
• [ ] I have added tests to cover my changes
• [ ] All new and existing tests passed

Describe changes:

• Please change the attached wiki files first in the development branch. HowTo-Create-your-own-FrequencyDetector.md HowTo-Create-your-own-SequenceDetector.md

Make sure these boxes are signed before submitting your Pull Request -- thank you.

Must haves

• [x] I have read and followed the contributing guide lines at https://github.com/ait-aecid/logdata-anomaly-miner/wiki/Git-development-workflow
• [x] Issues exist for this PR
• [x] I added related issues using the "Fixes #"-notations
• [x] This Pull-Requests merges into the "development"-branch

Fixes #1181

Submission specific

• [ ] This PR introduces breaking changes
• [ ] My change requires a change to the documentation
• [ ] I have updated the documentation accordingly
• [ ] I have added tests to cover my changes
• [ ] All new and existing tests passed

Describe changes:

When adding a new detector and running the tests, they usually fail at test26_filter_config_errors in YamlConfigTest.py as there is an integer that needs to be incremented. For example, see PR #1180 where this had to be fixed when adding a new detector. It is hard to spot why this test fails as it has nothing to do with the added detector and it is not an indicator of something that needs to be fixed. I therefore suggest to modify this test case so that no matter what integer comes after the "definition" keyword, the test passes. Then adding new detectors in the future should not make it necessary to always update this test.

LogResourceList:

   - url: "file:///var/log/apache2/access.log"
   - url: "unix:///var/lib/akafka/aminer.sock"
     type: json  # Konfiguriert den ByteStream
     parser_id: kafka_audit_logs  # Konfiguriert den zugehörigen Parser


Parser:
   - id: kafka_audit_logs
     type: AuditDingsParser

   - id: ApacheAccessModel
     start: true

Currently the complete docker image is build at once. This takes a lot of time for each build. We could shorten the build time by inheriting from a pre-built image.