Age of Empires II recorded game parsing and summarization in Python 3.

Related tags

Miscellaneouspythonparseraocmgzaoe2aoe2record
Overview

mgz

Age of Empires II recorded game parsing and summarization in Python 3.

Supported Versions

  • Age of Kings (.mgl)
  • The Conquerors (.mgx)
  • Userpatch 1.4 (.mgz)
  • Userpatch 1.5 (.mgz)
  • HD Edition >= 4.6 (.aoe2record)
  • Definitive Edition (.aoe2record)

Architecture

The core functionality of mgz is a parser that produces a Python data structure based on a recorded game file. It also offers abstracted representations that make it easier to use the data.

Parsers

mgz offers two parsers, fast and full. The fast parser skips data that is rarely needed, while the full parser tries to parse as much as possible. Naturally, the fast parser is faster than the full parser.

Abstractions

Abstractions take parser output as input and return an object with normalized data that is easier to use for most cases. There are two abstractions available, summary and model. The summary abstraction attempts to expose the maximum amount of usable data. The model abstraction is more limited but automatically performs more lookups.

Support

Version model summary fast (header) fast (body) full (header) full (body)
Age of Kings (.mgl) ✓ ✓ ✓
The Conquerors (.mgx) ✓ ✓ ✓
Userpatch <= 1.4 (.mgz) ✓ ✓ ✓ ✓
Userpatch 1.5 (.mgz) ✓ ✓ ✓ ✓ ✓ ✓
HD Edition >= 4.6 ✓ ✓ ✓ ✓
HD Edition 5.8 ✓ ✓ ✓ ✓ ✓ ✓
Definitive Edition <= 13.34 (.aoe2record) ✓ ✓ ✓ ✓
Definitive Edition > 13.34 (.aoe2record) ✓ ✓ ✓ ✓ ✓ ✓

Examples

Full Parser

import os
from mgz import header, body

with open('/path/to/file', 'rb') as data:
    eof = os.fstat(data.fileno()).st_size
    header.parse_stream(data)
    body.meta.parse_stream(data)
    while data.tell() < eof:
        body.operation.parse_stream(data)

Full Parser (header) + Fast Parser (body)

import os
from mgz import header, fast

with open('/path/to/file', 'rb') as data:
    eof = os.fstat(data.fileno()).st_size
    header.parse_stream(data)
    fast.meta(data)
    while data.tell() < eof:
        fast.operation(data)

Summary

from mgz.summary import Summary

with open('/path/to/file', 'rb') as data:
    s = Summary(data)
    s.get_map()
    s.get_platform()
    # ... etc

Model

from mgz.model import parse_match

with open('/path/to/file', 'rb') as data:
    match = parse_match(data)
    match.map.name
    match.file.perspective.number
    # ... etc

To JSON

import json
from mgz.model import parse_match, serialize

with open('/path/to/file', 'rb') as h:
    match = parse_match(h)
    print(json.dumps(serialize(match), indent=2))

Frequently Asked Questions

Q: Where are the end-of-game achievements/statistics?

A: In the postgame action, available only from Userpatch version.

Q: How can I tell the number of resources/kills/etc at a certain point?

A: You can't, without replaying the match in-game.

Q: How does a recorded game file work?

A: The first portion (the header) is a snapshot of the initial game state. The second portion (the body) is a list of moves made by players. The game loads the header, then applies each move to mutate the state according to the game rules.

Q: How can I install this package?

A: pip install mgz

Contribution

  • Pull requests & patches welcome

Resources

Comments
  • DE saved files aren't parsed correctly

    DE saved files aren't parsed correctly

    I understand DE saved files are kinda unsupported right now, Just wanna offer a corpus of ~250 DE games if that would be useful. Only 10 of my saved files parse without errors. Here is an example of the type of errors I get (each error message is from a different save file):

    invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 11
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 11
    (parsing) -> lobby
invalid mgz file: unspecified builtin map: 124
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: expected 16 to 16, found 6
    (parsing) -> scenario -> players
invalid mgz file: unspecified builtin map: 126
invalid mgz file: unspecified builtin map: 123
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: unspecified builtin map: 123
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: no decoding mapping for 11
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 11
    (parsing) -> lobby
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 16 to 16, found 4
    (parsing) -> scenario -> players
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: unspecified builtin map: 0
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 1032, found 718
    (parsing) -> scenario -> triggers
invalid mgz file: could not read enough bytes, expected 65535, found 29825
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 32768, found 31170
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 65535, found 29871
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 12544, found 2515
    (parsing) -> scenario
invalid mgz file: could not read enough bytes, expected 32768, found 29204
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 27996
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 27167
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 12544, found 2
    (parsing) -> scenario
invalid mgz file: expected 16 to 16, found 15
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 10
    (parsing) -> scenario -> players
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 59392, found 40667
    (parsing) -> scenario -> messages -> victory
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: could not read enough bytes, expected 32768, found 31621
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 32768, found 31020
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: could not read enough bytes, expected 32768, found 27266
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 32768, found 31179
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 31623
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 30962
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: could not read enough bytes, expected 32768, found 31599
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 27066
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 12544, found 5855
    (parsing) -> scenario
invalid mgz file: could not read enough bytes, expected 32768, found 27266
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 1
    (parsing) -> scenario -> players

    If it'd be useful, I could upload all of my saves.

    I'll start to study this code myself and see if I can help fix things.

    opened by connorjclark 14
  • RangeError: expected 50 to 50, found 0

    RangeError: expected 50 to 50, found 0

    Tried the following code:

    from mgz import header, body
from os import fstat

with open('C:\\Users\\rajje\\PycharmProjects\\helloworld\\replay.aoe2record', 'rb') as data:
    eof = fstat(data.fileno()).st_size
    header.parse_stream(data)
    while data.tell() < eof:
        body.operation.parse_stream(data)

    Go this error:

    Traceback (most recent call last):
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 1042, in _parse
    obj.append(self.subcon._parse(stream, context._, path))
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 1235, in _parse
    pad = _read_stream(stream, padlen)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 67, in _read_stream
    data = stream.read(length)
OverflowError: Python int too large to convert to C ssize_t

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 1050, in _parse
    raise RangeError("expected %d to %d, found %d" % (min, max, len(obj)))
construct.core.RangeError: expected 50 to 50, found 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "app.py", line 6, in <module>
    header.parse_stream(data)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\mgz\util.py", line 36, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.RangeError: expected 50 to 50, found 0
    (parsing) -> map_info
    opened by rajjeet 12
  • DE version 25.22 fails on header parse

    DE version 25.22 fails on header parse

    opened by Namek 11
  • Why my recordings can't be parsed?

    Why my recordings can't be parsed?

    Some recordings downloaded from Voobly are working for me but unfortunately, some are not. Including my own.

    This is what happens when I try to load my recording:

    Traceback (most recent call last):
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 506, in _parse
    return packer.unpack(self.fmtstr, _read_stream(stream, self.sizeof()))[0]
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 69, in _read_stream
    raise FieldError("could not read enough bytes, expected %d, found %d" % (length, len(data)))
construct.core.FieldError: could not read enough bytes, expected 4, found 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 508, in _parse
    raise FieldError("packer %r error during parsing" % self.fmtstr)
construct.core.FieldError: packer '<f' error during parsing

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1042, in _parse
    obj.append(self.subcon._parse(stream, context._, path))
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.FieldError: packer '<f' error during parsing
    (parsing) -> initial -> players -> objects -> objects -> properties -> building -> combat -> ai -> ai -> orders -> order -> target_location -> z

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1050, in _parse
    raise RangeError("expected %d to %d, found %d" % (min, max, len(obj)))
construct.core.RangeError: expected 4294918016 to 4294918016, found 35248

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1042, in _parse
    obj.append(self.subcon._parse(stream, context._, path))
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/mnt/c/aoe2/aoc-mgz/mgz/util.py", line 196, in _parse
    subobj = self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.RangeError: expected 4294918016 to 4294918016, found 35248
    (parsing) -> initial -> players -> objects -> objects -> properties -> building -> combat -> ai -> ai -> orders

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1050, in _parse
    raise RangeError("expected %d to %d, found %d" % (min, max, len(obj)))
construct.core.RangeError: expected 6 to 6, found 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 6, in <module>
    print(header.parse_stream(data))
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/mnt/c/aoe2/aoc-mgz/mgz/util.py", line 54, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.RangeError: expected 6 to 6, found 0
    (parsing) -> initial

    We are using: Game Mod: v1.5 RC | Allied Vision

    You can download the recording here: https://www.voobly.com/files/view/60445255/nfw5tdx83ea9eqmior7wkqsus9zo87ik

    opened by Defozo 11
  • WiC mod support?

    WiC mod support?

    Hi HappyLeaves,

    Do you know if there's something about WiC mod that breaks this parser? I was trying to parse the Clan Masters recs here (for example: http://aoczone.net/viewtopic.php?f=1342&t=123008) and they're all crashing near the end with:

    /root/miniconda3/envs/mgz/lib/python2.7/site-packages/construct/core.pyc in _read_stream(stream, length)
    303     data = stream.read(length)
    304     if len(data) != length:
--> 305         raise FieldError("expected %d, found %d" % (length, len(data)))
    306     return data
    307 

FieldError: expected 1512, found 4

    I'm guessing it may be something to do with achievements since I can't get that action to extract no matter what I do, but I'm not sure exactly.

    opened by coffenbacher 9
  • Current Patch DE Game Loading Error

    Current Patch DE Game Loading Error

    Im working on a data analytics site for a league. Im trying to analyze games with your library but can't seem to get games to load from the newest patch. Your info says that the library works for version 25.02 and earlier. How do these version correspond to game version? Ex. Current version is: 101.101.54684 7486483.

    The error I am getting: `Exception has occurred: RuntimeError (note: full exception trace is shown but execution is paused at: setUpClass) could not parse File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 418, in parse de = parse_de(header, version, save) File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 296, in parse_de lobby = de_string(data) File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 47, in de_string assert data.read(2) == b'\x60\x0a'

    During handling of the above exception, another exception occurred:

    File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 47, in de_string assert data.read(2) == b'\x60\x0a' File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 296, in parse_de lobby = de_string(data) File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 418, in parse de = parse_de(header, version, save) File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 426, in parse raise RuntimeError("could not parse") File "C:\Users\Owner\Git Repos\aoc-mgz\tests\test_fast.py", line 60, in setUpClass (Current frame) cls.data = parse(handle)`

    Let me know if what I'm doing wrong or if the current version of games is not supported.

    Test_Game.zip .0

    opened by ghost 8
  • New aoe2 de recorded files parsing ending early

    New aoe2 de recorded files parsing ending early

    Hi,

    First of all want to thank for for this great work.

    I was trying to parse files in aoe2 de recorded files of the past few days. The parsing is ending early with the last operation being embedded with default consuming the rest of the file (from my understanding). Maybe they added new embedded operation which is not handled right now. I would like to help in anyway.

    Again thanks.

    opened by arunpatala 7
  • Unable to use JSON parser

    Unable to use JSON parser

    The code snippet for the JSON parser doesn't seem to be working. I've got the full parser working but one of the latest DE test files (provided in this repo) is not working with the JSON parser. Here is the error I get:

    /<my_absolute_path>/bin/python /<my_absolute_path>/test.py
Traceback (most recent call last):
  File "/<my_absolute_path>/lib/python3.9/site-packages/mgz/fast/header.py", line 461, in parse
    de = parse_de(header, version, save)
  File "/<my_absolute_path>/lib/python3.9/site-packages/mgz/fast/header.py", line 338, in parse_de
    de_string(data)
  File "/<my_absolute_path>/lib/python3.9/site-packages/mgz/fast/header.py", line 47, in de_string
    assert data.read(2) == b'\x60\x0a'
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/<my_absolute_path>/test.py", line 31, in <module>
    match = parse_match(h)
  File "/<my_absolute_path>/lib/python3.9/site-packages/mgz/model/__init__.py", line 93, in parse_match
    data = parse(handle)
  File "/<my_absolute_path>/lib/python3.9/site-packages/mgz/fast/header.py", line 469, in parse
    raise RuntimeError("could not parse")
RuntimeError: could not parse

    For this file: de-26.16.zip

    Is the JSON parser not working yet for this version of DE recordings?

    opened by NathanFlegg 6
  • Unable to run any of the examples provided on the main page

    Unable to run any of the examples provided on the main page

    Hello,

    For some reason, I cannot run any of the examples provided on the main page. Are there special requirements that are not listed? My python version is 3.8.7.
    Here is a simple script that is used to test the five examples and log the error traceback to a file. All you need to change is EXAMPLE = 1|2|3|4|5 in the constants.

    # Imports.
import mgz
import os

# Constants.
PATH = "test.aoe2record"
EXAMPLE = 5

# Redirect the error to a file.
with open(f"log_example_{EXAMPLE}.txt", "w") as log_file:

    # Open a game and get information.
    try:
        with open(PATH, "rb") as data:

            if EXAMPLE == 1:
                from mgz import header, body
                eof = os.fstat(data.fileno()).st_size
                header.parse_stream(data)
                body.meta.parse_stream(data)
                while data.tell() < eof:
                    body.operation.parse_stream(data)

            elif EXAMPLE == 2:
                from mgz import header, fast
                eof = os.fstat(data.fileno()).st_size
                header.parse_stream(data)
                fast.meta(data)
                while data.tell() < eof:
                    fast.operation(data)

            elif EXAMPLE == 3:
                from mgz.summary import Summary
                s = Summary(data)
                s.get_map()
                s.get_platform()

            elif EXAMPLE == 4:
                from mgz.model import parse_match
                match = parse_match(data)
                match.map.name
                match.file.perspective.number

            elif EXAMPLE == 5:
                import json
                from mgz.model import parse_match, serialize
                match = parse_match(data)
                print(json.dumps(serialize(match), indent=2))

    except Exception:
        import traceback
        log_file.write(traceback.format_exc())

    test.zip contains test.aoe2record.

    Manually looking at the header, this game has the following characteristics:

    data = open(test.aoe2record, "rb")
from mgz.summary import *
header = decompress(data)
version, game, save, log = parse_version(header, data)
>>> version = <Version.DE: 21>
>>> game = 'VER 9.4'
>>> save = 25.22
>>> log = 5

    The 5 tracebacks are here: log_example_1.txt log_example_2.txt log_example_3.txt log_example_4.txt log_example_5.txt

    The 5 top-most errors are:

    construct.core.ConstError: expected b'\x00\x0b' but parsed b'\x00\x00'

construct.core.ConstError: expected b'\x00\x0b' but parsed b'\x00\x00'

construct.core.ConstError: expected b'\x00\x0b' but parsed b'\x00\x00'

  File "C:\WPy64-3870\python-3.8.7.amd64\lib\site-packages\mgz\fast\header.py", line 48, in de_string
    assert data.read(2) == b'\x60\x0a'
AssertionError

  File "C:\WPy64-3870\python-3.8.7.amd64\lib\site-packages\mgz\fast\header.py", line 48, in de_string
    assert data.read(2) == b'\x60\x0a'
AssertionError

    Thank you in advance for your help.

    opened by Guimoute 6
  • Faster Parsing

    Faster Parsing

    Hi, I want to parse a lot recorded matches. At the moment parsing one match takes 1-10 seconds on my machine. I think I am already using the fast parsing option.

    I am only interested in game duration and which player has won so I do not need to parse all data.

    Is there a way to speed up the parsing process?

    I have noticed that a large portion of the processing is parsing the map tiles. Can I somehow skip this?

    opened by denniske 6
  • Can't get field 'unit_ids' for 'move' action

    Can't get field 'unit_ids' for 'move' action

    Steps to reproduce:

    • Recorded game used (I've tried with many others and there is the same problem): https://www.voobly.com/files/view/53123676/qnc2045gpo3x3vkyyqks5h526pu725er

    • Code used:

    import sys
from mgz import header, body
import pprint

if __name__ == "__main__":
    pp = pprint.PrettyPrinter(indent=4)
    # For each input filename
    for arg in sys.argv[1:]:
        with open(arg, 'rb') as f:
            # Remember end of file
            f.seek(0, 2)
            eof = f.tell()
            f.seek(0)
            # Parse the header
            h = header.parse_stream(f)
            # Parse the body
            while f.tell() < eof:
                # Parse a body operation
                o = body.operation.parse_stream(f)
                if o.type == 'action' and o.action.type == 'move':
                    pp.pprint(o)

    Problem:
    None of the 'move' action has the field 'unit_ids'

    I see that this field should be present (excepted if moving the same unit as previous command):
    https://github.com/happyleavesaoc/aoc-mgz/blob/master/mgz/body/actions.py#L63

    I'm not fluent in python so please forgive me if I made a simple mistake

    Thanks for your work!

    opened by ElKrist 6
  • RuntimeError: bad object parse

    RuntimeError: bad object parse

    Trying to get the record in json format and I get the following error:

    File "scripts/record2json.py", line 14, in <module>
    match = parse_match(h)
  File "/home//.local/lib/python3.10/site-packages/mgz/model/__init__.py", line 94, in parse_match
    data = parse(handle)
  File "/home//.local/lib/python3.10/site-packages/mgz/fast/header.py", line 498, in parse
    players, mod = parse_players(header, num_players, version)
  File "/home//.local/lib/python3.10/site-packages/mgz/fast/header.py", line 461, in parse_players
    players = [parse_player(header, number, num_players) for number in range(num_players)]
  File "/home//.local/lib/python3.10/site-packages/mgz/fast/header.py", line 461, in <listcomp>
    players = [parse_player(header, number, num_players) for number in range(num_players)]
  File "/home//.local/lib/python3.10/site-packages/mgz/fast/header.py", line 126, in parse_player
    sleeping, end = object_block(data, end, player_number, 1)
  File "/home//.local/lib/python3.10/site-packages/mgz/fast/header.py", line 97, in object_block
    objects.append(dict(parse_object(data, pos), index=index))
  File "/home//.local/lib/python3.10/site-packages/mgz/fast/header.py", line 63, in parse_object
    raise RuntimeError("bad object parse")
RuntimeError: bad object parse

    When using Summary I get the following error:

    could not fast parse; falling back: bad object parse

    I can still get some data with Summary though. AgeIIDE_Replay_170.zip

    opened by jdekarske 0
  • RuntimeError: invalid mgz file: expected 3 to 3, found 2

    RuntimeError: invalid mgz file: expected 3 to 3, found 2 

    Traceback (most recent call last):
  File "C:\dev\ant_league\backend\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 369, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "C:\dev\ant_league\backend\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 59, in __call__
    return await self.app(scope, receive, send)
  File "C:\dev\ant_league\backend\lib\site-packages\fastapi\applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\middleware\errors.py", line 181, in __call__
    raise exc from None
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\middleware\errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\middleware\cors.py", line 86, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\middleware\cors.py", line 142, in simple_response
    await self.app(scope, receive, send)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\middleware\sessions.py", line 75, in __call__
    await self.app(scope, receive, send_wrapper)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\exceptions.py", line 82, in __call__
    raise exc from None
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\routing.py", line 52, in app
    response = await func(request)
  File "C:\dev\ant_league\backend\lib\site-packages\fastapi\routing.py", line 219, in app
    raw_response = await run_endpoint_function(
  File "C:\dev\ant_league\backend\lib\site-packages\fastapi\routing.py", line 154, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\concurrency.py", line 40, in run_in_threadpool
    return await loop.run_in_executor(None, func, *args)
  File "C:\Users\Namek\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\dev\ant_league\backend\src\website.py", line 127, in post_match
    match_info = utils.get_match_info(file.file)
  File "C:\dev\ant_league\backend\src\utils.py", line 44, in get_match_info
    s = Summary(data)
  File "c:\dev\ant_league\backend\mgz\mgz\summary\__init__.py", line 85, in __init__
    raise RuntimeError("invalid mgz file: {}".format(e))
RuntimeError: invalid mgz file: expected 3 to 3, found 2
    (parsing) -> initial

    MP_Replay_v101.101.53347.0_2021.09.16_214614_1.zip

    opened by Namek 0
Owner
GitHub Repository
pyshell is a Linux subprocess module

pyshell A Linux subprocess module, An easier way to interact with the Linux shell pyshell should be cross platform but has only been tested with linux

4 Mar 02, 2022
AKSWINPOSTINIT -- AKS Windows node post provisioning initialization

AKSWINPOSTINIT -- AKS Windows node post provisioning initialization Features This is a tool that provides one-time powershell script initilization for

Ping He 3 Nov 25, 2021
Clock in automatically in SCU.

auto_clock_in Clock in automatically in SCU. Features send logs to Telegram bot How to use? pip install -r requirements.txt () edit user_list, token_A

2 Dec 13, 2021
Find functions without canary check (or similar)

Ghidra Check Protector Which non-trivial functions don't reference the stack canary checker (or other, user-defined function)? Place your cursor to th

buherator 3 Jan 17, 2022
This is a simple leaderboard for 30 days of Google Cloud program for students of ASIET

30daysleaderboard #Hacktoberfest - Please don't make changes in readme file. Only improvement in the project will be accepted. Update - Now if you run

5 Oct 29, 2021
Python: Wrangled and unpivoted gaming datasets. Tableau: created dashboards - Market Beacon and Player’s Shopping Guide.

Created two information products for GameStop. Using Python, wrangled and unpivoted datasets, and created Tableau dashboards.

Zinaida Dvoskina 2 Jan 29, 2022
ABT aka Animated Background Tool is a windows only python program that makes it that you can have animated background.

ABT ABT aka Animated Background Tool is a windows only python program that makes it that you can have animated background. 𝓡𝓔𝓐𝓓 𝓜𝓔, An Important

Yeeterboi4 2 Jul 16, 2022
Weblate is a copylefted libre software web-based continuous localization system

Weblate is a copylefted libre software web-based continuous localization system, used by over 2500 libre projects and companies in more than 165 count

Weblate 7 Dec 15, 2022
inverted pendulum fuzzy control python code (python 2.7.18)

inverted-pendulum-fuzzy-control- inverted pendulum fuzzy control python code (python 2.7.18) We have 3 general functions for 3 main steps: fuzzificati

arian mottaghi 4 May 23, 2022
Agora-token-helper - Some help tools for AgoraToken

Agora Token Helper Support AgoraToken version 001 - 006. But for security reason

imagora 1 Jan 26, 2022
A Notifier Program that Notifies you to relax your eyes Every 15 Minutes👀

Every 15 Minutes is an application that is used to Notify you to Relax your eyes Every 15 Minutes, This is fully made with Python and also with the us

Ashely Sato 1 Nov 02, 2021
A small project of two newbies, who wanted to learn something about Python language programming, via fun way.

HaveFun A small project of two newbies, who wanted to learn something about Python language programming, via fun way. What's this project about? Well.

Patryk Sobczak 2 Nov 24, 2021
synchronize projects via yaml/json manifest. built on libvcs

vcspull - synchronize your repos. built on libvcs Manage your commonly used repos from YAML / JSON manifest(s). Compare to myrepos. Great if you use t

python utilities for version control 200 Dec 20, 2022
Hello, Welcome to this repo. don't forget to read guidelines in readme.md

Hacktoberfest_2021 If you looking for your first contribution, we are here to help. Just create a simple program using any language you like in our fo

Wafa Rifqi Anafin 117 Dec 14, 2022
A platform for developers 👩‍💻 who wants to share their programs and projects.

Fest-Practice-2021 This project is excluded from Hacktoberfest 2021. Please use this as a testing repo/project. A platform for developers 👩‍💻 who wa

Mayank Choudhary 40 Nov 07, 2022
🎉 🎉 PyComp - Java Code compiler written in python.

🎉 🎉 PyComp Java Code compiler written in python. This is yet another compiler meant for babcock students project which was created using pure python

Alumona Benaiah 5 Nov 30, 2022
A community-driven python bot that aims to be as simple as possible to serve humans with their everyday tasks

JARVIS on Messenger Just A Rather Very Intelligent System, now on Messenger! Messenger is now used by 1.2 billion people every month. With the launch

Swapnil Agarwal 1.3k Jan 07, 2023
Cairo-math-64x61 - Fixed point 64.61 math library for Cairo / Starknet

Cairo Math 64x61 A fixed point 64.61 math library for Cairo & Starknet Signed 64

Influence 63 Dec 05, 2022
Short, introductory guide for the Python programming language

100 Page Python Intro This book is a short, introductory guide for the Python programming language.

Sundeep Agarwal 185 Dec 26, 2022
Chicks get hostloc points regularly

hostloc_getPoints 小鸡定时获取hostloc积分 github action大规模失效，mjj平均一人10鸡，以下可以部署到自己的小鸡上

59 Dec 28, 2022