When mapping out methods / lump formats it would be quite handy to use importlib to reload branch scripts and reload a single lump

>>> import bsp_tool
>>> import my_branch_script
>>> bsp = bsp_tool.load_bsp("maps/subject.bsp", branch=my_branch_script)
ERROR LOADING LUMP_XX:
	...
# edit my_branch_script.py
>>> import importlib
>>> importlib.reload(my_branch_script)
>>> bsp.set_branch(my_branch_script)
>>> bsp.reload("LUMP_XX")

or even writing a test lump class live in editor

>>> import bsp_tool
>>> import enum
>>> class BranchScript:
...    class LUMP(enum.Enum):
...        LUMP_XX = 42
...    lump_header_address = {LUMP.LUMP_XX: (16 + 42 * 16)}
...    LUMP_CLASSES = dict()
...    SPECIAL_LUMP_CLASSES = dict()
...    methods = []
>>> branch_script = BranchScript()
>>> class Xx(bsp_tool.branches.base.MappedArray):
...     _format = "3f"
...     _mapping = [*"xyz"]
>>> branch_script.LUMP_CLASSES["LUMP_XX"] = Xx
>>> bsp = bsp_tool.load_bsp("maps/subject.bsp", branch=branch_script)
ERROR LOADING LUMP_XX
	...
>>> class NewXx(bsp_tool.branches.base.MappedArray):
...     _format = "4f"
...     _mapping = [*"xyzw"]
>>> bsp.branch.LUMP_CLASSES["LUMP_XX"] = NewXx
>>> bsp.reload("LUMP_XX")

Though even this theorised process could be simplified a little, but loading new lump classes faster would be a very helpful tool

https://github.com/snake-biscuits/bsp_tool/blob/56c425188073178d7d605c15b24be8e691ca831f/bsp_tool/branches/respawn/apex_legends.py#L53

Appears this is unused, but I'm not sure if that was just put there in a haste or is actually reality. I could not find any information on LUMP_PHYSLEVEL which I suspect may be the alternative lump which holds collision data.

Keep forgetting about the reverse order needed since we're matching gcc bitfield order I have no idea how/if bitfield order is standardised, and I fear endian flipped bitfields

Testing apex_legends.BVHNode index bitfields gave some wierd errors

>>> base.BitField(0xAAAAAABC, _format="I", _fields={"a": 24, "b": 4, "c": 4})
Traceback (most recent call last):
  ...
OverflowError: a is out of range! (max allowed value = 16777215)
>>> 0XFFFFFFFF == (2 ** 32) - 1
True
>>> 0xAAAAAA < (2 ** 24) - 1
True
>>> 0xAAAABC < (2 ** 24) - 1
True
>>> bsp_tool.branches.base.BitField(0xAAAAAABC, _format="I", _fields={"a": 24, "b": 4, "c": 4})
Traceback (most recent call last):
  ...
OverflowError: a is out of range! (max allowed value = 16777215)
>>> bsp_tool.branches.base.BitField(0x00000123, _format="I", _fields={"a": 24, "b": 4, "c": 4})
<BitField (a: 291, b: 0, c: 0)>
>>> bsp_tool.branches.base.BitField(0x00000123, _format="I", _fields={"c": 4, "b": 4, "a": 24})
Traceback (most recent call last):
  ...
OverflowError: c is out of range! (max allowed value = 15)
>>> 0x123 >> 24
0
>>> (0x123 >> 24) & 0xFFFFFF
0

__init__ shouldn't be throwing bounds errors, there should be a mask keeping us in bounds... will need to expand tests for regression as well

I am working on a program which automatically adds an eventlistener to any given csgo .bsp file. I have already made a progress on adding the entities to the bsp and the only thing left is to save the map to a new bsp file. I just wanted to know if any progress was made in the past 4 months ?

I made a small script to play around with bsp_tool:

from bsp_tool import load_bsp
from bsp_tool.branches.respawn.apex_legends import LUMP

file = "/home/lee/Code/Mine/Analysis/ApexLegends/Maps/KingsCanyon/mp_rr_canyonlands_staging.bsp"
bsp = load_bsp(file)
bsp.read_lump(LUMP.ENTITIES)

Note I do have the ...0000.bsp_lump.

But I'm unable to retrieve the entities lump:

$ python3 test.py
Loading mp_rr_canyonlands_staging.bsp (rBSP version 49)...
ERROR PARSING TEXDATA:
TEXDATA lump is an unusual size (400 / 36). Wrong engine branch?
Loaded  mp_rr_canyonlands_staging.bsp
Traceback (most recent call last):
  File "/home/lee/Code/Others/bsp_tool/test.py", line 6, in <module>
    bsp.read_lump(LUMP.ENTITIES)
  File "/home/lee/Code/Others/bsp_tool/bsp_tool/respawn.py", line 29, in read_lump
    self.file.seek(self.branch.lump_header_address[LUMP])
ValueError: seek of closed file

After this I'm going to probably export the vertices to see some point clouds (as you've done in lump_analysis.py).

Have you gotten chunks of the level to render with planes? Do you know how LOD is handled?

Shouldn't use setattr imo, as there is no type hinting on what sorts of lumps a bsp contains. https://github.com/snake-biscuits/bsp_tool/blob/2a3e1e7d10a6d94c7686b2799d3c7c2db898b4b6/bsp_tool/valve.py#L57

• [x] Tools:

  • Source SDK
  • Source SDK Base 2006
  • Source SDK Base 2007
  • [x] Level Editor(s):
    • Hammer
    • Hammer++ (TF2)
  • [x] Compiler(s): Source SDK
  • Documentation / Guides:
    • VDC: Team Fortress 2 Level Creation

• [x] Map(s):

  • [x] mapsrc/Hammer/Team Fortress 2/test2.vmf ->maps/Team Fortress 2/test2.bsp
    • [x] sources.txt crediting mapper(s) / contributor(s)
    • [x] compiled
    • [x] .bsp.info.txt file with bspinfo output
    • [x] tested
  • [x] mapsrc/Hammer/Team Fortress 2/test_displacement_decompile.vmf ->maps/Team Fortress 2/test_displacement_decompile.bsp
    • [x] sources.txt crediting mapper(s) / contributor(s)
    • [x] compiled
    • [x] .bsp.info.txt file with bspinfo output
    • [x] tested
  • [x] mapsrc/Hammer/Team Fortress 2/test_physcollide.vmf ->maps/Team Fortress 2/test_physcollide.bsp
    • [x] sources.txt crediting mapper(s) / contributor(s)
    • [x] compiled
    • [x] .bsp.info.txt file with bspinfo output
    • [x] tested
  • [x] mapsrc/Hammer++/Team Fortress 2/mp_lobby.vmf -> maps/Team Fortress 2/mp_lobby.bsp
    • [x] sources.txt crediting mapper(s) / contributor(s)
    • [x] compiled
    • [x] .bsp.info.txt file with bspinfo output
    • [x] tested

Active Mods / Modding Community (Optional):

• VDC

• [x] Tools:

  • [x] Level Editor(s): Trenchbroom
  • [x] Compiler(s): q2tools-220
  • Documentation / Guides:
    • TODO: see #30 for now

• [x] Map(s):

  • [x] Basic room with spawnpoint & lighting mapsrc/TrenchBroom/Quake/mp_lobby.map -> maps/Quake 2/mp_lobby.bsp
    • [x] sources.txt crediting mapper / contributor
    • [x] Compiled
    • [x] .bsp.info.txt file with compile log
    • [x] Tested

Active Mods / Modding Community (Optional):

• TODO: see #30 for now

• [x] Tools:

  • [x] Level Editor(s): Trenchbroom
  • [x] Compiler(s): q2tools-220
  • Documentation / Guides:
    • TODO: see #30 for now

• [x] Map(s):

  • [x] Basic room with spawnpoint & lighting mapsrc/TrenchBroom/Quake/mp_lobby.map -> maps/Quake/mp_lobby.bsp
    • [x] sources.txt crediting mapper / contributor
    • [x] Compiled
    • [x] .bsp.info.txt file with compile log
    • [x] Tested

Active Mods / Modding Community (Optional):

• TODO: see #30 for now

Thanks Fifty.

struct CellAABBNode_t {
    Vector3f  mins;
    uint32_t  num_children: 8;
    uint32_t  first_child: 16;
    uint32_t  child_flags: 8;
    Vector3f  maxs;
    uint32_t  num_obj_refs: 8;
    uint32_t  first_obj_ref: 16;
    uint32_t  obj_ref_flags: 8;
};

Looks like respawn structs use bitfields a lot This makes sense with the "Extreme SIMD" method of vec4(a.xyz, a.metadata)

valve.source.Leaf also uses a bitfield at present we just write a getter & setter property for each member

This means CellAABBNode would need 12 property methods Automating this would minimize errors & reduce bloat

Duplicate keys in entities are lost just like in QtPyHammer-devs/vmf_tool#13

UPDATE: This isn't actually a bug, turns out I forgot the code handles this case and didn't check before creating the issue :P Still, an update would be nice

Try handling this like https://github.com/pySourceSDK / ValveVMF Where instead of a dict a list of [(key, value)] tuples store all keys without duplicates

This will require:

• [ ] New parsing approach
• [ ] Refactor of Entities.search
  • [ ] Refactor all bsp methods (in branch scripts, e.g. worldspawn_volume) that touch ENTITIES lumps
    • TODO: list all such methods here so we can track progress
• [ ] Re-compiling test/maps/Team Fortress 2/test2.bsp to match QtPyHammer-devs/vmf_tool's test2.vmf
• [ ] A test that uses this test map to confirm this issue has not reappeared

NOTE: io_import_rbsp will also need updating to support these changes

Extending

This might also be a good opportunity to use ValveVMF's parser on values And even using ValveFGD & custom classes for entities However this would likely require some re-engineering (of ValveVMF) to allow feeding the system flat nodes (unnamed curly brace groups) Not to mention some extra details to the parser, and maybe a deeper look at .fgd generation

NOTE: Many source engine titles have .fgd files available via the VDC: link

Maybe extenstions/fgd.py or bsp_tool_examples would be a place to try this out It would be preferable for the core lib to continue to function without any external dependencies ValveVMF parsing can also be quite slow, so deferring it's use until needed would be ideal

• [x] Tools:

  • [x] Level Editor(s):
    • Hammer
    • Hammer++ TODO: figure out setup
  • [x] Compiler(s): Contagion SDK (comes with Contagion)
  • Documentation / Guides:
    • VDC: CS:GO Level Creation

• [ ] Map(s):

  • [ ] Basic room with spawnpoint & lighting
    • [ ] Entry insources.txt crediting mapper / contributor
    • [ ] Compiled
    • [ ] .bsp.info.txt file with compile log (.log in mapsrc)
    • [ ] Tested

Active Mods / Modding Community (Optional):

• VDC

NOTE: monochrome.contagion branch is currently no different from valve.left4dead2 (at current level of understanding)

~~The Source Engine definitions for ddispinfo_t mess with how C/C++ packs structs to create some oddly aligned structures~~

~~This means we can't use the Python struct module to define the whole structure, so we'll have to tack extra .from_bytes & .to_bytes methods on~~

~~It's going to get extremely messy but properly dealing with displacements is likely going to be the most complicated thing we ever do, besides advanced bitfields...~~

Displacement Neighbour sub-structs aren't mapped yet

Look at #74 & #73 for arguments for using inheritance to simplify this nightmare (valve.source.DisplacementInfo can do the black magic and the others can copy)

We will also need Test Maps (#16) with at least one Displacement for #63, #38, #40 & #72

4 DisplacementInfo variants to finish up:

• [x] valve.source
  • [x] Test map: tests/maps/Team Fortress 2/test2.bsp
  • [x] Test map: tests/maps/Team Fortress 2/test_decompile_displacement.bsp
• [ ] chaos_initiative.chaos
  • [ ] Test map: TODO
• [ ] nexon.vindictus
  • [ ] Test map: TODO
  • [ ] nexon.cso2_2018
    • [ ] Test map: TODO

NOTE: Team Fortress 2 test maps are for valve.orange_box, but they use the valve.source DisplacementInfo spec

~~Current DisplacementInfo implementations use base.Struct, which utilises __slots__, so no private members can be utilised afaik Stick to _classes if possible~~

Enjoy the pain

Many branch_scripts are for engine changes that just expand limits by increasing the size of indexing members Some of these changes were already performed in older branches e.g. id_software.quake_remake_old appears to be the first to update quake.Edge from uint16_t -> uint32_t

Lots of branches likely do this We'll need to collect identically named branches (plus v2 etc. variants) & diff .as_cpp() cpp should be more reliable for the type footprint, since member names & definitions are still being unified (see #73)

This should reduce the overall scale of the code base and help reduce those duplicates with largely copy-pasted definitions

LumpClasses end up re-declaring & copying documentation from their parents a lot, as it's largely intended for each branch_script to serve as documentation, as much as code

Though the LUMP_CLASSES etc. inheritance flies in the face of that a little... docs/supported somewhat remedies this At this point in time I still think LumpClasses should only be loosely dependant on their parents. though simplifying could help... If .as_cpp() collected comments from the annotations / type-hints we could probably simplify a lot of LumpClasses

This will involve some light refactoring

A bunch of lump names differ across different branch scripts Many of these lumps share functionality, and even structs in some cases Unifying these different lumps to singular names with clearly conveyed use & context would really help documentation

This could also hugely help research & tracing the iteration of various small engine concept (e.g. .bsp geometry rendering)

I've already started doing a little of this and am already seeing some Quake 3 -> MoH:AA -> CoD -> Titanfall connections

Refer to the new "Etymology" page on the wiki for more info on the rationale behind this

TL;DR: make more code the same so we write less code and don't get confused as frequently

• [x] Tools:

  • Momentum Mod Playtest 0.9.4 [there is a waiting list]
  • [x] Level Editor(s):
    • Hammer
    • Hammer++ (CS:GO) Setup Guide
  • [x] Compiler(s): Source SDK, BSPConversionLib
  • Documentation / Guides:
    • VDC: CS:GO Level Creation
    • Momentum Mod Docs

• [x] Map(s):

  • [x] Basic room with spawnpoint & lighting mapsrc/Hammer/Momentum Mod/mp_lobby.vmf -> maps/Momentum Mod/mp_lobby.bsp
    • [x] Entry insources.txt crediting mapper / contributor
    • [x] Compiled
    • [x] .bsp.info.txt file with compile log (.log in mapsrc)
    • [x] Tested
  • [ ] At least 1 Displacement
    • [ ] Entry in sources.txt crediting mapper / contributor
    • [ ] Compiled
    • [ ] .bsp.info.txt file with compile log (.log in mapsrc)
    • [ ] Tested

NOTE: converted q3/chronic w/ BSPConversionLib --newbsp & it threw no loading errors, unsure if that was a fluke quake3 conversion seems to break texture projection, might look into fixes

Active Mods / Modding Community (Optional):

• VDC
• MomentumMod

@F1F7Y over at MRVN-radiant/MRVN-radiant figured out plane indexing Time to revisit brush slicing https://github.com/MRVN-radiant/MRVN-radiant/blob/main/tools/remap/source/bspfile_titanfall.cpp#L607-L629

Also seems like the discard/ignore plane flags are linked to ray collision, but not player collision

Still doesn't solve model brush indexing for r2 so put in a lazy fix for that edge case so we can get sliced brush geo