srctools.binformat

The binformat module binformat contains functionality for handling binary formats, esentially expanding on struct’s functionality.

Constants

srctools.binformat.ST_VEC: struct.Struct = Struct('fff'): A struct.Struct with three floats, for unpacking Vec, Angle, etc.

srctools.binformat.SIZES: dict[str, struct.Struct]: A dict mapping each fixed-size number format character (i, L, h, etc) to the size of the data.

srctools.binformat.SIZE_CHAR = 1

srctools.binformat.SIZE_SHORT = 2

srctools.binformat.SIZE_INT = 4

srctools.binformat.SIZE_LONG = 8

srctools.binformat.SIZE_FLOAT = 4

srctools.binformat.SIZE_DOUBLE = 8: The size of each of these standard numeric types.

Structure tools

class srctools.binformat.DeferredWrites(file: FileWBinarySeek)

Several formats require offsets or similar data to be written referring to data later in the file.

Doing this in one pass would be quite complicated, but this class assists in writing such files. Initially null bytes are written in the slots, then the data is filled in at the end.

To use this class, initialise it with the open and seekable file. Call defer() when reaching the relevant parts of the file, passing a format string for the structure and a hashable key used to identify it later. Once the value has been determined, call set_data() to store the data. When the file is written out, call write() which will seek() back and fill in the values.

defer( key: Hashable, fmt: str | Struct, write: bool = False, ) → None

Mark that data of the specified format is going to be written here.

Parameters:

key – Any hashable object, used to identify this location later.
fmt – The structure of the data, either as an existing struct.Struct instance, or a string in the same format.
write – If true, write null bytes to occupy the space in the file. Set to false if you are doing this yourself.

set_data( key: Hashable, *data: int | str | bytes | float, ) → None

Specify the data for the given key.

Parameters:

key – Any hashable object, used to identify the location and format.
data – The values passed to struct.pack() to build the data.

pos_of(key: Hashable) → int

Return the previously marked offset with the given key.

Parameters:: key – Any hashable object, used to match this call to the earlier defer() call.

write() → None: Write out all the data. All values should have been set.

srctools.binformat.struct_read( fmt: Struct | str, file: FileR[bytes], ) → tuple[Any, ...]: Read a structure from the file, automatically computing the required number of bytes.

srctools.binformat.str_readvec(file: FileR[bytes]) → Vec: Shortcut to read a 3-float vector from a file.

srctools.binformat.read_array(fmt: str | Struct, data: bytes) → list[int]

Read a buffer containing a stream of integers.

The format string should be one of the integer format characters, optionally prefixed by an: endianness indicator. As many integers as possible will then be read from the data.

srctools.binformat.write_array(fmt: str | Struct, data: Collection[int]) → bytes

Build a packed array of integers.

The format string should be one of the integer format characters, optionally prefixed by an endianness indicator. The integers in the data will then be packed into a bytes buffer and returned.

srctools.binformat.read_nullstr( file: FileR[bytes], pos: None = None, encoding: str = 'ascii', ) → str

srctools.binformat.read_nullstr( file: FileRSeek[bytes], pos: int | None, encoding: str = 'ascii', ) → str

Read a null-terminated string from the file.

If the position is 0, this will instead immediately return an empty string. If set to any other value this will first seek to the location.

srctools.binformat.read_nullstr_array( file: FileR[bytes], count: int, encoding: str = 'ascii', ) → list[str]

Read the specified number of consecutive null-terminated strings from a file.

If the count is zero, no reading will be performed at all.

srctools.binformat.read_offset_array( file: FileRSeek[bytes], count: int, encoding: str = 'ascii', ) → list[str]

Read an array of offsets to null-terminated strings from the file.

This first reads the specified number of signed integers, then seeks to those locations and reads a null-terminated string from each.

Array Insertion

In files like BSPs, data is often stored in a large array, then other sections refer to specific indexes. These functions help in constructing such an array, while reusing entries to avoid unnecessary duplication. find_or_insert inserts a single item, while find_or_extend inserts a subsequence, returning the offset.

srctools.binformat.find_or_insert( item_list: list[T], key_func: Callable[[T], Hashable] = id, ) → Callable[[T], int]:

Create a function for inserting items in a list if not found.

This reuses existing values inside the list to reduce duplication, or allow preserving order from existing files. If the provided argument to the callable is already in the list, the index is returned. Otherwise, it is appended and the new index returned.

Parameters:

item_list – The list to insert into.
key_func – If provided, this returns a key used to match existing items.

Returns:

A function which can be called with items. The returned index is the location that value can be found at inside the list.

srctools.binformat.find_or_extend( item_list: list[T], key_func: Callable[[T], Hashable] = id, ) → Callable[[list[T]], int]:

Create a function for positioning a sublist inside the larger list, adding it if required.

This reuses existing values inside the list to reduce duplication, or allow preserving order from existing files.

Parameters:

item_list – The list to insert into.
key_func – If provided, this returns a key used to match existing items.

Returns:

A function which can be called with a list of items. The returned index is the offset that the subsequence can be found at inside the list.

Checksumming

srctools.binformat.checksum(data: bytes, prior: int = 0) → int

Compute the VPK checksum for a file (CRC32).

Pass a previous computation to allow continuing a previous checksum.

srctools.binformat.EMPTY_CHECKSUM = 0: The checksum value of an empty bytes buffer (b"").

LZMA (de)compression

srctools.binformat.decompress_lzma(data: bytes) → bytes: Decompress LZMA-encoded data, using the settings Source uses in BSP lumps.

srctools.binformat.compress_lzma(data: bytes) → bytes: Compress data using the LZMA algorithm and the settings Source uses in BSP lumps.