Speaking of scenegraphs, I think I'm almost ready to have a go at building a scenegraph in Panda3D. The initial levels, anyway.
The Kaitai parser is working fine. It's pretty cool, really, to write a few lines of code and have an entire geobin file load up, ready to be used.
My first little geobin parser looks like this:
from geobin import *
import psutil
import gc
print("Mem footprint: %fmb" % (psutil.Process().memory_info().rss/2**20))
print("Loading geobin...")
gb = Geobin.from_file("C:/Users/Scott/Downloads/tequila/Extracted_Full/geobin/maps/City_Zones/City_01_01/City_01_01.bin")
print("Mem footprint: %fmb" % (psutil.Process().memory_info().rss/2**20))
defs = gb.bin_data.def_array
num_defs = gb.bin_data.def_count
refs = gb.bin_data.ref_array
num_refs = gb.bin_data.ref_count
print("Num Defs: %d / Num Refs: %d" % (num_defs, num_refs))
print("Mem footprint: %fmb" % (psutil.Process().memory_info().rss/2**20))
print("Listing Refs")
for ref_index in range(0, num_refs, 1):
rb = refs[ref_index].ref_body
ref_name = rb.ref_str1.strval
ref_pos = (rb.ref_pos_x,rb.ref_pos_y,rb.ref_pos_z)
ref_o = (rb.ref_pyr_yaw, rb.ref_pyr_pitch, rb.ref_pyr_roll)
print("Refname:%s -- Pos%s Hpr%s" % (ref_name, ref_pos, ref_o))
print("Freeing geobin...")
del defs
del num_defs
del refs
del num_refs
del gb
print("Mem footprint: %fmb" % (psutil.Process().memory_info().rss/2**20))
and produced this:
PS C:\errbot.bogart\kaitai> python gb_test.py
Mem footprint: 16.636719mb
Loading geobin...
Mem footprint: 52.238281mb
Num Defs: 7239 / Num Refs: 33
Mem footprint: 52.238281mb
Listing Refs
Refname:grp_GC_Shutdown_Atmo -- Pos(-71.0, 15.931458473205566, -474.5) Hpr(0.0, 0.0, 0.0)
Refname:grp_audiogroup -- Pos(186.380859375, 67.989501953125, -938.831298828125) Hpr(0.0, 0.0, 0.0)
Refname:grp_Badge_tourism -- Pos(721.3763427734375, 164.83200073242188, -699.31494140625) Hpr(0.07853981852531433, 0.0, -0.0)
Refname:grp_beacons -- Pos(522.87060546875, 262.0234375, -574.75) Hpr(0.0, 0.0, 0.0)
Refname:grp_blackmap -- Pos(512.0, -128.0000457763672, -763.474609375) Hpr(0.0, 0.0, 0.0)
Refname:grp_blimp -- Pos(176.0, 707.0, -1040.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Day_Job_Volumes -- Pos(-59.5, -867.0, -914.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Geometry -- Pos(511.38037109375, 569.0, -703.999755859375) Hpr(0.0, 0.0, 0.0)
Refname:grp_Halloween_Event_06 -- Pos(-68.01048278808594, 4.472891807556152, -10.75) Hpr(0.0, 0.0, 0.0)
Refname:grp_Holiday_Event_06 -- Pos(511.4013671875, 409.2863464355469, -812.486572265625) Hpr(0.0, 0.0, 0.0)
Refname:grp_HollowsBeacons -- Pos(189.818359375, 245.71826171875, -2069.03076171875) Hpr(0.0, 0.0, 0.0)
Refname:grp_Makeover_Spawndefs -- Pos(642.7833251953125, 7.882775783538818, -797.4718017578125) Hpr(0.0, 0.0, 0.0)
Refname:grp_Manholes -- Pos(1235.75, 156.25277709960938, -242.75) Hpr(0.0, 0.0, 0.0)
Refname:grp_mission_doors -- Pos(120.030517578125, 465.505859375, -748.54931640625) Hpr(0.0, 0.0, 0.0)
Refname:grp_neighborhoods -- Pos(-32.0, 0.0, -351.9999694824219) Hpr(0.0, 0.0, 0.0)
Refname:grp_patrol_box -- Pos(-5.161986827850342, 18.90635108947754, -706.25) Hpr(0.0, 0.0, 0.0)
Refname:grp_PersistentNPC -- Pos(-25.9998779296875, -159.36572265625, -899.901611328125) Hpr(0.0, 0.0, 0.0)
Refname:grp_Plaques -- Pos(206.2913818359375, 24.575931549072266, -414.63629150390625) Hpr(0.0, 0.0, 0.0)
Refname:grp_PVP_Scripts -- Pos(140.43020629882812, 20.40468406677246, -241.53944396972656) Hpr(0.0, 0.0, 0.0)
Refname:grp_Rikti_Invasion_Event -- Pos(2260.0, 0.0, 3388.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_RiktiBetaEvent -- Pos(-83.82254028320312, 841.516357421875, -643.6749877929688) Hpr(0.0, 0.0, 0.0)
Refname:grp_ShadowShardInvasion -- Pos(511.4013671875, 409.1253356933594, -812.486572265625) Hpr(0.0, 0.0, 0.0)
Refname:grp_Truck_Door -- Pos(340.6702880859375, 135.29078674316406, -1199.5) Hpr(0.0, 0.0, 0.0)
Refname:grp_warpvolumes -- Pos(-1664.0, -896.0, 1408.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Winter_Event -- Pos(511.4013671875, 409.2863464355469, -812.486572265625) Hpr(0.0, 0.0, 0.0)
Refname:grp_HalloweenEvent_09 -- Pos(1199.0, 41.59211349487305, -1189.7000732421875) Hpr(0.0, 0.0, 0.0)
Refname:grp_ParagonCityTeleport -- Pos(-33.21428680419922, 0.0, -171.35714721679688) Hpr(0.0, 0.0, 0.0)
Refname:grp_Mission_Phase -- Pos(637.5, -0.4998469948768616, 67.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Mission_Trays -- Pos(2450.0, 56.017494201660156, -1372.5) Hpr(0.0, 0.0, 0.0)
Refname:grp_spawndefs -- Pos(772.814208984375, -16.52880859375, -699.10009765625) Hpr(0.0, 0.0, 0.0)
Refname:grp_Zowies -- Pos(2442.5, 57.2479248046875, -1374.5) Hpr(0.0, 0.0, 0.0)
Refname:grp_GalCty_PerstntNPC -- Pos(391.30340576171875, 68.48408508300781, -864.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Kraken_Event -- Pos(1354.25, -32.00006103515625, -736.5) Hpr(0.0, 0.0, 0.0)
Freeing geobin...
Mem footprint: 52.238281mb
PS C:\errbot.bogart\kaitai>
My concern at the moment is this bit: "Freeing geobin...Mem footprint: 52.238281mb". Freeing the geobin parser didn't actually free the memory it used, probably because the underlying Kaitai structures are still referencing the data. There's not a lot I can do about that, unless I want to roll my own parser and handle the memory management myself, which I don't. By itself, 52mb isn't any big deal. I'm more concerned with what happens when I start going into the object library and opening up dozens of geobins associated with object library pieces. I guess I'll cross that bridge when I come to it. Most of the low level Kaitai stuff is undocumented, and there may be some method it has to free up its memory that I just don't currently know about.
That aside, it's handy that the Kaitai parser loads up the entire geobin file at once. I've pretty much verified that the "right" way to deserialize the scenegraph is to do it in reverse, with respect to the way the data is stored in the geobin file. I could build it up leaf-to-branch-to-root but it should be easier and less error-prone to just start at the end of the refs array and the defs array and move backwards through them, building the tree naturally from the root to the leaves.
The refs are the first-level children of the root. Everything else depends from there. Bindump confirms that moving backwards from array-end to array-start goes like this:
-------- Defs 6055 --------
Name = grp_worldlink_beacons
Group =
-------- Group 0 --------
Name = grp6676
Pos =
X = -1283.57
Y = -31.8641
Z = 311.098
PYR =
Pitch = 0
Yaw = 0
Roll = 0
Flags = 0
-------- Group 1 --------
Name = grp1745
Pos =
X = -309.435
Y = -33.1197
Z = 1652.77
PYR =
Pitch = 0
Yaw = 0
Roll = 0
Flags = 0
Property =
-------- Property 0 --------
u034 = Layer
u035 = worldlink_beacons
u036 = 0
TintColor = []
Ambient = []
Omni = []
Cubemap = []
Volume = []
Sound = []
ReplaceTex = []
Beacon = []
Fog = []
Lod = []
Type =
Flags = 0
Alpha = 0
Obj =
TexSwap = []
SoundScript =
"Name" is the name of the parent node. The Group array represents the children to add to that parent. The other arrays likewise are attributes that either apply directly to the parent or that are attached as children to the parent.
Here's our bindump excerpt for the next two defs, moving backwards:
-------- Defs 6053 --------
Name = grp1744
Group =
-------- Group 0 --------
Name = Omni/MissionBeacons/_PlayerSpawn
Pos =
X = 0
Y = 0
Z = 0
PYR =
Pitch = 0
Yaw = -3.14159
Roll = 0
Flags = 0
Property =
-------- Property 0 --------
u034 = SpawnLocation
u035 = LinkFrom_City_01_02
u036 = 0
TintColor = []
Ambient = []
Omni = []
Cubemap = []
Volume = []
Sound = []
ReplaceTex = []
Beacon = []
Fog = []
Lod = []
Type =
Flags = Ungroupable (1)
Alpha = 0
Obj =
TexSwap = []
SoundScript =
-------- Defs 6054 --------
Name = grp1745
Group =
-------- Group 0 --------
Name = grp1384
Pos =
X = 43.5209
Y = -421.523
Z = -24.6548
PYR =
Pitch = 0
Yaw = -2.79253
Roll = 0
Flags = 0
-------- Group 1 --------
Name = grp1388
Pos =
X = 43.0209
Y = -421.523
Z = -9.65479
PYR =
Pitch = 0
Yaw = -3.14159
Roll = 0
Flags = 0
-------- Group 2 --------
Name = grp1403
Pos =
X = -52.4791
Y = -421.523
Z = -9.15479
PYR =
Pitch = 0
Yaw = 2.96706
Roll = 0
Flags = 0
-------- Group 3 --------
Name = grp1617
Pos =
X = -38.9791
Y = -421.523
Z = -25.1548
PYR =
Pitch = 0
Yaw = 2.61799
Roll = 0
Flags = 0
-------- Group 4 --------
Name = grp1618
Pos =
X = -39.9791
Y = -421.523
Z = -9.65479
PYR =
Pitch = 0
Yaw = -3.14159
Roll = 0
Flags = 0
-------- Group 5 --------
Name = grp1619
Pos =
X = -4.47913
Y = -422.023
Z = -1.15479
PYR =
Pitch = 0
Yaw = -3.14159
Roll = 0
Flags = 0
-------- Group 6 --------
Name = grp1744
Pos =
X = 43.0209
Y = -421.523
Z = -33.1548
PYR =
Pitch = 0
Yaw = -2.61799
Roll = 0
Flags = 0
Property = []
TintColor = []
Ambient = []
Omni = []
Cubemap = []
Volume = []
Sound = []
ReplaceTex = []
Beacon = []
Fog = []
Lod = []
Type =
Flags = 0
Alpha = 0
Obj =
TexSwap = []
SoundScript =
Sure enough - Def 6054 is the last child of Def 6055, and Def 6053 is the last child of Def 6054. The pattern repeats until we hit a leaf; typically an object library piece. Then the "de-traversal" moves back up the tree and follows the next branch to its end and so on.
So, as long as I've got a handle on creating appropriate Panda3D NodePaths and attaching them together correctly, the scenegraph pretty much writes itself, at least up to the point where the object library and the actual geometry come into play.
The object library geobins are technically identical in format to the map geobins, except that they don't have refs, and they have a corresponding .bounds file that needs to be attached to the scene as part of the leaf data. The one unusual thing is that "leaf" nodes in an object library have TWO terminating records. One is a record with the object name and an Object attribute as a child. The last is an empty node with the object's name as the parent. Presumably, the former is meant to indicate that it contains geometry and the latter is intended to have the geometry attached to it. At least, that's my best guess at the moment.
Despite what I said about the scenegraph "writing itself", there are a lot of attributes that I'm not sure what to do with or that I'm considering disposing of because I DON'T care about them. Sounds, or Level of Detail (LOD) nodes, for instance. The real game client cares, of course, but a bot doesn't care; at least not in any way I'm currently envisioning. If I end up having to define a bunch of custom nodes to handle that stuff, or the memory footprint becomes unwieldy, then it might be easier to dispose of it. I guess, like other things, that I cross that bridge when I come to it. I'm trying to keep in mind that if I do this thing "right", that some other person may find a use for these details even if they aren't particularly useful to me at the present time.