Yeah, the actual collision is static and can't change. In RV the collision meshes for the all 3 states of the buildings seemed to be compromises that didn't exactly match any of them visually and tried to prevent players from clipping through geometry without being too obvious about it. Most geometry in RV was designed to be very similar no matter which version is visible; it's mostly just banners and decorations that change.
It's not just a straight up combination of all the geometry, either. IIRC you could fly right through Recluse's cloak in the statue, but could stand on top of invisible spider arms when heroes controlled the area.
Also, if you visited RV after Ultra Mode went in, a sharp eye would spot shadows of spider arms and the arachnos logo on city hall that showed up even if the corresponding geometry was invisible. I think I /bug reported it once, but given how the engine works I don't think it would have been easy to fix.
Loading up the map in Icon results in all of the geometry being visible at once, but you can fly around and see that the collision doesn't always match what you see.