Spatial Reasoner

Turning scene geometry into spatial facts

XR frameworks can detect and track planes, objects, and scene geometry, but application logic usually receives positions and extents rather than statements such as on, behind, near, left, or touching. Spatial Reasoner provides that intermediate layer.

Each spatial object represents a detected or virtual entity using a stable identifier and an oriented 3D bounding box. The minimal description consists of a position at the center of the box’s base, its width, height and depth, and its yaw around the vertical axis. From this, the reasoner derives attributes such as footprint, volume and movement state, then evaluates relations between objects.

This was an unfunded project that I built together with Philipp Ackermann at ZHAW. We also co-authored the resulting conference paper.

Reference frames and spatial sectors

Terms such as left, ahead, and in front only make sense relative to a reference frame. The specification therefore distinguishes between world, object, egocentric and geodetic coordinate systems, and defines where each predicate is valid.

For directional reasoning, the space around a reference object is divided into a 3 × 3 × 3 grid. The resulting 27 sectors encode an inner sector, the six principal directions, combinations of two directions, and combinations of three directions. For example, ar means ahead-right and bru means behind-right-under. Fixed, dimension-based and nearby sector schemes allow the partition to be adjusted to the scale of a scene.

Single, double and triple sector divergence around the reference object.

Predicates and relation graphs

The framework defines predicates for proximity, directionality, adjacency, assembly, connectivity, sector membership, visibility, comparability, orientation, similarity and geography. Relations use a subject-predicate-object representation and are derived from the bounding-box geometry, configurable tolerances and the applicable reference frame.

The resulting relation graph can be inspected directly or used by later pipeline stages. Relations can also carry metric values such as distance or angle differences, which makes it possible to sort objects by more than a boolean match.

A partial relation graph generated by the reasoner.

Inference as a pipeline

Inference is specified as text, with operations connected by the pipe character. A pipeline can adjust thresholds, deduce selected relation categories, filter objects, follow relations, sort results, assign types, calculate aggregate values, generate new spatial objects, or write diagnostic output.

filter(id == 'user')
| pick(disjoint)
| sort(disjoint.delta <)
| slice(1)

This query starts from the observer, follows disjoint relations, sorts the related objects by distance, and returns the nearest one. More involved pipelines can classify an object from its dimensions and topology, query an application-specific OWL/RDF taxonomy with isa, or produce contextual objects. One example from the paper creates a corner object where two walls meet.

Implementations and inspection

The same pipeline definitions are used by the open-source Swift, Python, and C# implementations. This allows XR clients and server-side processes to use the same spatial rules without translating them into a different API for each platform. The C# implementation also has Unity bindings.

The log operation can emit JSON, Markdown with Mermaid relation graphs, or a 3D scene for inspection. These outputs are useful for checking which relations were inferred and why.

A generated connectivity graph for a small room scene.

Research output

Philipp Ackermann and I published the work as Spatial Reasoner: A 3D Inference Pipeline for XR Applications at the 2025 International Conference on Intelligent Computing and Virtual & Augmented Reality Simulations (ICVARS). The paper is available through IEEE Xplore and via DOI 10.1109/ICVARS66454.2025.11198690. The implementation and specification are available on GitHub.

The AR Patterns project addresses a related communication problem: describing event-driven AR behavior before committing to a platform-specific implementation.