ArtLang: Structured Language-to-Kinematics Grounding for Articulated 3D Actuation

Anonymous Authors

Articulated-object reconstructions recover explicit geometry and kinematics, but their parts often remain semantically anonymous and must be controlled through part indices and numerical joint parameters. We present ArtLang, a framework for open-vocabulary language control of persistent reconstructed articulated assets. ArtLang represents an asset as a semantic-kinematic articulation graph and augments its surface with language features and graph-constrained motion. Open-vocabulary proposals are bound to reconstructed parts while allowing uncertain parts to remain unnamed. A typed parser converts a command into a directive graph containing referring expressions, actions, magnitudes, reference frames, and relations. We then solve a global graph-to-graph grounding problem that jointly reasons about semantic, spatial, relational, and kinematic compatibility, with support for null assignments and abstention under ambiguity. Accepted directives are converted into continuous joint targets within the observed motion range and executed through forward kinematics. Experiments on synthetic reconstructions, mesh-based assets, and real captures demonstrate reliable language grounding and continuous articulated control across repeated parts, spatial references, relational commands, and ambiguous instructions.

Challenges

Addressing an articulated object in language raises three challenges. 1. Repeated parts: a cabinet has several doors and drawers that share one noun, so “the door” is ambiguous on its own. 2. Spatial ambiguity: same-noun parts must be told apart by position or relation — is “the drawer below the top drawer” the second drawer? 3. Motion semantics: open, close, and halfway must map onto each part's actual joint and calibrated motion range. ArtLang treats all three as one graph-to-graph binding: the command is parsed into a directive graph of entities, qualifiers, and relations, and grounded onto the reconstructed articulation graph by a global assignment over semantic, spatial, relational, and action compatibility — so “close everything” actuates every movable part, while “close the left and right doors” selects exactly the two doors and leaves the drawer at rest.

repeated parts: several parts share one noun

spatial ambiguity: same-type parts told apart only by position

ArtLang Pipeline

From a mesh (or capture) and its observed states to a language-controllable asset. ArtLang is a graph-to-graph pipeline from language to an articulated object's semantic and kinematic structure. (A) A part-aware reconstruction (ArtMesh backend) is represented as an articulation graph with a factorized semantic-kinematic field: every surface point carries its part identity, a distilled language feature, and the graph-constrained motion of its part. (B) Open-vocabulary features and VLM part proposals are bound to persistent parts by semantic-geometric partial matching, giving each part a noun and an FG-CLIP-2 anchor. (C) A typed parser turns the command into a directive graph, which is globally grounded against the articulation graph under semantic, spatial, relational, and action compatibility, with support for null assignments and abstention under ambiguity. (D) Accepted directives are converted to calibrated joint targets inside the observation-supported interval and executed by forward kinematics.

Sample ArtLang Results

Free-form commands are resolved into (part, joint coordinate) targets and executed as rigid motions — spatial qualifiers disambiguate same-noun parts (left / right / top), magnitude modifiers and explicit angles map onto the calibrated joint interval, multi-directive sentences ground globally, and relational references resolve against the graph. Each clip shows the command and the resulting actuation of the reconstructed model. (Articulate-100 objects are captured with state 0 open, so commands typically close a part.)

close the top drawer

close the left door halfway

open the right door

Given only a mesh and a single-view image of the end state, ArtLang recovers the full articulated object — parts, joints, and the language handle.

Oven 7187

StorageFurniture 45940

Table 31249

Motion Extrapolation

The articulation coordinate supports continuous control beyond the observed states: a directive selects a part and any value of the actuation parameter t places it along its learned trajectory — t = 0 is the rest state, t = 1 the learned closed state, and negative values extrapolate past the rest pose in the opening direction, through poses never seen at training. Each clip sweeps one part while all others stay at rest; because t scales the learned motion, the sweep doubles as a visual audit of the recovered kinematics.

per-part t-sweep, t ∈ [−1, 1] — Table 22301

per-part t-sweep, t ∈ [−1, 1] — StorageFurniture 45940

Baselines vs. ArtLang

Commanded actuation compared against the ground truth and prior systems. Baselines either miss the referred part (spatially unaware grounding), cannot address the language at all, or actuate with an incorrect motion — ArtLang's global graph-to-graph assignment selects the right part and moves it along the right joint.

Baseline comparison: GT, Ours (mesh / multi-view / single-view), Articulate-Anything, OPD — Two-state actuation against baselines. Columns: ground truth, ArtLang from a mesh, from multi-view, and from a single view, Articulate-Anything, and OPD. Baselines frequently articulate the wrong part or move it along the wrong joint, while all three ArtLang input regimes match the ground-truth states.

Reconstruction

Reconstructing from multi-view observations of the two states, ArtLang matches its ArtMesh backbone on joints and geometry — attaching the language handle costs no reconstruction quality.

StorageFurniture 45612

Foldingchair 100523

Eyeglasses 101293

Real-World Captures (SplArt)

The same pipeline applied to real SplArt objects: despite noisier geometry and less clean boundaries than the synthetic set, the semantic and spatial axes still resolve the intended part, and the motion axis actuates it along the recovered joint — a user can address a captured object without touching a joint index.

Real-world objects from SplArt. The three-axis handle grounds and actuates parts on captured objects where boundaries are less clean than in the synthetic benchmarks. As these are real captures, there is neither a ground-truth mesh nor the original capture lighting, so the lighting and color of every method's renders may look off relative to the reference photo.

Citation

@article{artlang2026,
  title   = {ArtLang: Structured Language-to-Kinematics Grounding for
             Articulated 3D Actuation},
  author  = {Anonymous Authors},
  journal = {Under review},
  year    = {2026}
}