ArtLang: Articulation-Structured Language Fields Enable Language-Guided Articulation Reasoning and Manipulation

Anonymous Authors

Articulated-object reconstructions recover explicit geometry and kinematics, but their parts often remain semantically anonymous and must be controlled through part indices and numerical joint parameters. We present ArtLang, a framework for open-vocabulary language control of persistent reconstructed articulated assets. ArtLang represents an asset as a semantic-kinematic articulation graph and augments its surface with language features and graph-constrained motion. Open-vocabulary proposals are bound to reconstructed parts while allowing uncertain parts to remain unnamed. A typed parser converts a command into a directive graph containing referring expressions, actions, magnitudes, reference frames, and relations. We then solve a global graph-to-graph grounding problem that jointly reasons about semantic, spatial, relational, and kinematic compatibility, with support for null assignments and abstention under ambiguity. Accepted directives are converted into continuous joint targets within the observed motion range and executed through forward kinematics. Experiments on synthetic reconstructions, mesh-based assets, and real captures demonstrate reliable language grounding and continuous articulated control across repeated parts, spatial references, relational commands, and ambiguous instructions.

Method

ArtLang is a graph-to-graph pipeline from language to an articulated object's semantic and kinematic structure. (A) A part-aware reconstruction (ArtMesh backend) is represented as an articulation graph with a factorized semantic-kinematic field: every surface point carries its part identity, a distilled language feature, and the graph-constrained motion of its part. (B) Open-vocabulary features and VLM part proposals are bound to persistent parts by semantic-geometric partial matching, giving each part a noun and an FG-CLIP-2 anchor. (C) A typed parser turns the command into a directive graph, which is globally grounded against the articulation graph under semantic, spatial, relational, and action compatibility, with support for null assignments and abstention under ambiguity. (D) Accepted directives are converted to calibrated joint targets inside the observation-supported interval and executed by forward kinematics.

Challenge: Semantic & Spatial Ambiguity

Articulated objects repeat their parts: a cabinet has several doors and drawers that share one noun, so a phrase like “the door” is ambiguous on its own — language must resolve which parts move, not just what kind. ArtLang treats this as graph-to-graph binding: the command is parsed into a directive graph of entities, qualifiers, and relations, and grounded onto the reconstructed articulation graph by a global assignment over semantic, spatial, relational, and action compatibility — so “close everything” actuates every movable part, while “close the left and right doors” selects exactly the two doors and leaves the drawer at rest.

semantic ambiguity: unable to distinguish part names

spatial ambiguity: unable to distinguish same type of parts

Language-Guided Articulation

Free-form commands are resolved into (part, joint coordinate) targets and executed as rigid motions — spatial qualifiers disambiguate same-noun parts (left / right / top), magnitude modifiers and explicit angles map onto the calibrated joint interval, multi-directive sentences ground globally, and relational references resolve against the graph. Each clip shows the command and the resulting actuation of the reconstructed model. (Articulate-100 objects are captured with state 0 open, so commands typically close a part.)

close the top drawer

close the left door halfway

open the right door

Continuous Language-Driven Actuation

The articulation coordinate supports continuous control: a directive selects a part and any value of the actuation parameter t places it along its learned trajectory — t = 0 is the rest state, t = 1 the learned closed state, and negative values extrapolate past the rest pose in the opening direction, through poses never seen at training. Each clip sweeps one part while all others stay at rest; because t scales the learned motion, the sweep doubles as a visual audit of the recovered kinematics.

per-part t-sweep, t ∈ [−1, 1] — Table 22301

per-part t-sweep, t ∈ [−1, 1] — StorageFurniture 45940

Results on Articulate-100

Given a mesh and a single-view image of the end state, ArtLang recovers the articulated object.

Oven 7187

StorageFurniture 45940

Table 31249

Reconstruction

Reconstructing from multi-view observations of the two states, ArtLang matches its ArtMesh backbone on joints and geometry — attaching the language handle costs no reconstruction quality.

StorageFurniture 45612

Foldingchair 100523

Eyeglasses 101293

Real-World Captures (SplArt)

The same pipeline applied to real SplArt objects: despite noisier geometry and less clean boundaries than the synthetic set, the semantic and spatial axes still resolve the intended part, and the motion axis actuates it along the recovered joint — a user can address a captured object without touching a joint index.

Real-world objects from SplArt. The three-axis handle grounds and actuates parts on captured objects where boundaries are less clean than in the synthetic benchmarks. As these are real captures, there is neither a ground-truth mesh nor the original capture lighting, so the lighting and color of every method's renders may look off relative to the reference photo.

Citation

@article{artlang2026,
  title   = {ArtLang: Articulation-Structured Language Fields Enable
             Language-Guided Articulation Reasoning and Manipulation},
  author  = {Anonymous Authors},
  journal = {Under review},
  year    = {2026}
}