Text2Sim
Research in progressFrom language to worlds we can test.
I am building a text-driven scene system in CyberWorld: turning room descriptions into structured layouts, assembling reusable 3D assets, and checking them in Unreal Engine before exporting aligned visual observations and geometry.
Playback is unavailable here. Download the original WebM to view it locally.
Six types now include meeting, reading and storage rooms. Model counts refer to the nine-room R2J static pilot; the original synchronized-data batch remains separate.
A robot enters the world we built.
The scene now hosts a Unitree G1. Starting with the official-source model in a standing pose, I integrated its articulated visual parts into the living room and created a short approach to the coffee table, with alternating steps and a final turn toward the table.
One robot entity, 35 visual parts. The environment library remains at 50 source models / 21 categories. My Unitree G1 practice ↗
A scene is more than an image.
For robotics and perception, a useful scene needs more than a convincing appearance. It needs geometry, scale, support relations, navigable space and annotations that agree with what the camera actually sees.
Start with intent.
A text description specifies the room, openings, furniture and spatial constraints. A language model produces a semantic room graph; a geometry-aware solver turns those proposals into a concrete assembly.
Keep the world measurable.
Modular architecture is combined with original-source assets and PBR materials. Native Unreal checks then verify the selected scene, while capture preserves one physical state across RGB, depth, masks and poses.
Language
Room dimensions, objects, style and relations
Semantic graph
Structured proposals and explicit constraints
3D assembly
Blender architecture and reusable real assets
Native simulation
UE Chaos, support, clearance and rendering
Aligned data
RGB-D, instance IDs, intrinsics and 6D poses
More rooms. More ways to furnish them.
Explore 15 new static rooms with 30 actual UE views, from shared meeting tables to bookshelves and storage aisles. The original text-generated examples are also available in their own collection.
Dining room
- Source models
- Source instances
- Material preset
Inspect the recorded composition
Built to stand, fit and be seen.
The assembly is checked against actual geometry. Rendering is then inspected with controlled, same-camera comparisons.
Flat architectureOriginal PBRActual paired UE captures: fixed component camera and physical state. Architecture-material ablation only; furniture keeps its original materials.
Source scale & orientation
Real objects retain source scale 1.0. Asset-specific facing rules keep chairs directed toward their actual tables. The completed batch verifies 802 source instances and 169 chair-facing checks.
Support, stability & clearance
Native triangle queries check support surfaces; Chaos verifies the designated dynamic objects. Enclosure and continuous 70 cm capsule sweeps check room closure and a usable route.
Inspectable failure & repair
The 100 accepted rooms follow 116 native attempts, with 16 failures retained. A bounded repair ledger records changes before simulation; final poses are not manually corrected after settling.
One physical state. Aligned outputs.
Switch between modalities from one recorded frame. Each accepted scene contributes a close and a context view, sharing the same measured scene state.
Original R2E batch, including logged repairs. The new R2I / R2J static rooms add RGB views, with no new Chaos runs or GT frames.
r2e_001 / full_close_v1 · 640 × 480
Rendered RGB
Inspect one actual camera / pose record
Translations are in meters. Axes use the preserved source canonical frame; category-level or human canonical approval is a separate step. Depth colors, axes and 3D boxes are derived from recorded data. Boxes use native mesh-local bounds; dashed edges indicate the back of each box, not depth-tested occlusion.
More variety, from real source assets.
The expanded library is now being exercised in rooms. Nine furniture-diversity scenes use 35 source models across 18 categories and 97 instances: dining tables, desks, coffee tables, five chair models, and objects such as clocks, plates, teapots and plant pots.
Actual Blender Cycles previews of selected admitted source assets.
See all nine table models in their rooms
Three dining tables, three desks and three coffee tables, in actual UE captures.
View all 21 categories and model counts
Does the image match the language?
I evaluate visual-text consistency separately from geometric correctness. Current CLIP experiments distinguish coarse room retrieval, object-category retrieval and scene-level category presence.
The following results use the frozen original R2E corpus and its 11-model palette. The 15 new static rooms have not received new retrieval scores or GT annotations.
Room family + style R-Precision
R2G · CLIP ViT-B/32
R-Precision = relevant scenes in top-R / R
6 fixed English queries · 100 scene candidates · 4 views per scene · R = 16 or 17. Random expectation: 16.67%.
This tests coarse family/style labels. It does not establish fine-grained object counts, spatial relations or the full-text 85% research target.
What changes when we look at objects?
A paired diagnostic compares whole-frame input against RGB crops located using ground-truth instance masks. Use the tabs to inspect the different tasks.
Evaluation protocol, full numbers and limits
| R2H task | Whole frame | GT-box crop | Change (pp) |
|---|---|---|---|
| Object R-Precision · macro 11 | 15.30% | 76.14% | +60.84 |
| Object Top-1 · micro | 14.57% | 79.82% | +65.25 |
| Scene R-Precision · macro 7 | 76.44% | 74.43% | −2.01 |
R2H uses 200 original 640 × 480 GT RGB frames, 659 visibly eligible objects and 1,271 crops. The paired whole-frame baseline uses letterboxing and the same eligible views. There is one source model per class across 11 object classes; unseen-model generalization is untested.
Scene presence retrieval keeps invisible positives. Four universally present classes are excluded from its primary seven-class mean. R2G and R2H use different views, queries and tasks and should not be treated as a before/after comparison. Protocols were frozen before scoring.
R2G family-only auxiliary R-Precision is 89.96% (3 queries, random expectation 33.33%). This auxiliary result is not the complete 85% text-to-scene target. Sim-to-real training improvement remains unverified.
From one room to a broader scene collection.
The checked generation pipeline now sits alongside pilots for new functions and furniture. The next step is to bring this wider variety into the planner, synchronized capture and semantic evaluation.
R2A → R2B
An enclosed room and reusable components
Build a complete dining room, correct chair facing, preserve original object scale, and qualify reusable architecture and source assets.
R2C → R2D
Structured layouts and synchronized observations
20 development prompts produce 18 positive layouts and 2 correct infeasibility rejections. The 18-room pilot exports 36 full-scene GT frames.
R2E · 100
A bounded, audited 100-scene run
Complete native scene checks, 400 beauty views and 200 synchronized frames, with all failures and repair records retained. Expansion to 1,000 is on hold pending storage capacity and further diversity work.
R2F → R2H
Asset diversity and visual consistency
Expand to a 50-model static library, inspect three enriched rooms, and diagnose room- and object-level CLIP retrieval on the original frozen corpus.
R2I · 6
Meeting, reading and storage rooms
Three new functional types, each in two material presets. Six static rooms and 12 UE views retain original source scale and pass native support, enclosure and route checks.
R2J · 9
Furniture variety becomes visible
Nine rooms and 18 new UE views exercise 35 source models across 18 categories. Five chair models and nine table models keep their source geometry and materials, with facing measured for each seating model. These pilots add no new Chaos runs, GT or retrieval scores.
R2K → R2L
G1 joins the scene, then approaches a table
Integrate the official-source G1 model, preserve its 29-joint hierarchy and original scale, then capture a 10.75-second kinematic approach in UE. Alternating steps, turns and the final facing pose are replayable in a native Level Sequence.
Built from traceable evidence.
Room imagery on this page comes from actual CyberWorld / Unreal Engine captures. Asset previews come from recorded Blender Cycles renders. Depth colors, mask colors, pose axes and 3D bounding boxes visualize original saved annotations. Scene pictures have only been resized and compressed for the web.
Asset sources: Amazon Berkeley Objects · Google Scanned Objects · Poly Haven. Powered by Poly Haven.
G1 source model: MuJoCo Menagerie / Unitree G1 · BSD-3-Clause. Derived from the Unitree Robotics model description; pinned source revision. The video is the original UE capture sequence encoded as WebM, with no generated or composited imagery.
Project snapshot: 29 October 2026. Results are tied to their stated corpus and protocol. This page is a research progress record; it makes no publication, benchmark-leadership or production-readiness claim.
Source model references and recorded licenses
Recorded model-level references for the 50-model library. Scene images combine these sources with authored architecture and materials.
Evidence behind this page
R2E: final_report_v1.json · R2F: final_report_v1.json / asset_catalog_v1.json · R2G: results_v1.json · R2H: summary_v1.json · R2I / R2J: final_report_v1.json / native_rooms_*.json / RGB_progress_v1.json / independent_verification_v1.json · R2K / R2L: final_report_v1.json / media_verification_v1.json / independent_verification_v1.json. Image and video sources are retained with this project snapshot. Numerical results follow the stated batches and protocols; this page does not run those experiments.