Dataset Intelligence for Robotics

🤗Datasets
1

Hello People,
I built Calibra , it aims for catching bad demonstrations before they waste GPU time then selecting the episodes that actually matter.
here is the github repo : GitHub - omertt27/Calibra: Dataset observability and coreset selection for robotics imitation learning · GitHub
I would appreciate any feedbacks or advice.

4

Hi. For now, I tried a few things:

Overall, I think the direction is getting clearer and more useful. In particular, the earlier idea of diagnostics → inspect a shortlist → decide what to do seems like the right abstraction boundary for this kind of tooling; you also described that intent explicitly in the earlier Calibra thread, and the current project now separates integrity, quality, coverage/review, and pruning much more clearly.

I pinned Calibra at 6aac9a1 and did some CPU-only probes against public LeRobot datasets. I did not rerun policy training or simulator rollouts, so I would treat the observations below as checks on the measurement / selection contracts, not as evidence that the benchmark conclusions themselves are wrong.

The three things I would prioritize are:

  1. Make the action contract explicit before computing motion-quality metrics.
  2. Add one real LeRobot v3 multi-task fixture.
  3. Add a sensitivity test that verifies diversity_weight actually changes the selection geometry when intended.

A small equal-budget control for the quality ablation would also make that part of the benchmark easier to interpret.

The first one had by far the largest effect in my tests.

1. Action semantics look worth making first-class

On the public lerobot/pusht dataset, the current metadata declares a 2-dimensional action; both dimensions are listed as motors in meta/info.json.

At the pinned Calibra revision, ControlSmoothnessAnalyzer defaults to:

action_type="position"
gripper_dims=[-1]

so for this particular 2-D dataset the default smoothness path effectively excludes the second action dimension.

I reran the benchmark-style 165-episode PushT training split twice:

  • exact current default: position, gripper_dims=[-1]
  • same position interpretation, but both action dimensions active: gripper_dims=[]

The result was much more sensitive than I expected:

Quantity Current default Both action dims
Quality-rejected episodes 42 / 165 23 / 165
Quality-approved episodes 123 / 165 142 / 165
Diagnosed regime moderate_noise high_noise

The two reject sets had a Jaccard overlap of only about 0.275.

That difference then propagated into the actual coreset membership:

Retention Selected episodes in each run Intersection Jaccard
5% 8 / 8 2 0.143
10% 16 / 16 3 0.103
25% 41 / 41 13 0.188

I would not interpret this as “gripper_dims=[] is necessarily the correct setting.” Rather, it suggests that the action representation is important enough that it probably should be part of the dataset/analyzer contract rather than an implicit global default.

For example, even a small internal representation along these lines might make the boundary explicit:

action representation:
    space: joint | EEF | ...
    command: absolute_position | delta_position | velocity | ...
    frame: ...
    active_dims: ...
    gripper_dims: ...
    dt/fps source: ...

The exact schema is obviously up to you. The useful part would be that a smoothness score, quality gate, regime decision, and prune result could all say which action interpretation they were computed under.

There is a related small wiring detail: at the revision I tested, profile_dataset.py exposes --control-mode and records it in profile metadata, but I could not see that value being passed into ControlSmoothnessAnalyzer.action_type; the analyzer still uses its default unless constructed differently elsewhere. Making the action contract one object passed through the pipeline might remove this kind of split-brain state automatically.

If this area is already changing on main, please treat the numbers above as specific to the pinned revision rather than a claim about the latest code.

PushT probe details and interpretation guardrails

The reason I think this is worth resolving before spending more GPU time is that it changes several downstream quantities at once:

  • per-episode jerk / velocity-discontinuity metrics
  • the Stage-1 quality pool
  • the inferred noise regime
  • the candidate pool passed to Stage 2
  • final coreset membership

I also inspected a small stratified subset consisting of low-density tail episodes, quality-rejected episodes, and ordinary controls, using an image-backed PushT copy aligned against the tabular dataset.

The strongest jerk events were not obviously equivalent to corrupted demonstrations from the small visual windows I checked. That is not enough evidence to call them useful recovery behavior either. My takeaway was simply:

jerk / low density / unusual motion looks safer as a review signal than as a semantic label.

That seems consistent with the workflow proposed in LeRobot issue #3760: cheap action-only episode metrics narrow the visual inspection set, but the diagnostic itself does not modify the dataset.

I would therefore keep at least three concepts separate:

  • integrity failure — e.g. structurally invalid timestamps, impossible metadata, frozen/corrupted data;
  • behavioral/quality risk — e.g. jerky, hesitant, unusual, inefficient;
  • training utility — whether including that episode actually improves the target policy.

The first can sometimes justify a hard gate. The second often looks more like review/ranking evidence. The third ultimately depends on the learning problem.

2. A tiny LeRobot v3 multi-task fixture would probably catch a useful edge case

I also tried a real multi-task LeRobot v3 dataset, lerobot/libero_10.

It has 379 episodes and 10 tasks. In current LeRobot v3, LeRobotDatasetMetadata explicitly manages info.json, stats.json, tasks.parquet, and episode metadata. The task table plus each frame/episode’s task_index is enough to reconstruct the task identity.

When I passed the dataset through the pinned Calibra LeRobotReader, all 379 resulting episode task_description values were None.

So this appears to be narrower than “LeRobot v3 support is broken”: the tabular data loads, but per-episode task identity is currently lost on this real v3 multi-task path.

A very cheap regression test would be a two-episode / two-task v3 fixture containing:

meta/info.json
meta/tasks.parquet
data/...parquet
    episode_index
    task_index

and then asserting that the two Calibra episodes retain different task descriptions.

That seems especially worthwhile because task-aware review/coverage becomes more important as the project moves from “is this recording healthy?” toward “which demonstrations actually matter?”

Why I think this is a contract boundary rather than just a parser detail

The current LeRobot metadata implementation documents tasks.parquet as part of the metadata model, and task registration is persisted through that table.

So I think it may be useful for Calibra to treat:

dataset-global metadata
episode identity
task identity
action semantics
observation semantics

as distinct ingestion contracts.

That also gives later metrics a useful failure mode:

“Task-aware coverage was not evaluated because task identity was unavailable”

rather than silently turning a multi-task dataset into an apparently single-task one.

This would fit nicely with the Not Evaluated idea already introduced on the integrity side.

3. diversity_weight may benefit from one sensitivity unit test

Another cheap check produced a very repeatable result.

At the pinned revision, I varied positive diversity_weight values across:

0.10, 0.30, 0.70, 0.85, 0.90

First on a synthetic fixture, and then on the real PushT train split.

For both action-dimension interpretations, and at 5%, 10%, and 25% retention, every tested positive weight produced the same selected episode IDs.

Looking at the Stage-2 feature path in pruning.py, my reading is that feature blocks are multiplied by their weights and then each resulting column is independently min-max normalized.

If that is the intended sequence, a positive constant scale on a column is largely canceled by the subsequent column normalization.

I may be misunderstanding the intended meaning of diversity_weight, but either way this looks very easy to make explicit with a unit test:

Given a fixture where quality and diversity prefer different episodes:

weight A -> selection A
weight B -> selection B

If that test is supposed to pass with different selections, then normalization probably needs to happen before the relative block weighting (or otherwise preserve the scale).

If instead the weights are only intended to enable/disable families of features while normalization deliberately removes magnitude, then documenting that would resolve the ambiguity too.

I like this one because it is almost free to test and does not require deciding whether any particular robotics metric is “correct.”

4. One equal-k control could make the quality ablation easier to read

The detailed PushT benchmark is particularly useful because it reports the negative result that quality filtering can hurt at small budgets instead of hiding it.

There is one small control I think would help interpret that result.

The benchmark describes the methods as operating at “equal episode budget k”, but at the pinned revision the targeted benchmark code uses two counts:

k   = fraction Ă— full training pool
k_q = fraction Ă— quality-approved pool

With the published 165-episode train pool and 123-episode quality pool, that gives approximately:

Retention Full-pool k Quality-pool k_q
5% 8 6
10% 16 12
25% 41 31

This does not affect the cleanest comparison between calibra, random_full, and diversity_only where the episode budget is matched.

It mainly affects how I would interpret quality_only / random_quality: two things change simultaneously—

  1. the candidate pool has been quality-filtered;
  2. fewer final episodes are retained.

I think both comparisons are actually useful, because they answer different questions:

  • same fraction of each pool: “If I filter first and then keep the same percentage, what happens?”
  • same absolute k: “Given exactly the same final training budget, does quality filtering help or hurt selection?”

So rather than replacing the current experiment, I would add the equal-k version next to it. If the quality-only result stays poor at equal k, the negative result becomes considerably stronger.

5. I would keep “quality”, “rarity”, and “downstream utility” as separate axes

This is the part of Calibra that I find most interesting.

Your own PushT results already show that the best strategy depends on retention budget: very aggressive quality filtering can remove useful tail coverage, while at a larger budget quality + diversity can work better.

There are several nearby pieces of robotics work that seem useful as reference points, but they are measuring different things:

  • DemInf / Robot Data Curation with Mutual Information Estimators scores demonstrations through action diversity and predictability, and reports improvements from filtering on RoboMimic plus real ALOHA/Franka experiments.
  • CUPID moves further downstream and estimates each demonstration’s influence on closed-loop policy return. An interesting result there is that conventional “higher quality” demonstrations are not always the demonstrations that produce the most robust policy.
  • DataMIL is explicitly task-aware: it estimates which data helps a particular target policy/task, and reports cases where visually similar samples have very different utility because their action distributions differ.
  • The older RoboMimic study is also a useful caution for proxy evaluation: they reported that the checkpoint with the best validation objective could be 50–100% worse than the best policy under rollout evaluation.

I do not think Calibra needs to become CUPID or DataMIL. In fact, one attractive property of Calibra is that most of its diagnostics can run before expensive policy training.

I would frame the layers approximately like this:

Layer Question Cheap evidence can often answer it?
Integrity Is the recording structurally trustworthy? Often yes
Quality risk Is this motion suspicious / unusual / inefficient? Often partially
Coverage Does this episode add behavioral support? Often partially
Task utility Will this episode improve this particular policy/task? Usually needs stronger downstream evidence

That preserves the inexpensive pre-training value proposition without asking a cheap heuristic to answer a more expensive causal question.

A possible low-cost default flow

If I were using Calibra operationally, the default route I would find easiest to trust would be:

1. Resolve the data/action contract first

Check:

  • schema/version
  • units
  • absolute vs delta vs velocity action
  • joint vs EEF action
  • coordinate frame
  • gripper dimensions
  • sampling rate / timestamp source
  • task identity

If one of those is unknown, surface that uncertainty in the report.

2. Hard integrity next

Strong structural failures can be quarantined or blocked.

Examples:

  • malformed/non-monotonic timestamps
  • missing frames/data corruption
  • impossible schema relationships
  • known calibration incompatibility

3. Rank behavioral anomalies instead of immediately assigning meaning

Things like:

  • jerk
  • hesitation
  • unusual duration
  • low-density trajectories
  • atypical endpoint

can produce a shortlist.

Then:

diagnostics -> shortlist -> inspect -> decide

This is almost exactly the workflow proposed independently in LeRobot #3760.

4. Do coreset selection only after the measurement contract is fixed

Otherwise a change in selected data can come from:

dataset behavior
        +
metric definition
        +
representation assumptions
        +
selection algorithm

which makes the benchmark harder to interpret.

5. Separate proxy evidence from downstream evidence

Cheap trajectory prediction or coverage proxies are useful for iteration.

For stronger claims such as “these demonstrations actually matter”, closed-loop success or another task-level outcome eventually becomes the more convincing endpoint.

6. A small consistency point: Not Evaluated could perhaps propagate into curation too

One smaller observation: the integrity side now does a nice job of distinguishing not evaluated from evaluated and clean.

In the pruning/quality path I tested, when the relevant per-episode quality metrics were absent, their composite contribution defaulted to clean/zero and the quality filter allowed the episodes through.

That may be exactly the permissive behavior you want. But as the system becomes more automated, I think it would be useful if downstream curation could still retain the distinction:

clean
bad
unknown / not evaluated

rather than letting “missing evidence” become indistinguishable from “evidence of cleanliness.”

This feels like a natural continuation of the explicit evaluation-coverage work already added to the integrity layer.

Scope of what I tested

For reproducibility, these observations were made against Calibra commit:

6aac9a1c03106207a1e2bd023cb4bb594a605e88

I used CPU-only checks and public LeRobot data.

The relevant existing Calibra test modules also passed in that environment (102 tests), which is why I see the points above more as contract / coverage / benchmark-boundary issues than as evidence of a generally broken implementation.

I did not:

  • retrain the BC policy;
  • rerun ACT or Diffusion Policy training;
  • run simulator/robot rollouts;
  • establish that low-density episodes are semantic edge cases;
  • establish that jerk events are corruption;
  • establish that gripper_dims=[] is universally the right PushT configuration.

Those would all be stronger claims than the probes support.

The main result of the tests is narrower:

some current quality/selection results are highly sensitive to representation assumptions that are cheap to make explicit and cheap to regression-test.

If I had to pick only a few next checks

I would probably do these before another large benchmark run:

  1. Add an explicit action-contract fixture and recompute the existing PushT diagnostics with the intended contract.
  2. Add a two-task LeRobot v3 fixture using tasks.parquet.
  3. Add a diversity_weight sensitivity unit test.
  4. Add one equal-k quality-only control.

All four are relatively cheap, and each removes an ambiguity that otherwise propagates into more expensive experiments.

Only after those are fixed would I spend more GPU time deciding whether the quality/diversity crossover itself changes.

The encouraging part, to me, is that none of these require changing the basic goal of Calibra. They mostly make the boundaries between data integrity, behavioral diagnostics, coverage, selection, and downstream usefulness more explicit.

That seems compatible with the project’s current direction rather than a different direction from it.