I ran a small experiment in Colab:
The main observation: rule recovery and rule execution can diverge
I found a small, concrete example that seems relevant to the causal question in Model-Native Structural Representation (MNSR): a model can recover the relationships in a policy accurately when asked to describe them, yet make mistakes when asked separately to apply those relationships.
The experiment used one synthetic six-rule routing policy, one 4B GGUF model, three textual renderings, four controlled variants, and two tasks: reconstruct the rules as JSON or execute them for five cases. The model generally recovered the intended phases, conditions, and priorities. Direct execution was less reliable, particularly when two rules competed to write the same field or a later rule depended on the final value of that field.
The central numbers are:
| Observation | Result | Interpretation |
|---|---|---|
| Strict recovery of all named rule relationships | 10/12 | The complete recovered set matched with exact rule identifiers. |
Recovery after a narrowly defined Rule: label normalization |
12/12 | The two strict failures were a syntactic name-prefix issue; all relation atoms matched after that correction. |
| Direct execution, five cases and three fields all exact | 1/12 | Only one full execution answer had every output field right. |
| Direct execution, individual output fields | 167/180 | Most fields were correct, but errors were distributed across many complete answers. |
| Deterministic execution of recovered rule records | 12/12 | The recovered records contained enough information for the reference evaluator to reproduce every expected case result. |
The last result is an internal consistency check, since the same deterministic evaluator defined the expected answers; it is not an independently validated execution engine. Also, the recover and execute results came from separate prompts and model calls. Therefore, I am not claiming to have observed the model’s latent representation during execution, or to have established a causal mechanism. The useful observation is a behavioral distinction between eliciting correct relations and independently carrying out their consequences.
This also is not an ArcGrav performance benchmark. The third rendering was a small ArcGrav-inspired shorthand made for this test, not a verified implementation of the official ArcGrav syntax. The ArcGrav announcement and repository describe a broader prompt-native language and workflow approach. The findings here cannot rank that language against prose or Markdown.
What I held constant and what I changed
All renderings came from one typed policy fixture. Its six rules specified a route phase followed by a followup phase. The route phase selected the applicable rule with the greatest numerical strength; the followup phase could see the resolved route and set notify and audit. The five input cases tested ordinary routing, urgency, legal hold, and their interactions.
I generated three descriptions of those same intended semantics: prose, a Markdown table, and ArcGrav-inspired bracketed sections. These were semantically matched by construction, but were not matched for length or stylistic quality. The prose was explicit, not exhaustively optimized. The bracketed rendering had extra scaffolding. Format effects, if any, could therefore mix representation with presentation or cue familiarity.
For each format, I used four variants:
baseline: the original rule set;renamed: changed rule and case identifiers without changing the decisions;reordered: changed declaration order, which the rule semantics explicitly made irrelevant;priority_flip: changed theurgentstrength from 20 to 40, above thehold_overridestrength of 30, so one expected answer intentionally changed.
Renaming and reordering preserve the relevant policy meaning; priority_flip is a negative control with different correct semantics. The scores were calculated against the correct reference for each variant, not against baseline answers copied across conditions.
Each format-variant pair received a recover request and an execute request. That is 3 x 4 x 2 = 24 calls, comprising 12 recovery and 12 execution responses. It is one task family with controlled variants, not 24 independent tasks. Each condition was sampled once at temperature=0.
| Representation | Exact-name recovery | Name-normalized recovery | All-five-case execution exact | Correct execution fields |
|---|---|---|---|---|
| Prose | 4/4 | 4/4 | 0/4 | 55/60 |
| Markdown | 4/4 | 4/4 | 0/4 | 55/60 |
| ArcGrav-inspired | 2/4 | 4/4 | 1/4 | 57/60 |
| Total | 10/12 | 12/12 | 1/12 | 167/180 |
I would not interpret the one-point difference as evidence of a format winner. The more useful information in this tiny run is the location of the errors, rather than any general relative performance.
One case makes the execution problem visible
Case C was an internal request with urgent=true and legal_hold=true. In the baseline policy:
urgent, strength 20, writesroute=expedite;hold_override, strength 30, writesroute=reviewfor an urgent internal request under legal hold;- the highest-strength matching rule wins for
route, so its final value isreview; - a later
notify_exprule writesnotify=opsonly when the final route isexpedite; audit_holdwritesaudit=truefor an internal request under legal hold.
The baseline correct result was:
{"route":"review","notify":null,"audit":true}
But in the baseline all three direct execution responses returned:
{"route":"review","notify":"ops","audit":true}
The route was correct; the notification did not match the final route. This looks like a failure at the boundary between resolving a field and using it in a later phase. It is consistent with a state-propagation error, but the emitted text cannot tell us what intermediate values the model actually stored.
The priority_flip condition changed only urgent to strength 40, above the hold rule’s 30. In that condition case C should have route=expedite, followed by notify=ops and audit=true. All three direct execution answers instead kept route=review. Yet the separate recovery answers correctly captured the new strength of 40. This gives a second concrete target: resolving competing rules by numeric priority, not by the intuitive meaning of their names.
A small next test before a large benchmark
The MNSR research program already proposes semantically equivalent representations, topology recovery, targeted ablations, and execution measurements. I would regard this experiment as a worked example within that plan. The useful incremental step is to use an observed failure to intervene at the recovery/execution boundary.
For the same policy and case C, compare three LLM execution conditions:
- Direct: execute from the original rendered policy alone.
- Gold structure: execute after receiving the correct canonical six-rule records explicitly.
- Recovered structure: first elicit the rule records in a separate request, then supply those records to a fresh execution request.
Keep the deterministic rule evaluator as a reference, not as a mandatory runtime for ArcGrav. This creates a useful decision path:
- Gold structure fixes the failure: explicit availability of the right relations helps, although a longer or more structured prompt could also be responsible.
- Gold structure does not fix it: the remaining bottleneck may concern rule selection, arithmetic precedence, phase state, or the answer contract.
- Gold works but recovered structure does not: inspect the exact recovered relations, identifiers, or missing constraints.
- Both gold and recovered work: separating interpretation from execution may be practically useful, without proving a particular latent causal mechanism.
A comparably sized neutral structured block could later test for a generic prompting effect. Repeated runs would be warranted if the first comparison changes the research conclusion; a full-scale benchmark need not be the first step.
What this pilot can and cannot support
The strongest defensible result is one completed synthetic probe showing different observed success rates for explicit relation extraction and direct rule application, plus concrete errors in precedence and cross-phase dependence. Its scope is one model, one underlying policy, one response per condition, a small set of transformations, and an unverified ArcGrav-inspired renderer. The high recovery score also leaves too little recovery variation to test whether differences in recoverability predict downstream success across tasks.
Neither the success of a JSON extraction prompt nor an error on a direct execution prompt reveals the internal computation by itself. At the same time, the pilot does suggest small tests that could discriminate interpretations without discarding the project’s prompt-native goals. I think this is a useful place to connect behavioral evaluation, controlled intervention, and the representation questions MNSR is asking.
The exact fixture, scoring details, theoretical connections, and conditional next steps are below.
Exact policy, scoring, and reproducibility
The canonical policy
I used a deliberately transparent policy router rather than an open-ended knowledge question. Its fields were kind (internal or external), urgent (boolean), and legal_hold (boolean). It produced route (ignore, review, or expedite), notify (null or ops), and audit (boolean).
The semantics were fixed before the model generated any responses:
- A rule matches only when all its specified
whenconditions hold. Omitted conditions are unconstrained. - In each phase and for each output field, the matching rule with the greatest numeric strength determines the field’s value. Rule declaration order is not a tiebreaker.
- Evaluate
routefirst. Then evaluatefollowupagainst the original inputs plus the final resolvedroute. Followup cannot alter route. - Defaults are
notify=nullandaudit=falsewhen no rule writes the field. - Conflicting writes tied at the greatest strength would be invalid under the deterministic reference. No evaluated case depends on resolving such a tie.
The six rules were:
| Rule | Phase | Strength | Conditions | Effect |
|---|---|---|---|---|
external |
route | 10 | kind=external |
route=ignore |
internal_base |
route | 10 | kind=internal |
route=review |
urgent |
route | 20 | kind=internal AND urgent=true |
route=expedite |
hold_override |
route | 30 | kind=internal AND urgent=true AND legal_hold=true |
route=review |
audit_hold |
followup | 20 | kind=internal AND legal_hold=true |
audit=true |
notify_exp |
followup | 20 | route=expedite |
notify=ops |
The five cases and baseline gold outputs were:
| Case | kind | urgent | legal_hold | Expected route |
Expected notify |
Expected audit |
|---|---|---|---|---|---|---|
| A | internal | false | false | review | null | false |
| B | internal | true | false | expedite | ops | false |
| C | internal | true | true | review | null | true |
| D | external | true | true | ignore | null | false |
| E | internal | false | true | review | null | true |
For example, case D still has urgent=true and legal_hold=true, but it does not satisfy conditions restricted to kind=internal. Case E is internal under legal hold but not urgent; it requires an audit without triggering the urgent route or the urgent-and-held override. Those cases check for accidental overgeneralization, while case C checks the conflict and the cross-phase dependency.
In priority_flip, changing urgent from strength 20 to 40 makes its route assignment stronger than hold_override at 30. The correct outputs for A, B, D, and E stay unchanged, but C becomes route=expedite, notify=ops, audit=true. This difference is intentional: it is a negative control for whether the model uses the declared number rather than a fixed narrative assumption that hold_override must win.
Three renderings of the same typed records
The fixture generated the prose, Markdown, and bracketed renderings from the same Rule and Case objects. This avoids accidentally writing different policies while comparing formats, although it does not make their token counts or linguistic cues identical.
The prose renderer included sentences of the form:
The named rule "hold_override" is in phase "route" with strength 30;
it applies when kind="internal" AND urgent=true AND legal_hold=true;
it writes route="review".
The Markdown renderer placed name, phase, strength, conditions, and writes in a five-column table, with a short header explaining phase order, defaults, and precedence. The illustrative bracketed renderer used entries of this general form:
[Rule:hold_override]:
[Phase]: route
[Strength]: 30
[When]: kind="internal" AND urgent=true AND legal_hold=true
=> route="review"
This is a readable approximation, not a byte-for-byte transcript of the bracketed renderer, which used an additional symbolic conjunction and other global sections. It is included to show why a parser or model could treat Rule: as part of the label instead of part of the rule name. The actual prompts, not these illustrative excerpts, are the scoring source. The bracketed renderer also included some borrowed descriptive scaffolding, so I do not use its token count as an estimate of optimized ArcGrav.
For renamed, the fixture replaced identifiers consistently: for example, external became R_z5, urgent became R_z9, and case C became Case_84. For reordered, the rules appeared in a different sequence, beginning with the notification rule, even though priority and phase order were unchanged. These check invariance to naming and declaration order. They do not create additional independent policy graphs.
Recovery and execution were different prompts
The recover prompt asked the model to extract exactly six records as a JSON array. Every record needed name, phase, integer priority, when conditions, target field, and written value. A one-rule illustration of the expected schema is:
{
"rules": [
{
"name":"notify_exp", "phase":"followup", "priority":20,
"when":{"route":"expedite"}, "field":"notify", "value":"ops"
}
]
}
The full answer, of course, required all six rules. In the separate execute prompt, the model was asked to return a single results object mapping each of the five case IDs to exactly three fields: route, notify, and audit. No intermediate graph was requested or forced in that call. A brief common system instruction requested literal interpretation of the supplied policy and JSON-only output. Neither call included the correct decisions as examples.
For the recovery score, each returned rule was converted to relation atoms: phase, numeric priority, target write, and one atom per condition. There were 28 atoms per canonical six-rule set (six phases, six priorities, six writes, ten conditions). The scorer used the rule’s name as part of each atom’s identity.
The two strict recovery failures occurred for the bracketed baseline and priority_flip conditions. Those outputs used names like Rule:external where the canonical ID was external. Because all atom identities were name-sensitive, their strict relation F1 was zero. The output was therefore genuinely non-exact under the requested identifier contract, but this did not mean the model had lost the phase, condition, or numeric relationships.
As a sensitivity check, I removed only a leading Rule: from returned rule names and ran the same scorer again. The two affected outputs then matched 28/28 atoms each; the other ten already matched. That yielded 12/12 name-normalized recovery. The strict 10/12 remains the official exact-name result of the probe, not something to overwrite or hide. A consumer that relies on precise identifiers may rightly treat the prefix as a real failure. A consumer interested in rule semantics may reasonably inspect the normalized relation score as a separate measurement.
For execution, a response was fully exact only if every case and every one of its three fields matched. Twelve calls of five cases and three fields give 180 scored fields. The model returned 167 correct fields, but only one call was completely exact. This difference between field-level accuracy and whole-answer reliability matters for workflows that cannot tolerate an incorrect decision hidden among mostly correct cells.
I additionally parsed the raw model-recovered rules and passed them to the existing deterministic evaluator. It reproduced the correct output table for all five cases in each of the 12 recovery conditions. The evaluator ignored the display name when computing conditions, priorities, and writes; thus even the two Rule:-prefixed results could execute correctly there. This was checked separately after the Colab run. Because the same evaluator generated the gold answers, the check shows operational sufficiency of the recovered fields under this evaluator, not independent confirmation of a language implementation.
Reproducibility and GPU verification
The actual run used the Unsloth Qwen3-4B-Instruct-2507-GGUF model repository with the following pinned artifact:
repository: unsloth/Qwen3-4B-Instruct-2507-GGUF
revision: a06e946bb6b655725eafa393f4a9745d460374c9
GGUF file: Qwen3-4B-Instruct-2507-Q4_K_M.gguf
SHA-256: 3605803b982cb64aead44f6c1b2ae36e3acdb41d8e46c8a94c6533bc4c67e597
runtime: JamePeng prebuilt llama_cpp_python 0.4.7+cu128
device: NVIDIA L4, CUDA0 backend registered
offload: 37/37 layers
GPU memory: approximately 3 MiB before model load, 3308 MiB afterward
generation: temperature=0, top_p=1, seed=191964
context: n_ctx=4096, n_batch=256, n_threads=4
max_tokens: 1300 for recover; 500 for execute
repeats: one completion per condition
The notebook loaded the model once, reset the prompt/KV state between calls, and randomized the 24-job order with a fixed seed. The results archive recorded prompts, raw outputs, parsed scores, configuration, and the CUDA load evidence. All 24 jobs completed; their outputs were JSON-parseable, with recorded finish reason stop. The 37/37 layer offload and GPU memory increase are worth recording because a Colab GPU by itself does not establish that llama_cpp_python actually used CUDA. A prebuilt wheel was installed; no Colab-side compilation was used.
For perspective, the baseline execution prompt-token counts were 447 Markdown, 516 prose, and 567 ArcGrav-inspired; the corresponding recovery counts were 476, 545, and 596. They describe this fixture only. The prose and bracketed versions were not equally optimized, and a complete workflow cost includes any initial notation explanation, responses, retries, and correction work. These numbers do not establish a general token-efficiency ranking for official ArcGrav.
I have not attached the raw Colab archive or notebook at a public URL in this post. The public links identify the project and model; they are not claimed as links to these exact generated prompts. If the experiment is shared separately, the raw response JSON, generated prompt variants, scorer, pinned revision, and CUDA proof are the useful reproducibility set.
Theoretical connections and what the pilot does not prove
Information, recoverability, and use are different quantities
I see a useful conceptual precedent in Larkin and Simon’s Why a Diagram is (Sometimes) Worth Ten Thousand Words (1987). Their distinction between informationally equivalent representations and representations with different processing affordances is relevant to MNSR’s notion of representational impedance. Two descriptions can carry the same intended relations while making those relations differently accessible to the receiving system. It is a conceptual analogy, not evidence about transformer computation or a validation of ArcGrav.
A more direct empirical parallel is Lu, Wang, and Huang, Encoded but Not Decoded (AACL-IJCNLP 2026). Their study distinguishes behavioral performance from information recoverable by output-layer and activation-level probes, and uses interventions to investigate part of that gap. It is about syntax and controlled dependencies, not this task router. My recover prompt is also not equivalent to their activation probe: it elicits a fresh textual answer and may itself make the model reason differently. Still, the study motivates an important measurement question: can a model expose the relevant relation when queried yet fail to deploy it in its final behavior?
For clarity, I would distinguish textual elicitation (can a model return the right rule records when asked?), behavioral deployment (does it produce the right decisions under the execution contract?), and internal decodability (are the relations detectable from model states under a controlled probing method?). This pilot measures the first two only. High textual recovery cannot be translated into a claim that a latent graph was faithfully present during the separate execution call. Likewise, a wrong final answer does not show that a relation was completely absent from internal state.
The intervention logic in Jia et al., Causal Decomposition of LLM Agent Failure via Oracle-State Intervention (REALM 2026) is another useful methodological connection. That work studies longer agent trajectories and replaces accumulated context with correct structured state to investigate failure sources. The task and reported statistics are not directly comparable to this router. The transferable idea is to provide known-correct state at a specified boundary and observe whether downstream behavior changes. For MNSR, providing a canonical rule graph before the decision is a small behavioral version of such an intervention, although extra instructions and changed token budgets remain confounds.
How this relates to the author’s stated hypotheses
The MNSR paper already includes an experimental program for semantically isomorphic formats, topology reconstruction, cue ablation, prior alignment, complexity scaling, and mechanistic probing. So I would not present those as missing suggestions. The pilot’s incremental contribution is a specific error pair, an exact scoring distinction, and a cheap intervention that can be embedded in that existing program.
- Structural recovery (H1): in this six-rule fixture, recovered relationships were nearly exact, subject to a
Rule:naming issue. This is a ceiling observation on one task, not a general recovery advantage for any format. - Round-trip performance (H4): recovery alone did not guarantee correct separately prompted execution. But because normalized recovery was at ceiling, there is no useful variation here for estimating whether better recovery predicts better execution across tasks. This does not refute the author’s round-trip hypothesis.
- Complexity and prior alignment (H2/H3): renaming and reordering test small invariances, not a true relational-complexity ladder. The renderings differed in scaffolding and familiarity, so this run does not isolate model-native alignment.
- Longitudinal coordination (H5): a two-phase calculation is not a multi-turn tool-using agent. It does expose a simple state dependency that could later be extended into a longitudinal task.
- Tradeoff frontier (H6): the paper already allows that no representation need dominate. The observed lack of a clear winner fits the need for conditional evaluation, but is much too small to establish the shape of that frontier.
- Alternative channels and learned interfaces (H7/H8): this was text-in/text-out, not a comparison of native structured channels or learned encoders.
One practical implication is to measure failure by relation type: identity, phase, precedence, conflict, state dependency, and verification. Whole-task exactness remains necessary, but without a relation-specific breakdown it is difficult to distinguish representational recovery from the mechanics of applying the recovered relation.
Metamorphic controls, priorities, and authority
Zhou et al., LGMT: Logic-Grounded Metamorphic Testing offers useful background on logically invariant transformations. In this experiment, identifier renaming and rule reordering were intended to preserve decisions. The strength change was not invariant; the expected output was recalculated. Keeping those categories separate avoids reporting a semantic edit as if it were harmless paraphrasing.
There is also an important distinction between the task’s numeric rule strength and an actual instruction’s authority. Work such as Many-Tier Instruction Hierarchy in LLM Agents concerns conflicts among instructions from differently privileged sources. By contrast, strength=40 here is a value within a user-supplied policy; it does not grant system privilege or authorize a tool operation. That matters if the discussion later turns to prompt injection or irreversible actions. Expressing a priority in a prompt and enforcing the host’s security boundary are different responsibilities.
Taken together, these connections support a narrower but testable version of the interesting question: when the correct relational structure is made explicitly available, does a particular model become better at a specific downstream operation, compared with a matched alternative? That can be examined behaviorally first, and only then, if warranted, with activation-level methods. Neither a universal format ranking nor a theory of the model’s hidden computation is required to make the small study informative.
Practical follow-up branches and pitfalls
A compact intervention matrix
In addition to the three LLM conditions above, an optional placebo condition can append a comparably sized, non-decisive structured note to the original policy. This helps distinguish benefits from the content of the gold graph from generic benefits of extra tokens or formatting. Keep the five cases and output schema unchanged, score both whole-set exactness and individual fields, and repeat the interesting conditions if the first result matters. Since the intervention also changes prompt length and context, any improvement is not automatically proof of reduced internal reconstruction cost.
If the immediate research goal is different
- Actual ArcGrav syntax: first replace this illustrative renderer with an author-validated rendering of the same policy, then compare it with carefully matched prose and Markdown. Otherwise, a statement about the bracketed toy rendering remains only that.
- Cross-model portability: repeat a few fixed cases with a second model or tokenizer before claiming generality. A different model may use familiar formatting cues differently.
- Efficiency: record tokens for the entire workflow, latency, corrections, and successful completion. The shortest initial prompt may not minimize human or model effort if it increases retries.
- Long-running agents: introduce explicit state transitions and verification, preferably with sandboxed, reversible tool actions. Do not infer tool-use reliability from a one-shot JSON router.
- Authoring and maintenance: measure the cost of writing, reviewing, changing, and debugging a representation. Even equal execution accuracy can leave substantial practical differences.
The author’s AgMark benchmarking component may be a suitable place to retain prompts, outputs, scores, and corrective steps without building a new evaluation framework for every research question. This is an option, not a requirement or a request to redo the author’s earlier reported runs.
Guardrails for interpreting future results
I would preserve the exact input semantics, record which variants intentionally change meaning, and score identity separately from relation content. Keep the raw outputs for each condition; an aggregate rate can conceal wrong booleans, stale state, prefix mismatches, or an otherwise correct answer with one consequential wrong field. A stronger claim about a representation should survive a few independent task graphs, repeated runs, and controls for prompt length and instruction detail, rather than depending on this one toy rule set.
For high-stakes tool operations, textual priority notation can communicate an intended workflow but cannot substitute for host-side authorization, input validation, or reversible execution. Those are separate layers that can be added where needed while preserving the project’s prompt-native, infrastructure-free default. The deterministic evaluator used here was a research oracle, not a proposed compulsory ArcGrav runtime.
Even without an accuracy advantage here, explicit structure might improve auditability, editing, handoffs, or correction. Those benefits deserve their own measurements.