Breaking the Reversal Curse with Identity Bridges — the ICML 2026 fix that shouldn't work but does

Train a language model on ten thousand facts of the form "Alice's husband is Bob." Then ask it "Who is Bob married to?" It will confidently produce a random name.

This is the reversal curse, first documented by Berglund et al. in 2023. The finding was strange and unsettling: models memorize A → B associations but fail entirely on the trivially derivable B → A. Not degrade — fail. Near-zero accuracy on reversed queries even for facts they can regurgitate perfectly forward.

Follow-up work argued this was a fundamental architectural limitation of autoregressive transformers. The direction of the causal mask, the direction of gradient flow, the direction of attention itself — everything about the model points one way. Reversal, it seemed, was not something you could paper over with more data.

A paper accepted to ICML 2026 by Ma, Huang, Zhu, and Sojoudi shows that we were wrong. And the fix is not a new architecture, or a new training objective, or a bigger model. It is one weird kind of training example, added to the data mix in the right form.

They call it an Identity Bridge. And there is a delicious catch: the obvious version of it does not work at all.

What is the reversal curse, concretely

Take Llama-3.2-1B-Instruct. Give it a hundred synthetic biographical facts of the shape:

Q: The husband of Alice is? A: Bob.
Q: The husband of Carol is? A: David.
...

Finetune to convergence. The model nails forward queries — near-100% accuracy.

Now ask it the reverse:

Q: The wife of Bob is? A: ???

The model will say something that is not Alice. Accuracy on reversed queries hovers near zero. This is not a subtle degradation; it is a wall.

The intuitive explanation goes like this. During training, the model sees [Alice] [husband] → [Bob] as a sequence. Gradients only flow backward from the answer token to earlier context. So the model learns to predict "Bob" given "Alice" and "husband." But nothing in that training signal ever asked the model to predict "Alice" given "Bob" and any other cue. The two directions are, as far as the loss function is concerned, unrelated problems. The model has no reason to link them.

The unsettling implication: language models are not learning relationships. They are learning directed associations that happen to look like relationships. When you flip the query direction, they have nothing.

The obvious fix that doesn't work

The instinctive move, if you accept the diagnosis above, is to just add reversed data. Train on "Alice's husband is Bob" and also on "Bob's wife is Alice." Problem solved.

That is a fine solution when you have full control of your training data. It is not a solution to the general problem, because for most real facts you have exactly one direction of the statement and the other direction is precisely what you want the model to figure out from first principles.

The next instinct is symbolic: add examples that teach the model the concept of identity. Statements like:

Q: The name of Alice is? A: Alice.
Q: The name of Bob is? A: Bob.

If the model can learn "the identity of X is X," maybe it will connect the tokens to themselves and reversal will fall out.

The paper's Figure 7 shows what happens when you actually try this: essentially zero improvement on reversal accuracy. The simple identity examples are apparently learned as their own isolated pattern — memorize that "The name of X" produces "X" — with no propagation to the harder task.

This is where the paper's central move happens.

The trick: Identity Bridges have to go through the relation

The version that works is an "OCR" (Overlap-Chain-Relation) identity, not a simple identity. For the husband-wife case, the identity bridge example looks like:

Q: The wife of Alice's husband is? A: Alice.

Read it carefully. It is an identity — the answer is Alice, who is also the subject. But the identity does not exist in isolation. It is stated through the relation. To answer this, the model has to internally traverse: Alice → (husband) → Bob → (wife) → Alice. The answer is where you started, but the path uses the target relations.

The claim: this pattern, mixed into forward-only training data, teaches the model something categorically different than a bare identity would. It teaches it that wife and husband are inverses of each other, expressed as a closure through the identity of a real entity.

The training mix that works — the paper labels it OCR-6 — has three ingredients:

1. Forward examples, repeated six times each: The husband of Alice is Bob.
2. Identity bridge examples in OCR form: The wife of Alice's husband is Alice.
3. A held-out reversal test set — never seen during training: The wife of Bob is Alice.

You never train on the reversal. You never even train on the raw fact wife(Bob) = Alice. You train on chains that pass through it. And the model learns to answer the reversal query it has never seen.

What the numbers look like

The setup: Llama-3.2-1B-Instruct, 100 pairs each of husband/wife and parent/child, three random seeds, Adam optimizer, weight decay 0.30.

Fifty percent looks less impressive than "we solved the reversal curse." But remember the baseline: it is not 50% versus 90%. It is 50% versus zero, on a task where the entire prior literature said no amount of extra data could help.

Under the ablations, the number climbs to 55% with weight decay 0.30 (versus 48% at 0.25 and 43% at 0.35). Weight decay matters because — and this is the theoretical heart of the paper — the identity bridge works through implicit regularization, and weight decay is what makes that regularization actually take effect during optimization.

Why does this work? Nuclear norms, briefly

The paper's theoretical section proves something surprisingly strong: even a one-layer transformer, trained with gradient descent under weight decay, can escape the reversal curse when given identity bridges. This matters because prior arguments for the "fundamental limitation" hypothesis were often about the geometry of one-layer attention — the simplest possible model. If a one-layer model can do it, the excuse "transformers just can't" evaporates.

The intuition, translated out of the math:

Think of the model's key-query weight matrix as a 2×2 block. The upper-left block encodes forward knowledge ("given a subject and forward relation, produce object"). The lower-right block encodes something else. The upper-right block is where reversal knowledge would live — mapping objects back to subjects.

Under forward-only training, gradient descent has no pressure to fill in the upper-right block. It stays at zero. This is the reversal curse.

Under forward + identity bridge training, the identity bridge examples create loss terms that require *both* diagonals of the weight matrix to be nonzero. Once weight decay is applied, gradient descent has an implicit bias toward the minimum-nuclear-norm solution that fits the constraints. That minimum-norm solution turns out to be one where the reversal block automatically gets populated — because it is cheaper (in nuclear norm) to have symmetric off-diagonal structure than an asymmetric one.

You are not directly teaching the model reversal. You are teaching it a constraint whose lowest-cost solution happens to include reversal for free.

The formal statements are Theorems 3.3 and 3.4 of the paper, if you want the actual proofs. The rough shape is enough for the practitioner intuition: identity bridges reshape the loss landscape so that the model's laziest solution is the one you wanted.

Why the naive identity doesn't work

Now the naive form makes sense too. Q: The name of Alice is? A: Alice. does not touch any relation. It gives the model a direct token-to-token identity mapping that lives in an unrelated part of the weight matrix — probably somewhere in the embedding layer or a token-level pathway. That pathway can be filled in without ever touching the relational blocks where forward and reversal knowledge live.

The OCR form, by contrast, forces the identity to route through the relations you actually care about. wife of X's husband cannot be answered by any token-copying shortcut; the model has to internally compose husband and wife and land back on the starting entity. That composition is what gets propagated into the reversal block.

This is a general lesson worth pausing on. Regularization only helps when it lives in the right subspace. A regularizer that constrains parameters you did not care about does nothing for the parameters you did.

The catch: entity token length matters, a lot

Figure 8 of the paper is where things get less rosy. The 50% reversal accuracy is on two-token name entities like "Alice." When they swap to:

• One-token number entities ("42", "17") — reversal accuracy jumps to ~100%.
• Three-token name entities ("Alice Marie Johnson") — reversal accuracy collapses to ~10%.

The reason, per the paper, is a shortcut-learning failure mode. The model can learn to copy the entity token(s) after the relation token as a positional heuristic — "when you see the relation, copy back the entity you just saw" — and that shortcut competes with the actual reversal-learning pathway. The shortcut caps out around 60%, meaning the reversal pathway plus shortcut can occasionally reach ~50%, but multi-token entities make the shortcut harder without making the reversal easier.

For real-world facts — where entities are usually multi-token names, addresses, book titles, etc. — this is a serious caveat. Identity bridges are not a done deal for arbitrary knowledge. They are a proof of existence: reversal can be induced with the right regularizer, and the theoretical picture is now clear. Scaling this to messy multi-token real-world entities is future work.

How to actually try this

Here is a minimal Python sketch for constructing an identity-bridge-augmented dataset in the paper's OCR format. Drop it into your finetuning pipeline for any dataset that has clean inverse relations.

from itertools import chain

# Map each forward relation to its inverse
INVERSE = {
    "husband": "wife",
    "wife": "husband",
    "parent": "child",
    "child": "parent",
    "employer": "employee",
    "employee": "employer",
    # ... extend as needed
}

def format_forward(subject, relation, obj):
    return f"Q: The {relation} of {subject} is? A: {obj}."

def format_identity_bridge(subject, relation, obj):
    """OCR-form identity: closes the relation loop back to the subject."""
    inv = INVERSE[relation]
    return f"Q: The {inv} of {subject}'s {relation} is? A: {subject}."

def build_ocr_dataset(relations, forward_reps=6):
    """
    relations: list of (subject, relation, obj) triples.
        e.g. [("Alice", "husband", "Bob"), ("Carol", "wife", "David"), ...]
    forward_reps: how many times to repeat each forward example.
        The paper uses 6 (hence "OCR-6").
    """
    forward = [format_forward(s, r, o) for s, r, o in relations for _ in range(forward_reps)]
    bridge  = [format_identity_bridge(s, r, o) for s, r, o in relations]
    return list(chain(forward, bridge))

# Example
relations = [
    ("Alice", "husband", "Bob"),
    ("Carol", "wife", "David"),
    ("Elena", "parent", "Frank"),
]
dataset = build_ocr_dataset(relations)
for line in dataset[:5]:
    print(line)

A few things to know if you actually run this:

• Use weight decay 0.30 or thereabouts. The paper is unambiguous that weight decay is doing meaningful work here, not just a mild regularizer. The effect drops sharply above 0.35 or below 0.25.
• Test on truly held-out reversal queries. Do not include reversal statements anywhere in training — the whole point is to see if reversal emerges from the identity bridge alone.
• Start with one-token entities if you want a clean signal. Numbers or short single-word names. Then scale up to see where your setup falls off.
• The paper used Llama-3.2-1B-Instruct. Larger models may behave differently; the theoretical claim is about the one-layer case, and empirical scaling is not covered.

What this changes about how we think about LLM finetuning

Three things.

One, reversal curse is not a fundamental limitation of autoregressive transformers. It is a data problem with a data solution, and the data solution is more subtle than "add more examples." It is a structural augmentation that induces a specific kind of regularization.

Two, the form of the augmentation matters more than its content. Simple identity examples and OCR identity bridges look almost the same at first glance. One does nothing; the other unlocks a 50-point accuracy gain. The difference is entirely structural — whether the identity closes a relational loop or lives in isolation. This is the kind of detail that separates "we tried adding some more data" from "we induced the right implicit bias."

Three, this is more evidence for a broader thesis about LLM behavior: models have all the capacity they need; they just do not know they should use it. The forward-only training loss does not create pressure to fill in reversal knowledge, so it does not get filled in. Adding examples that shift the loss landscape can unlock capacity that was there all along.

For practitioners fine-tuning models on domain facts — company internal knowledge, technical documentation, biographical databases — the practical takeaway is direct: do not just train on forward directions and hope reversal generalizes. Construct identity bridges in your dataset before you finetune. It is cheap, it is well-motivated theoretically, and empirically it moves the needle from zero to fifty percentage points on the reversal task you probably care about but were not measuring.

What's next

The paper's own limitations section names the two things you would want to know before betting on this in production:

1. Multi-token entities are still hard. Three-token names dropped accuracy to ~10%. Real knowledge is full of multi-token entities.
2. Shortcut learning is a competing failure mode. The model can sometimes learn a pattern-matching heuristic that partially satisfies the identity bridge without actually learning reversal — capping the achievable accuracy at around 60%.

Both feel tractable. Shortcut learning is something the field has studied extensively in other contexts, and multi-token entities should respond to some combination of tokenizer changes and identity-bridge templates that are aware of entity length. My guess is that within a year we will see a follow-up that pushes the 50% number up to 80–90% on realistic entities. If that lands, it becomes a standard step in any factual-knowledge finetuning pipeline.

Until then: the reversal curse has a real, reproducible, theoretically grounded fix. It is a strange one — self-referential examples that route through the very relations you want reversed — but it works. Try it on your own finetunes.

*Paper: Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge, Xutao Ma, Yixiao Huang, Hanlin Zhu, Somayeh Sojoudi. ICML 2026 poster, July 8, 2026.*