LLM Neuroanatomy II: Modern LLM Hacking and Hints of a Universal Language?: A Technical Deep Dive

Executive Summary

• Repeat Your Self (RYS) is a zero-training optimization technique that enhances Transformer performance by duplicating specific "reasoning" layers in the middle of the model architecture to extend the processing time in format-agnostic latent space.
• Analysis of Qwen3.5-27B and Qwen2-72B reveals a distinct three-phase internal structure: Encoding (surface form to hidden state), Reasoning (universal format-agnostic space), and Decoding (hidden state back to surface form).
• The latest research utilized a beam search across 3,024 candidates and a surrogate model scoring 2 million configurations to identify optimal layer duplication patterns for modern LLMs.
• Evidence suggests that modern LLMs operate in a "Universal Language" space within their middle layers, where semantic meaning is preserved regardless of the input language (English, Mandarin, Arabic, etc.) or encoding (Base64).

Technical Architecture: The ERD Hypothesis

The core of this technical analysis revolves around the Encoding-Reasoning-Decoding (ERD) hypothesis. This framework suggests that Transformers do not process information uniformly across all layers but instead follow a tripartite neuroanatomical structure.

1. The Encoding Phase (Early Layers)

In the initial layers, the model focuses on transforming raw tokens into high-dimensional hidden states. Data from cosine similarity probes indicates that this is where the model resolves the "surface form" of the input. For example, if the same sentence is provided in English and Mandarin, the hidden states in these layers start as highly dissimilar but rapidly converge as the model strips away language-specific syntax.

2. The Reasoning Phase (Middle Layers)

This is the "Universal Language" space. The research demonstrates near-perfect cosine similarity between hidden states representing the same semantic concept, even when the inputs are in different languages or encoded formats (like Base64).

• The RYS Technique: The Repeat Your Self (RYS) method targets these specific layers. By duplicating a block of these "reasoning" layers—without changing weights or performing additional training—the model's "thinking time" is effectively extended.
• Search Methodology: Identifying which layers to duplicate is a massive combinatorial problem. The author used a beam search on 3,024 candidates and a surrogate model to score 2 million possible configurations to find the optimal "relayering" sequence.

3. The Decoding Phase (Late Layers)

In the final layers, the hidden states diverge once more. The model "decodes" the abstract semantic concept back into the specific language or format requested for the output.

Performance Analysis

The RYS method was first validated on Qwen2-72B, where duplicating a block of seven middle layers propelled it to the top of the HuggingFace Open LLM Leaderboard. Current research focuses on whether this property persists in modern, more compact models like Qwen3.5-27B.

Benchmark and Search Scale

While specific benchmark scores for the Qwen3.5-27B RYS variants are currently being processed through validation sweeps, the search scale demonstrates the complexity of modern LLM optimization:

Functional Anatomy and Model Size

The research highlights a critical trade-off in model size:

• Large Models (70B+): Exhibit "cleaner" separation between encoding, reasoning, and decoding phases.
• Small Models (<30B): Tend to have more "entangled" functional anatomy. However, preliminary findings on Qwen3.5-27B suggest that the RYS property is robust even in these more compact circuit structures.

Technical Implications

1. Zero-Training Optimization

The RYS method implies that current LLM training may leave significant performance "on the table." If a model's reasoning can be improved simply by duplicating existing weights, it suggests that the bottleneck is often the depth of the reasoning chain rather than the quality of the learned weights themselves.

2. The Universal Language Latent Space

The observation of high cosine similarity across languages in the middle layers suggests that LLMs develop a "universal" internal representation of concepts. This has profound implications for cross-lingual transfer and the development of more efficient multilingual models.

3. "Brain Scanning" for Transformers

The use of math probes to create heatmaps of layer similarity introduces a new diagnostic tool for AI researchers. Instead of treating the Transformer as a black box, this "neuroanatomical" approach allows developers to see exactly where reasoning is occurring and where the model is simply "decoding" tokens.

Limitations and Trade-offs

• Compute-Heavy Discovery: While RYS requires no training compute, the discovery compute is significant. Searching through 2 million configurations requires massive hardware (e.g., Dual Grace-Hopper systems).
• Entanglement: In smaller models, the boundaries between encoding and reasoning are blurred. Duplicating the wrong layers can lead to a degradation in performance if the "encoding" logic is repeated unnecessarily.
• Inference Overhead: Adding layers through RYS increases the parameter count and, consequently, the KV cache and inference latency. A 27B model might become a 35B model, requiring more VRAM.

Expert Perspective

The significance of RYS cannot be overstated: it is a paradigm shift from "training harder" to "architecting smarter." The fact that a model can be "hacked" into a higher performance tier without a single gradient update challenges our understanding of Transformer scaling laws. The evidence of a "Universal Language" in the middle layers suggests that we are closer to a unified theory of machine intelligence than previously thought. The success of this method on Qwen3.5-27B proves that the ERD structure is a fundamental property of the Transformer architecture, not an artifact of specific datasets.

Technical FAQ

How does RYS compare to traditional Fine-Tuning?

Unlike fine-tuning, RYS does not change any weights. It is a structural modification (relayering). While fine-tuning teaches a model new tasks or styles, RYS essentially gives the model more "cycles" to process information using its existing reasoning capabilities.

Is the "Universal Language" space present in all models?

The researcher is currently testing this on MiniMax M2.5 and GLM-4.7. Initial results suggest it is a general property of Transformers, but the "cleanliness" of the separation depends on the model's size and training maturity.

Does relayering break the model's ability to follow instructions?

No, provided the "encoding" and "decoding" layers are left intact. Since the decoding layers are responsible for mapping the reasoning back to the prompt's requirements, the model maintains its instruction-following capabilities but with improved logical depth.

References

• Maunder, E. (2024). Cosine Similarity Probes in Multilingual Transformers.
• Ng, D. N. (2024). LLM Neuroanatomy Part 1: The RYS Method.
• HuggingFace Open LLM Leaderboard (2024-2026 Archive).

Sources

• David Noel Ng - LLM Neuroanatomy II

All technical specifications, pricing, and benchmark data in this article are sourced directly from official announcements. Competitor comparisons use publicly available data at time of publication. We update our coverage as new information becomes available.