Research Problem: Large Language Models (LLMs) like Claude 3.5 Haiku operate as "black boxes," making it difficult to understand their internal mechanisms, assess their reliability, or ensure safety. This research aims to reverse-engineer these internal computational processes.
Key Contributions:
Application of a circuit tracing methodology using Cross-Layer Transcoders (CLTs) and "attribution graphs" to visualize and understand LLM computations.
Detailed case studies on Claude 3.5 Haiku demonstrating internal mechanisms for multi-step reasoning, planning (in poetry), multilingual processing, arithmetic, medical diagnosis simulation, hallucination control, safety refusals, jailbreak dynamics, Chain-of-Thought (CoT) faithfulness, and models with hidden goals.
Identification of interpretable "features" and their interactions as the basis of these mechanisms.
Methodology/Approach:
Replaces standard MLP layers with CLTs to create a "replacement model" using sparse, interpretable features.
Generates prompt-specific "attribution graphs" showing causal links between features.
Validates hypotheses derived from graphs using intervention experiments (activating/inhibiting features).
Simplifies complex graphs by grouping related features into "supernodes."
Results: The paper provides evidence for complex internal strategies within Haiku, including multi-step reasoning chains, forward planning, language-agnostic representations, heuristic-based arithmetic, internal simulation of diagnosis, specific circuits for refusal/hallucination control, and mechanisms underlying jailbreaks and CoT (un)faithfulness. It highlights the abstract nature and surprising complexity of these internal computations.
Discussion Points
Strengths:
Innovative methodology (CLTs, attribution graphs) for peering into the black box.
Provides concrete, visual evidence for complex internal processes like planning and multi-step reasoning.
Enables causal interventions to test hypotheses and manipulate model behavior.
High potential impact for understanding, safety, and control if the methods mature.
Impressive visualization and detailed analysis across diverse case studies.
Weaknesses:
Interpretation is indirect, relying on a "replacement model".
High intervention strengths sometimes needed, raising questions about faithfulness or feature scaling.
Potential for cherry-picking successful examples.
Complexity of graphs requires significant manual effort for labeling and interpretation.
Doesn't fully explain attention mechanism computations (Noted as a paper limitation).
Key Questions:
How much of this is genuine reasoning vs. sophisticated pattern matching/memorization?
How exactly are features labeled and grouped into supernodes (manual vs. automated)?
How does the model learn multilingual representations (data influence)?
What is the nature of the model's internal "language" or representation?
Why are such high intervention strengths sometimes required?
Why aren't these findings published in traditional academic venues?
Applications:
AI Safety: Auditing for deception, hidden goals, unsafe reasoning.
Model Control: Potentially creating fine-grained controls ("sliders") for model behavior.
Data Curation: Identifying problematic data points by tracing circuit behavior.
Security: Understanding vulnerabilities and potential "hacking" via feature manipulation.
Cognitive Science: Drawing analogies to understand biological intelligence.
Connections:
Sparse Autoencoders (SAEs) for interpretability.
Conceptually related to KANs (feature-centric view).
Builds on general interpretability and circuit research.
Aligns with Anthropic's focus on AI safety and interpretability.
Notes and Reflections
Interesting Insights:
Models perform multi-step reasoning and planning internally ("in their head").
Existence of abstract, language-agnostic features is significant.
Arithmetic seems based on learned heuristics/lookups, not calculation.
Discrepancy between internal mechanisms and explicit CoT explanations.
Specific circuits control refusals and hallucinations, which can be manipulated.
Jailbreaks can exploit the model's step-by-step processing and grammatical constraints.
Bias/goals can become deeply integrated with the model's persona.
The biological analogy seems apt given the complexity and emergent nature of mechanisms.
Lessons Learned:
LLM internals are incredibly complex but potentially decipherable.
Interpretability tools are becoming powerful but still have significant limitations.
Causal interventions are key for validating mechanistic claims.
Models might develop unexpected or non-human-like strategies.