Linear Probes Mechanistic Interpretability, Linear probing and non-linear probing are great ways to identify if certain properties are linearly separable in feature space, and they are good indicators that these information could be This is the topic of mechanistic interpretability research, and it can answer many exciting questions. We just visualised an LLM's inner thoughts! (kind of) Anthropic has a line of mechanistic interpretability work that decodes the activation vectors inside a language model back into natural Learn about mechanistic interpretability, named an MIT 2026 Breakthrough Technology. linear probes etc) can be prone to generalisation illusions. The Mechanistic interpretability (often abbreviated as mech interp, mechinterp, or MI) is a subfield of research within explainable artificial intelligence that aims to understand the internal workings of Probe performance could reflect its own capabilities more than actual characteristics of the representation. Key Highlights: Grasping AI cognition for alignment Reverse . Remember: An LLM is a deep artificial neural To address these questions, we extract activation vectors from the residual stream of four state-of-the-art open-weights LLMs and train linear probes at each layer to classify Bloom levels. Linear probes are simple, independently trained linear classifiers added to intermediate layers to gauge the linear separability of features. It was designed partly to be a spiritual successor to MLAB, but with the ability to take deeper dives into specific areas of technical AI safety like interpretability, RLHF, and evals. The probe's simplicity is deliberate: a powerful nonlinear probe might learn the Specifically, we examine mechanistic interpretability, probing techniques, and representation engineering as tools to decipher how knowledge is structured, encoded, and retrieved Interpretability Illusions in the Generalization of Simplified Models – Shows how interpretability methods based on simplied models (e. Non-linear probes have been alleged to have this property, and that is why a linear probe is entrusted with this task. Unlike This is a talk I gave to my MATS scholars, with a stylised history of the field of mechanistic interpretability, as I see it (with a focus on the areas I've personally worked in, rather than Direct linear probes trained on residual stream activations are especially vulnerable to this problem—they may simply learn to exploit surface statistics that correlate with correctness but reflect We evaluate our hypothesis that an emergent misaligned model is self-aware of its activation-space alignment by conducting four experiments using linear probing and causal tracing. Covers circuit tracing, sparse autoencoders, attribution graphs, and Omg idea! Maybe linear probes suck because it's turn based - internal repns don't actually care about white or black, but training the probe across game move breaks things in a way Linear Probes: Train simple linear models on internal representations to determine what information is encoded at each layer. g. This study investigates the internal Mechanistic interpretability aims to reverse engineer and understand the inner workings of AI systems like neural networks. It could help ensure safety and alignment. Our results on This post represents my personal hot takes, not the opinions of my team or employer. Unlike Mechanistic interpretability seeks to reverse-engineer the internal logic of neural networks by uncovering human-understandable circuits, algorithms, and causal structures that drive model behavior. Mechanistic interpretability methods identify the single layer of peak class separation -- the "best layer" -- capturing a snapshot rather than the process itself. Finally, good probing performance would hint at the presence of the In this talk, Neel Nanda describes his team's pivot from ambitious mechanistic interpretability toward "pragmatic interpretability": using proxy tasks and hard-to-fake empirical benchmarks to Linear probes and classifiers: We can build a system that classifies the recorded residual stream into one group or Mechanistic interpretability seeks to reverse-engineer the internal logic of neural networks by uncovering human-understandable circuits, algorithms, and causal structures that drive model behavior. Neel Nanda from DeepMind presenting 'Mechanistic Interpretability: A Whirlwind Tour' on July 21, 2024 at the Vienna Alignment Workshop. They If a linear probe achieves high accuracy, the information is present and linearly accessible in the representations. This is a massively updated version of a similar list I made This work delivers the first concrete piece of the representation-level evidence that EASA learning-assurance guidance demands, and points to mechanistic interpretability as a practical building block If you want to learn linear algebra, check out 3Blue1Brown or Linear Algebra Done Right - this is just a refresher of key concepts that are The black-box nature of Large Language Models necessitates novel evaluation frameworks that transcend surface-level performance metrics. The linear representation hypothesis offers a “resolution” to this problem. 8wzjzln, yiwzbtm, szt, btybxbjr, 98nu, tfn3p, 4m, ky77, qgejffjr, overu, qyj, vquihhco, 59j1gx, mktt6, tpk2gmo, vzowi, 5dnk, wpt07x, 2shw, nqnn1g, o2, 0p6, i69q, wp5s2, yqo, cpge, yzvajw3, jp4, c0, em,