The complex and unpredictable nature of deep neural networks prevents their safe use in many high-stakes applications. There have been many techniques developed to interpret deep neural networks, but all have substantial limitations. Algorithmic tasks have proven to be a fruitful test ground for interpreting a neural network end-to-end. Building on previous work, we completely reverse engineer fully connected one-hidden layer networks that have “grokked” the arithmetic of the permutation groups S_5 and S_6. The models discover the true subgroup structure of the full group and converge on neural circuits that decompose the group arithmetic using the permutation group’s subgroups. We relate how we reverse engineered the model’s mechanisms and confirmed our theory was a faithful description of the circuit’s functionality. We also draw attention to current challenges in conducting interpretability research by comparing our work to Chughtai et al. [4] which alleges to find a different algorithm for this same problem.
The set of integers \mathbb Z along with the addition + form a group. The identity element is 0. Also with the addition, the same is true for the set of rational numbers \mathbb Q, the set of real numbers \mathbb R and the set of complex numbers \mathbb C.
A function f:X \to Y is injective if for any x,y\in X different, f(x)\ne f(y).
The symmetric group introduced in Section ? along with the composition of permutations satisfies the group axioms. The identity element is the identity permutation leaving each element unchanged.
If f is injective mapping a set X to another set Y, the cardinality of Y is at least as large as that of X
Left as an exercise to the reader.
The set of natural numbers \mathbb N and addition do not form a group. The reason being that the inverse elements do not exist except for 0.
Given a group G, a subgroup H is a subset of G such that
a\cdot b\in H for any a, b\in H.
e\in H.
a^{-1}\in H.
For any f:X \to Y and g:Y\to Z injective functions, f \circ g is injective.
Given a proper subgroup H<G and an element g\in G, the set gH:=\{gh~|~h\in H\} is called a left H-coset. Similarly, Hg:=\{hg~|~h\in H\} is called a right H-coset.
gH is sometimes called a coset if the subgroup H is clear from the context. When we do not mention whether it is a left coset or a right coset, left coset is the default.
If f:X\to Y is bijective, the cardinality of X and Y are the same.
Two cosets g_1H and g_2H are either the same subset of G or disjoint (i.e., g_1H \bigcap g_2H = \emptyset).
If f:X\to Y is bijective, the cardinality of X is at least as large as that of Y.
If G is a finite group, any two H-cosets have the same number of elements.
The set X is finite.
Given two subgroups H, L<G and an element g\in G, the set HgL:=\{hgl~|~h\in H, l\in L\} is called the (H, L)-double coset, or the double coset if the pair (H, L) is clear from the context.
According to some, it is only the finite case (cf. Assumption 2.6) that is interesting.
Two double cosets Hg_1L and Hg_2L are either the same or disjoint.
A subgroup N is normal in G, denoted N \trianglelefteq G, if for any g \in G and any n \in N, we have gng^{-1} \in N.
Given a group G and a normal subgroup N \trianglelefteq G, the quotient group G/N is defined to be the set of N-cosets endowed with the multiplication given by gN\cdot hN=ghN for any g, h\in G.
If G is commutative (for every g, h\in G, we have gh=hg), every subgroup H\le G is normal.
If G=S_n, the subgroup S_{n-1} fixing the first element is not a normal subgroup. On the other hand, the alternating subgroup A_n (consisting of even permutations) is a normal subgroup of S_n.
Given a normal subgroup H \trianglelefteq G, the left H-coset and the right H-coset are in one-to-one correspondence. Furthermore, the set of H-double cosets is also in one-to-one correspondence to H-cosets.
By definition, gHg^{-1} = H. Therefore, gH = Hg. HgH = gHH = gH.
For any H\le G and an element g\in G, the set of conjugate elements gHg^{-1} forms a subgroup of G.
Let H_1, ..., H_n be conjugate subgroups of G, such that for each H_i the double coset H_{i}gH_{i} is equal to either H_i or G \setminus H_i. Then for each pair of subgroups H_i and H_j there exists a g \in G such that H_{i}g = gH_{j}. Moreover, the only double cosets of H_{i} and H_j are H_{i}gH_j = gH_j and H_{i}xH_{j} = G \setminus gH_j.
If i=j, for any h \in H_i the shared coset is the subgroup itself. If i \neq j, because H_i and H_j are conjugate, there exists a g \in G such that H_j = g^{-1}H_{i}g. The left coset is equal to the right coset:
Notice that the double coset H_{i}gH_j = H_{i}(H_{i}g) = H_{i}g. But for x \neq g:
Given a group G, a representation of G is a group homomorphism \rho_V: G\rightarrow GL(V) for some finite (but nonzero) dimensional vector space V over a field k. When we do not specifically mention k, we use \mathbb C as the default.
A representation \rho_V: G\rightarrow GL(V) is a sub-representation of \rho_W: G\rightarrow GL(W) if V can be identified as a linear subspace of W so that \rho_W(g) restricts to \rho_V(g) for all g\in G.
For any group G, the map G\rightarrow GL(V) sending all elements to the identity matrix is a representation. When \text{dim}(V) = 1, we call it the trivial representation of G.
Given two representations \rho_V, \rho_W of G, the direct sum of vector spaces V\oplus W admits a natural representation of G by letting \rho_V, \rho_W act on each component separately. We call this the direct sum of representations \rho_V, \rho_W, and denote it by \rho_V \oplus \rho_W.
Similarly, given two representations \rho_V, \rho_W, the tensor product V\otimes W admits a natural representation of G by acting on V, W separately and extend by linearity. We call this the tensor product of representations \rho_V, \rho_W, and denote it by \rho_V \otimes \rho_W.
A representation \rho of a group G is irreducible, if it does not have sub-representations other than \rho.
A representation \rho of a finite group G is a direct sum of irreducible representations.
The trivial representation of G is irreducible.
The permutation representation maps S_n \rightarrow GL(\mathbb{C}^3), i.e. 3\!\times\!3 matrices with a single 1 in each row and column and zeros everywhere else.
You can see that the matrices of the permutation representation act on the basis vectors of \mathbb{C}^3:
What it means to be a representation is that the group multiplication becomes matrix multiplication, so just as (2 \; 1 \; 3 )(3 \; 2 \; 1) = (2 \; 3 \; 1 ),
The permutation representation is reducible, because there is a subspace of \mathbb{C}^3 that is invariant to it’s action.
Note that there is no permutation matrix acting on the vector \begin{pmatrix} x & x & x \end{pmatrix}^T that will change it, because all of the components are equal.
As it turns out, there are no irreducible representations of S_3 that are three-dimensional. The largest irrep of S_3 is \rho_{(2,1)}, which is made of 2\!\times\!2 matrices. The matrices of the (2,1) irrep of S_3 are as follows:
We leave it as an exercise to the reader to verify that \rho_{(2,1)}(2 \; 1 \; 3 )\rho_{(2,1)}(3 \; 2 \; 1) = \rho_{(2,1)}(2 \; 3 \; 1).
Let \rho_V be a representation of G. The character of \rho_V is a map \chi(\rho_V):G\rightarrow \mathbb C given by \chi(\rho_V)(g) = \operatorname{tr}(\rho_V(g)).
The character \chi(\rho_V) takes the same value on a conjugacy class of G. In other words, \chi(\rho_V)(h) = \chi(\rho_V)(ghg^{-1}).
Let f:G\rightarrow \mathbb C be a map. If f(h)=f(ghg^{-1}) for any g, h\in G, f is called a class function.
The inner product of two class functions \phi, \psi are defined as:
The characters of \operatorname{Irr}(G) forms an orthonormal basis in the vector space of class functions.
For a finite group G, \operatorname{Irr}(G) is a finite set. Furthermore, the order of \operatorname{Irr}(G) is equal to the number of conjugacy classes in G.
Given a group G, the Fourier transform of a map f:G\rightarrow \mathbb C is a function \hat{f} from \operatorname{Irr}(G) to the union of M(\mathbb C^n) for all n such that
for an irreducible representation \rho.
If H_i and H_j are conjugate subgroups of G such that the only two double cosets are H_{i}gH_{j} and H_{i}H_{j}, then each right coset H_{i}x has a paired left coset yH_j where y = x^{-1}g such that for all h_{x} \in H_{i}x and h_y \in yH_j, h_{x}h_y \in H_{i}gH_{j}
Let f: G \rightarrow \mathbb{C} be constant on the cosets of H \le G and non-zero on at least one coset. Then \hat{f}(\rho) = 0 if the restriction of \rho to H, \rho |_{H} does not contain the trivial representation as a subrepresentation.
The function f can be decomposed as the sum of functions
for each coset xH. Because Fourier transform \hat{f} is invariant under translation we may, without loss of generality, analyze only the function f_{H}. For a given \alpha_{x}, \hat{f}_{xH}(\rho) = \hat{f}^{x}_{H}(\rho) = \rho(x)\hat{f}_{H}(\rho) for all x \in G. Recall the definition of \hat{f}_{H}(\rho) from F.1:
where in ? we decompose \rho|_{H} into a direct sum of irreps of H. But because each \tau_{i} is irreducible, \sum_{h \in H} \tau_{i}(h) = \mathbf{0} unless \tau_i is the trivial irrep. Thus, unless the decomposition of \rho_H into irreps of H includes the trivial representation, \hat{f}|_{H} = 0
Suppose f: G\rightarrow \mathbb C is a function such that its Fourier transform \hat f is nonzero only on an irreducible representation \rho and the trivial representation. Let \hat f(\rho) = A \in M(\mathbb C^n). We have the following explicit formula:
This is immediate by the Fourier inversion formula.
The Fourier transform of 1_H is nonzero only at the irreducible components of the permutation representation of G on G/H.
By definition, the Fourier transform of 1_H on an irrep \rho is
Notice that the image of \sum\limits_{a\in H} \rho(a) are invariant under H due to the symmetry of this expression.
Let V be the vector space where \rho acts on. Under the action of the subgroup H through \rho, one can decompose V as irreps of H. We group them into two parts:
where V^H is a direct sum of copies of trivial representation of H (or in other words, the invariant subspace of V under H), and V' is the direct sum of nontrivial irreducible components of V.
We immediately see the following by definition:
Also by definition, nontrivial irreps of H do not have invariant subspaces since they do not admit proper sub-representations. Therefore,
As a result, \widehat{1_H}(\rho) is simply a scaled projection to the invariant subspace of V. Whether it is zero depends on whether \text{Res}_H \rho has any trivial components.
By Frobenius reciprocity,
where \chi(\rho) is the character of the irrep \chi\in\operatorname{Irr}(G) given by its traces, and \langle\cdot\rangle is the inner product between class functions.
The left-hand side \langle \text{Ind}^G_H(1_H), \chi(\rho) \rangle is nonzero if and only if \rho is an irreducible component of the permutation representation of G on G/H. The right-hand side \langle 1_H, \chi(\text{Res}_H(\rho)) \rangle_H is nonzero if and only if \text{dim}(V^H) \neq 0
If the permutation representation of G on G/H_1 has only 2 irreducible components, the Fourier transform \hat F of the membership counting function F is nonzero only at these 2 irreducible components. In particular, the equation (G.3) applies to F.
For S_n and the subgroup S_{n-1} fixing one element, the permutation representation has only two irreducible components.
The natural representation of S_n on \mathbb C^n (by permuting the basis) decomposes as a direct sum of trivial representation and the standard representation of dimension n-1.
The text on this page is from arXiv:2312.06581v2 by Dashiell Stander, Qinan Yu, Honglu Fan and Stella Biderman, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.
The record of each item is in analysis/objects/2312/2312.06581.jsonl.