Neural FOXP2 -- Language Specific Neuron Steering for Targeted Language Improvement in LLMs

TL;DR

This work treats language defaultness in multilingual LLMs as a controllable property arising from a sparse, low‑rank circuit of language neurons. It proposes Neural FOXP2, a mechanistic, inference‑time intervention that localizes language‑control features via per‑layer sparse autoencoders, derives compact steering directions with layerwise SVD, and applies a signed sparse edit within a trusted window to make Hindi or Spanish the default without retraining. The pipeline is evaluated on Hindi and Spanish across MT, QA, NLI, and summarization using a frozen LLaMA‑3 8B backbone, with guardrails to limit drift and leakage and to preserve semantic fidelity. Across tasks, FOXP2 yields measurable defaultness gains (token‑mass and LID signals) while maintaining task performance, and ablations validate the necessity and sufficiency of the localized, low‑rank control components. The approach provides an auditable, resource‑efficient pathway to targeted language improvement in multilingual models, balancing practical deployment with safety and mechanistic interpretability.

Abstract

LLMs are multilingual by training, yet their lingua franca is often English, reflecting English language dominance in pretraining. Other languages remain in parametric memory but are systematically suppressed. We argue that language defaultness is governed by a sparse, low-rank control circuit, language neurons, that can be mechanistically isolated and safely steered. We introduce Neural FOXP2, that makes a chosen language (Hindi or Spanish) primary in a model by steering language-specific neurons. Neural FOXP2 proceeds in three stages: (i) Localize: We train per-layer SAEs so each activation decomposes into a small set of active feature components. For every feature, we quantify English vs. Hindi/Spanish selectivity overall logit-mass lift toward the target-language token set. Tracing the top-ranked features back to their strongest contributing units yields a compact language-neuron set. (ii) Steering directions: We localize controllable language-shift geometry via a spectral low-rank analysis. For each layer, we build English to target activation-difference matrices and perform layerwise SVD to extract the dominant singular directions governing language change. The eigengap and effective-rank spectra identify a compact steering subspace and an empirically chosen intervention window (where these directions are strongest and most stable). (iii) Steer: We apply a signed, sparse activation shift targeted to the language neurons. Concretely, within low to mid layers we add a positive steering along the target-language dominant directions and a compensating negative shift toward the null space for the English neurons, yielding controllable target-language defaultness.

Figures (18)

• Figure 1: Neural FOXP2: VAE/autoencoder-based discovery of language-selective neurons (Hindi or Spanish). The workflow proceeds in three stages: (a) Sample activations from a language model by running language-specific prompts and extracting intermediate activations $h^{(\ell)}(x)$ at selected layers $\ell$ (left panel; “Embedding $\rightarrow$ Transformer Blocks”). (b) Learn a feature dictionary using an autoencoder (here, a VAE/SAE-style encoder–nonlinearity–decoder pipeline) that represents activation vectors as a sparse/structured combination of latent features (middle panel; “Encoder matrix (tied with decoder) $\rightarrow$ Add bias + apply ReLU”). (c) Interpret the resulting language-specific dictionary features by mapping discovered features back to model units and compiling a Language Neuron Dictionary: each row lists a neuron id $(\ell,i)$ along with its highest-affinity Hindi versus Spanish tokens (right panel), yielding a human-auditable signature of a putative “language neuron” and enabling downstream masking/editing in Neural FOXP2.
• Figure 2: English is the lingua franca by default. Mean target-language defaultness$\mathbf{\Delta M \;=\; M_{\ell} - M_{\mathrm{En}}}$ for 4 languages: Hindi, Spanish, French, and German. Across all targets, $\Delta M$ remains biased toward English (negative mass advantage), indicating that defaultness behaves as a prior rather than a capability limit. This motivates FOXP2’s language-specific steering: shifting logit mass toward the target language.
• Figure 3: Geometric Interpretation of FOXP2 as Language Steering in Latent Space. Across these panels, FOXP2 appears as defaultness by motion and direction, not by understanding. Rather than reshaping the model’s conceptual topology, FOXP2 applies a low-rank, signed activation shift—a small set of language vectors that nudges default language without touching beliefs or semantics. It teaches the model which language to start in, not why it should start there.
• Figure 4: Reproducibility protocol. Scan-friendly, end-to-end requirements to reproduce FOXP2: pinned versions (weights/tokenizer/detector), immutable manifests (data/token sets), exact hook sites, dictionary training recipe, localization and geometry extraction rules, signed edit schedules with hard guardrails, and deterministic aggregation that regenerates camera-ready tables.
• Figure 5: Reproducibility checklist. A one-page audit sheet: pin every dependency (model/tokenizer/detectors/data), freeze all intermediate stage outputs $(\mathcal{N},\mathcal{S},\mathcal{W})$, and ensure logs + a pinned command deterministically regenerate the camera-ready tables and figures.