OrthoGeoLoRA: Geometric Parameter-Efficient Fine-Tuning for Structured Social Science Concept Retrieval on theWeb

TL;DR

OrthoGeoLoRA addresses fundamental geometric drawbacks of standard LoRA by reparameterizing the low-rank update into an SVD-like form $ΔW = B Σ A^{⊤}$ with $A,B$ constrained on the Stiefel manifold. Through geometric reparameterization, orthogonality is enforced while preserving compatibility with standard optimizers, delivering a drop-in PEFT upgrade that avoids gauge freedom, scale ambiguity, and rank collapse. On a hierarchical ELSST-based retrieval benchmark, OrthoGeoLoRA consistently outperforms LoRA and other PEFT variants in ranking metrics, while using similar parameter budgets and training pipelines. The results demonstrate improved optimization dynamics, fuller utilization of the low-rank capacity, and practical benefits for resource-constrained Web4Good deployments. The work also discusses societal and ethical considerations, including data bias in synthetic descriptions and the sustainability implications of efficient model adaptation for social science infrastructures.

Abstract

Large language models and text encoders increasingly power web-based information systems in the social sciences, including digital libraries, data catalogues, and search interfaces used by researchers, policymakers, and civil society. Full fine-tuning is often computationally and energy intensive, which can be prohibitive for smaller institutions and non-profit organizations in the Web4Good ecosystem. Parameter-Efficient Fine-Tuning (PEFT), especially Low-Rank Adaptation (LoRA), reduces this cost by updating only a small number of parameters. We show that the standard LoRA update $ΔW = BA^\top$ has geometric drawbacks: gauge freedom, scale ambiguity, and a tendency toward rank collapse. We introduce OrthoGeoLoRA, which enforces an SVD-like form $ΔW = BΣA^\top$ by constraining the low-rank factors to be orthogonal (Stiefel manifold). A geometric reparameterization implements this constraint while remaining compatible with standard optimizers such as Adam and existing fine-tuning pipelines. We also propose a benchmark for hierarchical concept retrieval over the European Language Social Science Thesaurus (ELSST), widely used to organize social science resources in digital repositories. Experiments with a multilingual sentence encoder show that OrthoGeoLoRA outperforms standard LoRA and several strong PEFT variants on ranking metrics under the same low-rank budget, offering a more compute- and parameter-efficient path to adapt foundation models in resource-constrained settings.

Figures (5)

• Figure 1: OrthoGeoLoRA overview. Unconstrained Euclidean parameters $(\widehat{A},\widehat{B},s)$ are mapped to orthonormal factors $A,B$ and a non-negative diagonal $\Sigma$, yielding $\Delta W=B\,\Sigma\,A^{\top}$. The data flow $x\!\rightarrow\!A^{\top}x \!\rightarrow\!\Sigma(\cdot)\!\rightarrow\!B(\cdot)$ is shown together with dimensions and constraints; see text for details.
• Figure 2: Singular value spectrum of the learned update matrix ($r=8$). LoRA's sharp decay is a clear indicator of rank collapse, while OrthoGeoLoRA maintains a healthier spectrum, demonstrating full utilization of its allocated rank.
• Figure 3: Validation MRR over training steps. OrthoGeoLoRA exhibits faster convergence and achieves a higher peak performance, indicating a more efficient optimization landscape.
• Figure 4: Performance (MRR) as a function of rank $r$. OrthoGeoLoRA consistently outperforms LoRA and scales more effectively with increased parametric capacity.
• Figure 5: An example of a fully constructed prompt for the concept "Social Stratification" under the role of "Critical Social Theorist." The prompt combines role assignment, strict constraints on implicit expression, and a specific analytical focus.