Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reproducibility Analysis and Enhancements for Multi-Aspect Dense Retriever with Aspect Learning

Keping Bi, Xiaojie Sun, Jiafeng Guo, Xueqi Cheng

TL;DR

This work addresses reproducibility in multi-aspect dense retrieval by re-evaluating MADRAL on the public MA-Amazon dataset and identifying the detrimental effect of learning an implicit 'OTHER' aspect. It proposes two enhancements—replacing 'OTHER' with the explicit CLS token and representing aspects with the first few content tokens—and demonstrates that these variants substantially improve retrieval performance. By systematically comparing aspect representation and fusion designs, the study provides concrete guidance for stronger multi-aspect dense retrievers and highlights trade-offs in aspect learning objectives during fine-tuning. The findings offer practical, reproducible design choices and pave the way for future research in robust, aspect-aware retrieval systems, with code to be released for public use.

Abstract

Multi-aspect dense retrieval aims to incorporate aspect information (e.g., brand and category) into dual encoders to facilitate relevance matching. As an early and representative multi-aspect dense retriever, MADRAL learns several extra aspect embeddings and fuses the explicit aspects with an implicit aspect "OTHER" for final representation. MADRAL was evaluated on proprietary data and its code was not released, making it challenging to validate its effectiveness on other datasets. We failed to reproduce its effectiveness on the public MA-Amazon data, motivating us to probe the reasons and re-examine its components. We propose several component alternatives for comparisons, including replacing "OTHER" with "CLS" and representing aspects with the first several content tokens. Through extensive experiments, we confirm that learning "OTHER" from scratch in aspect fusion is harmful. In contrast, our proposed variants can greatly enhance the retrieval performance. Our research not only sheds light on the limitations of MADRAL but also provides valuable insights for future studies on more powerful multi-aspect dense retrieval models. Code will be released at: https://github.com/sunxiaojie99/Reproducibility-for-MADRAL.

Reproducibility Analysis and Enhancements for Multi-Aspect Dense Retriever with Aspect Learning

TL;DR

This work addresses reproducibility in multi-aspect dense retrieval by re-evaluating MADRAL on the public MA-Amazon dataset and identifying the detrimental effect of learning an implicit 'OTHER' aspect. It proposes two enhancements—replacing 'OTHER' with the explicit CLS token and representing aspects with the first few content tokens—and demonstrates that these variants substantially improve retrieval performance. By systematically comparing aspect representation and fusion designs, the study provides concrete guidance for stronger multi-aspect dense retrievers and highlights trade-offs in aspect learning objectives during fine-tuning. The findings offer practical, reproducible design choices and pave the way for future research in robust, aspect-aware retrieval systems, with code to be released for public use.

Abstract

Multi-aspect dense retrieval aims to incorporate aspect information (e.g., brand and category) into dual encoders to facilitate relevance matching. As an early and representative multi-aspect dense retriever, MADRAL learns several extra aspect embeddings and fuses the explicit aspects with an implicit aspect "OTHER" for final representation. MADRAL was evaluated on proprietary data and its code was not released, making it challenging to validate its effectiveness on other datasets. We failed to reproduce its effectiveness on the public MA-Amazon data, motivating us to probe the reasons and re-examine its components. We propose several component alternatives for comparisons, including replacing "OTHER" with "CLS" and representing aspects with the first several content tokens. Through extensive experiments, we confirm that learning "OTHER" from scratch in aspect fusion is harmful. In contrast, our proposed variants can greatly enhance the retrieval performance. Our research not only sheds light on the limitations of MADRAL but also provides valuable insights for future studies on more powerful multi-aspect dense retrieval models. Code will be released at: https://github.com/sunxiaojie99/Reproducibility-for-MADRAL.
Paper Structure (23 sections, 8 equations, 4 figures, 4 tables)

This paper contains 23 sections, 8 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. Preliminaries of Multi-Aspect Dense Retrievers
  4. Component Variants: Aspect Representation
  5. Reusing CLS Token (MTBERT).
  6. Declaring k Extra Embeddings (MADRAL).
  7. Reusing First-k Content Tokens.
  8. Component Variants: Aspect Fusion
  9. Weighting Mechanism
  10. Objects to Fuse
  11. Aspect Learning and Overall Training Objectives
  12. EXPERIMENTAL SETTINGS
  13. Multi-aspect Amazon ESCI Dataset (MA-Amazon).
  14. Methods for Comparison.
  15. Evaluation Metrics.
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Two multi-aspect dense retrieval models proposed by Kong et al. madr.
  • Figure 2: The upper figures illustrate the aspect representation and learning of MTBERT, MADRAL, and our variant. The lower figures show aspect fusion methods to yield the final representation $E_X$. The weighted combination can be CLS gating or presence weighting. $a_o$ denotes the special aspect "OTHER".
  • Figure 3: Effect of AL Coefficient $\lambda_f$
  • Figure 4: Effect of Annotation Amount