Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Foundations of GenIR

Qingyao Ai, Jingtao Zhan, Yiqun Liu

TL;DR

This chapter analyzes how modern generative AI reshapes information access by separating information generation from information synthesis. It argues that information generation enables tailored content creation guided by user instructions, while information synthesis grounds outputs by leveraging external knowledge through retrieval-augmented approaches. It surveys foundational aspects such as Transformer architectures, scaling, training pipelines, and multi-modal capabilities, and discusses retrieval-augmented generation and corpus modeling as core IA paradigms. It highlights opportunities for more capable and grounded IA systems, while acknowledging challenges like hallucination risk, data quality, and inference costs, and outlines directions for future study.

Abstract

The chapter discusses the foundational impact of modern generative AI models on information access (IA) systems. In contrast to traditional AI, the large-scale training and superior data modeling of generative AI models enable them to produce high-quality, human-like responses, which brings brand new opportunities for the development of IA paradigms. In this chapter, we identify and introduce two of them in details, i.e., information generation and information synthesis. Information generation allows AI to create tailored content addressing user needs directly, enhancing user experience with immediate, relevant outputs. Information synthesis leverages the ability of generative AI to integrate and reorganize existing information, providing grounded responses and mitigating issues like model hallucination, which is particularly valuable in scenarios requiring precision and external knowledge. This chapter delves into the foundational aspects of generative models, including architecture, scaling, and training, and discusses their applications in multi-modal scenarios. Additionally, it examines the retrieval-augmented generation paradigm and other methods for corpus modeling and understanding, demonstrating how generative AI can enhance information access systems. It also summarizes potential challenges and fruitful directions for future studies.

Foundations of GenIR

TL;DR

This chapter analyzes how modern generative AI reshapes information access by separating information generation from information synthesis. It argues that information generation enables tailored content creation guided by user instructions, while information synthesis grounds outputs by leveraging external knowledge through retrieval-augmented approaches. It surveys foundational aspects such as Transformer architectures, scaling, training pipelines, and multi-modal capabilities, and discusses retrieval-augmented generation and corpus modeling as core IA paradigms. It highlights opportunities for more capable and grounded IA systems, while acknowledging challenges like hallucination risk, data quality, and inference costs, and outlines directions for future study.

Abstract

The chapter discusses the foundational impact of modern generative AI models on information access (IA) systems. In contrast to traditional AI, the large-scale training and superior data modeling of generative AI models enable them to produce high-quality, human-like responses, which brings brand new opportunities for the development of IA paradigms. In this chapter, we identify and introduce two of them in details, i.e., information generation and information synthesis. Information generation allows AI to create tailored content addressing user needs directly, enhancing user experience with immediate, relevant outputs. Information synthesis leverages the ability of generative AI to integrate and reorganize existing information, providing grounded responses and mitigating issues like model hallucination, which is particularly valuable in scenarios requiring precision and external knowledge. This chapter delves into the foundational aspects of generative models, including architecture, scaling, and training, and discusses their applications in multi-modal scenarios. Additionally, it examines the retrieval-augmented generation paradigm and other methods for corpus modeling and understanding, demonstrating how generative AI can enhance information access systems. It also summarizes potential challenges and fruitful directions for future studies.
Paper Structure (24 sections, 2 equations, 2 figures)

This paper contains 24 sections, 2 equations, 2 figures.

Table of Contents

  1. Information Generation
  2. Model Architecture
  3. Word Embedding
  4. Position Embedding
  5. Attention
  6. Layer normalization
  7. Scaling
  8. Training
  9. Training Objectives
  10. Training Stages
  11. Prompt Optimization
  12. Multi-modal Applications
  13. Multi-modal Understanding
  14. Multi-modal Generation
  15. Information Synthesis
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Transformer architecture: the overview on the left and the illustration of one layer on the right NIPS2017_attention.
  • Figure 2: Su et al. su2024dragin generate queries for RAG based on the internal attention distribution of LLMs.