Retrieval Augmented Learning: A Retrial-based Large Language Model Self-Supervised Learning and Autonomous Knowledge Generation

TL;DR

Retrial-Augmented Learning (RAL) tackles the domain-knowledge gap in $MDP$-based decision problems by offering a train-free, reward-free self-supervised approach that uses a dynamic Retrieval-Augmented Generation (RAG) module to organize intermediate data. It introduces a three-stage loop—hypothesis proposal, validation, and knowledge generation—operating over three databases and multiple LLM roles to autonomously generate and apply validated knowledge. In the LLM-PySC2 setting, RAL reduces hallucinations and accelerates learning, while exhibiting robust OOD performance and cross-model data transferability at a fraction of the cost of post-training large models. The work demonstrates practical benefits for cost-sensitive decision-making and autonomous knowledge generation, while acknowledging explorative and long-horizon validation limitations and outlining directions for future enhancements.

Abstract

The lack of domain-specific data in the pre-training of Large Language Models (LLMs) severely limits LLM-based decision systems in specialized applications, while post-training a model in the scenarios requires significant computational resources. In this paper, we present Retrial-Augmented Learning (RAL), a reward-free self-supervised learning framework for LLMs that operates without model training. By developing Retrieval-Augmented Generation (RAG) into a module for organizing intermediate data, we realized a three-stage autonomous knowledge generation of proposing a hypothesis, validating the hypothesis, and generating the knowledge. The method is evaluated in the LLM-PySC2 environment, a representative decision-making platform that combines sufficient complexity with domain-specific knowledge requirements. Experiments demonstrate that the proposed method effectively reduces hallucination by generating and utilizing validated knowledge, and increases decision-making performance at an extremely low cost. Meanwhile, the approach exhibits potential in out-of-distribution(OOD) tasks, robustness, and transferability, making it a cost-friendly but effective solution for decision-making problems and autonomous knowledge generation.

Figures (32)

• Figure 1: General process of RAL. In the learning process, the agent generates hypotheses about better policies, validates the hypothesis in similar situations, and summarizes the knowledge and experience when a policy has been sufficiently validated. When the proposed policies of a situation have been thoroughly evaluated, the agent directly uses the retrieved knowledge to make better decisions.
• Figure 2: RAL framework. At each step, the agent retrieve a list of hypothetical strategies or experience from Database $H(o|h)$ and $E(o|e)$, test the hypothetical policy or exploit the experience to make better decisions. At the same time, the agent learn from the state transition of last step, proposing a different strategy, validate the current strategy or present fully validated hypothetical strategy into experience.
• Figure 3: System Prompt for generating hypothetical policies.
• Figure 4: System Prompt for generating validations of a proposed policy.
• Figure 5: System Prompt for generating experience of a proposed policy.