Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Intellecta Cognitiva: A Comprehensive Dataset for Advancing Academic Knowledge and Machine Reasoning

Ajmal PS, Ditto PS, Jithin VG

TL;DR

Intellecta tackles the paucity of reasoning-focused datasets by constructing a large-scale synthetic-plus-textbook resource totaling 11.53B tokens, with 8.01B synthetic and 3.52B textbook data. The approach leverages Mixtral-8x7B-Instruct-v0.1 to generate both advanced thought processes and textbook-style explanations, underpinned by rigorous curation (OCR ingestion, deduplication via Simhash, toxicity screening with Perspective API, and topic clustering via DBSCAN). Empirical evaluation with a 634M-parameter boomer model trained on 11.5B tokens demonstrates competitive cross-domain performance on benchmarks like ARC and HellaSwag, supporting claims of robust generalization from a comparatively compact model. The work contributes a scalable, ethically curated dataset with broad educational coverage and provides prompts and methods to reproduce thought-process and textbook data generation, highlighting potential for more efficient reasoning-enabled language models in real-world educational and cognitive tasks.

Abstract

Intellecta dataset emerges as an innovative synthetic dataset, engineered to enhance the cognitive processing capabilities of contemporary language models. With a composition of 11.53 billion tokens, integrating 8.01 billion tokens of synthetic data with 3.52 billion tokens of rich textbook data, Intellecta is crafted to foster advanced reasoning and comprehensive educational narrative generation. Leveraging the Mixtral-8x7B-Instruct-v0.1 model, the dataset facilitates the generation of complex thought processes and detailed, textbook-style explanations, thus enabling language models to engage in both critical thinking and profound educational discourse. This hybrid dataset stands as a testament to the potential of synthetic data in pushing the boundaries of AI, offering a repository that is not only vast and varied but also refined to align with ethical standards and intellectual rigor.

Intellecta Cognitiva: A Comprehensive Dataset for Advancing Academic Knowledge and Machine Reasoning

TL;DR

Intellecta tackles the paucity of reasoning-focused datasets by constructing a large-scale synthetic-plus-textbook resource totaling 11.53B tokens, with 8.01B synthetic and 3.52B textbook data. The approach leverages Mixtral-8x7B-Instruct-v0.1 to generate both advanced thought processes and textbook-style explanations, underpinned by rigorous curation (OCR ingestion, deduplication via Simhash, toxicity screening with Perspective API, and topic clustering via DBSCAN). Empirical evaluation with a 634M-parameter boomer model trained on 11.5B tokens demonstrates competitive cross-domain performance on benchmarks like ARC and HellaSwag, supporting claims of robust generalization from a comparatively compact model. The work contributes a scalable, ethically curated dataset with broad educational coverage and provides prompts and methods to reproduce thought-process and textbook data generation, highlighting potential for more efficient reasoning-enabled language models in real-world educational and cognitive tasks.

Abstract

Intellecta dataset emerges as an innovative synthetic dataset, engineered to enhance the cognitive processing capabilities of contemporary language models. With a composition of 11.53 billion tokens, integrating 8.01 billion tokens of synthetic data with 3.52 billion tokens of rich textbook data, Intellecta is crafted to foster advanced reasoning and comprehensive educational narrative generation. Leveraging the Mixtral-8x7B-Instruct-v0.1 model, the dataset facilitates the generation of complex thought processes and detailed, textbook-style explanations, thus enabling language models to engage in both critical thinking and profound educational discourse. This hybrid dataset stands as a testament to the potential of synthetic data in pushing the boundaries of AI, offering a repository that is not only vast and varied but also refined to align with ethical standards and intellectual rigor.
Paper Structure (13 sections, 4 figures, 1 table)

This paper contains 13 sections, 4 figures, 1 table.

Table of Contents

  1. Design Goals
  2. Introduction
  3. Source and Curation
  4. Source of Dataset
  5. Synthetic Data Generation
  6. Curation of Intellecta
  7. Dataset Description
  8. Evaluation and Results
  9. Conclusion
  10. References
  11. Appendix
  12. Prompt for Thought data
  13. Prompt for Textbook data

Figures (4)

  • Figure 1: Distribution of Textbook and Synthetic Data in the Intellecta Dataset, highlighting the proportions of various domains within the synthetic subset, and emphasizing the dataset's comprehensive coverage across multiple disciplines.
  • Figure 2: Histogram of Toxicity Scores: This histogram represents the frequency distribution of toxicity scores across the dataset, post-screening with the Perspective API. A higher concentration of low toxicity scores illustrates the efficacy of the filtering process in curating an ethically aligned dataset.
  • Figure 3: Histogram of Topics in the Intellecta Dataset: This histogram displays the variety of subjects covered, each rated for educational value, highlighting the dataset's commitment to providing a comprehensive learning experience.
  • Figure 4: Cluster Analysis of Topics in the Intellecta Dataset: This scatter plot visualizes the semantic clustering of data topics, emphasizing the dataset's depth and the multifaceted nature of the educational content provided.