Chunking Methodology:

AI research papers are lengthy and feature complex formatting. By scraping and chunking these papers, we create a substantial data source for semantic analysis, with the example demonstrating the impact of four different chunking sizes on search results.

Tokens	Overlap	Vectors
100	15	778k
200	20	374k
300*	?	241k
400	30	171k

Embedding Options:

Two embedding options are provided, varying according to the selected chunk size:

1. Embed Top K Document Chunks: Aggregates the top X chunk results across multiple documents, facilitating a broad yet relevant context. Optionally only include unique documents.
2. Embed Top K Chunk with J Additional Context: Focuses on a single document source, embedding the matched chunk along with surrounding chunks to enrich the LLM's contextual understanding.

Experiment with different options and rerun queries to gain insights on the how chunk size and retrieval method affect search results and generated responses.

The bible is a well-known hierarchy of books, chapters and verses. In this example, verses were converted to vectors using text-embedding-3-small with metadata for book and chapter index. A second vector index for chapters was created to compare semantic search results.

The example has four retrieval pipelines options for embedding text based on query specifics.

• Subjective RAG Pipeline: send search results to LLM for subjective analysis.
• Small-to-Big Embedding: return a larger contextual segment around the matched chunk

Templates provided demonstrate evaluation mechanisms, systematically assessing and summarizing content, enhancing the flexibility of AI agents and chatbot systems. The approach allows for more tailored interactions, as it equips AI and chatbot systems with the capability to judge and adapt responses based on specific content criteria.

We utilized the Basic English Version of the Bible, made available through this GitHub repository, for our demo.

Data was scraped from the COVID-19 dataset on Science.org/collections/coronavirus.

1. Prepare document list: Use a spreadsheet to configure title, url, and other metadata.
2. Load document list: configure chunking options and embedding model.
3. Scraping documents: fetch and scrape text content using web urls.

Splitting text into chunks

1. Sentences: blocks of 15 sentences with 3 overlap
2. Chunk Size: blocks of sentences sized less than 300 tokens
3. Recursive: syntactic paragraph and sentence chunking

In some cases overlap is used to ensure no context is lost. Compare the techniques to assess the relevance of chunks.