Answer course questions using multi-modal retrieval

This project is a multi-media question-answering and search system built as an AI teaching assistant for Electrical Engineering courses at UIUC, positioned as "better than Google" for that specific domain. It runs 11 separate models in parallel for text/image retrieval, generation, moderation, and ranking, and still achieves a median 2-second response time. Answers are grounded in retrieval-augmented generation over course material: textbooks, lecture videos, and student QA forums, ordered subjectively by importance - though the source notes none of this underlying data is publicly available, since the project was not granted redistribution rights by the original authors.

A specific research contribution documented here is semantic search retrieval during RLHF (reinforcement learning from human feedback), using a dataset iteratively produced by hiring a team of five Electrical Engineering students, covering material for UIUC's ECE 120 (Intro to Electrical Engineering) course; that RLHF comparison dataset is published openly on Hugging Face. To evaluate whether each new feature helps or hurts, the team maintains an in-house evaluation dataset of QA pairs written by expert electrical engineers, generates answers from each model, and asks GPT-3 to judge whether each generated answer is "better" or "worse" than the human-written ground truth - with the documented limitation that GPT-3 evaluating GPT-3's own outputs is likely biased toward rating itself favorably, and the source suggests re-running the same evaluation with Cohere's models as a check.

Use this project as a reference implementation or starting point for building a retrieval-augmented QA system over your own domain-specific document set - the source explicitly encourages plugging in your own Pinecone database of documents and reusing the pipeline in your own work, and it is fully open source apart from the commercial textbook content it was originally built against. It's a fit for anyone studying how to combine multiple retrieval, generation, and ranking models in parallel while keeping response latency low, or how to structure an iterative, LLM-judged evaluation loop for a QA system.

It is not a turnkey, drop-in teaching assistant for other courses without setup work: you need to build your own Pinecone document store from your own source material (the project ships data-cleaning utilities for textbook PDFs and Whisper video transcripts as a starting point) and supply your own API keys for the models it calls.

Input is your own document set (textbook PDFs, lecture transcripts, or plaintext) converted into a Pinecone vector database using the project's own data-cleaning notebooks, plus API keys for the models enumerated in run_ta_gradio.sh. Setup:

pip install -r requirements.txt
bash run_ta_gradio.sh

Output is a running Gradio web app that answers user questions by retrieving relevant passages from the Pinecone store and generating a response across the 11 parallel models, plus a feedback.json file collecting real user feedback submitted through the app, and, when the evaluation script is run, a comparison of each model's answers against the expert-written ground truth.

The system integrates with Pinecone as its vector database for retrieval-augmented generation, Gradio for the web UI (TA_gradio_ux.py), Whisper for generating lecture-video transcripts to feed into the document store, and GPT-3 (with Cohere suggested as a cross-check) for automated answer-quality evaluation. main.py aggregates the project's multiple LLM implementations, and prompting.py holds the project's prompt-engineering work.

Educators, researchers, and developers building or studying retrieval-augmented teaching-assistant or domain-QA systems - especially anyone who wants a working example of running several retrieval, generation, and ranking models in parallel with low latency, or of structuring an iterative, LLM-judged evaluation loop to track whether new features actually improve answer quality.