Every class has its own conventions: a preferred approach to a problem, a way it wants solutions written, a set of references it's built on. These all build up over time and create the course's structure. That makes the open internet a good source for a general approach, but a weak one when it comes to doing well in a specific class. Wikipedia's definition is fine, but it isn't the one your professor would give, and the course's version is what you're graded on in the end.
Last semester I found myself feeding textbooks, lecture notes, slides and hundreds of references into chatbots and still being unable to trust the answers they gave me. I kept going back to the PDFs to check whether a claim was actually in them or if it followed the class' standards. It worked until it didn't, and I was still manually doing the checking the tool was supposed to handle.
So I built the version I wanted. The idea isn't new, a tutor that relies on your course material is something plenty of students have thought of, and the initial version wasn't hard to make. The part that took real work, and the reason I'm writing this up, was making it something I could realistically rely on and would actually want to use.
How the personal course tutor works
The idea is simple: you give it your course sources, the textbooks, references, and lecture notes, and it answers only from those pages: in their terminology, with citations like [Campbell Biology, p. 2], and a plain "the course materials don't cover this" when the answer isn't there.
Under the hood it's a standard retrieval pipeline built on LlamaIndex. The PDFs are parsed with LlamaParse, which preserves the layout and notation a plain text extractor mangles, then split into passages, embedded locally with bge-small, and stored in a Chroma vector database. A question retrieves the most relevant passages and hands them to Claude, under a prompt that says to answer only from the excerpts, use their terminology, cite everything, and refuse when the material isn't there. Every model call runs through the Respan gateway, so it's logged and can be graded later.
The first version worked. I asked it about the Calvin cycle and got a correct answer in the textbook's phrasing, cited to the right page. It looked exactly like what I wanted, but it was a simple test which prompted further investigation. This is when the challenge began.
Why I couldn't trust its answers
A fluent, cited answer sounds the same whether it's right or invented. The citation makes it feel reliable, but a confident wrong answer looks no different than a confident right one.
Moreover, the pipeline can fail in two places. Retrieval can bring back the wrong passages, or miss the ones that matter. Generation can take perfectly good passages and go past what they actually support. From the outside these look identical, so my only way to tell was to open the textbook and check, which is exactly what I wanted to avoid.
What I needed wasn't a more elaborate answer. It was a way to look inside the pipeline and know, with certainty, whether I could trust it and why.
Scoring each step
The way to look inside is to stop judging the answer as a whole and score each step on its own, so a bad answer can be traced back to its root cause. The approach comes from Evaluate a RAG pipeline, a guide by Frank Chen, Respan's Head of DevRel and my manager: make retrieval and generation separate spans and put an evaluator on each.
Retrieval gets two: context relevance (do the passages bear on the question) and context completeness (are they enough to answer it). Generation gets two more: groundedness (is the answer supported by those passages) and context utilization (does it use them). I added a code check of my own, citation validity: every [Title, p. N] has to match a passage that was actually retrieved, so a made-up citation is caught right where it appears.
Each answer also has an inspect view with the retrieved chunks, their scores, and the latency per step, so when an answer looks off I can see whether retrieval gave the model bad context or the model just ignored good context.
Making the scores trustworthy
Before judging answers based on your evaluators' given scores, we need to assess how good the evaluators actually are at grading. You can calibrate it against answers you've already judged yourself, adjust in small steps until it gives the expected outcome, and only then trust it on questions where you don't know the answer. Until a grader can match your expectations on things you can judge yourself, its scores on everything else mean nothing.
Mine went wrong in three ways before they went right.
They scored the wrong thing. A grader can be confidently, consistently wrong, and the number gives no hint of it. The only way to find out is to hand it a case you have already judged yourself and check that it agrees.
They had no resolution. A grader that puts every answer on two or three values cannot show you a small regression. When everything came back 1.00, the grader had stopped measuring and started agreeing.
They had nothing to compare against. One score on its own doesn't say much. Grading only became useful once it was a repeatable experiment over a fixed set of questions, where each run became comparable to the previous one.
Every fix was the same move. A grader is code, so it gets what the rest of the code gets: known inputs, an expected answer, and a re-run whenever something changes.
What the scores showed
Once the evaluators were trustworthy, the scores showed where the pipeline stood.
A simple example is a nonsense question. I asked the biology tutor about eigenvalues, which the book obviously never covers. Retrieval scored zero and generation scored high, because the tutor refused instead of guessing, and both signals were right: nothing relevant came back, and the model was correct to decline.
The results displayed an even split across the whole question set. Generation was great: verified answers, real page citations, and correct refusals when there were no sources to back a proper answer. On the other hand, retrieval sat at 0.34, and its failures were specific. Ask "Define a vector space and list all of its axioms" and three plausible chunks come back, but none of them containing the axioms. The model was handed the wrong pages, so nothing in the prompt could improve the output. That was the failure I couldn't see before, and now, thanks to the evals, it had a number on it.
Fixing what the scores caught
From there the loop was simple: change one thing, re-run the question set, and keep the change only if the numbers moved the right way. A wrong answer now came with a location attached to it, so I could aim a fix at the stage that caused it.
Some of what it caught was small. The citation check once flagged a well-cited answer at 0.14, and it turned out the check was right and I was wrong: the tutor had written compound citations like [Axler, p. 130, Exercise 16], and the check read "Exercise 16" as a page that was never retrieved. One rule added to the prompt templates, restricting the bracket to a title and a page, and the same answer came back at 1.00.
Retrieval was the bigger job, and it needed a measurement of its own, since the evaluators score the answer rather than the passages that produced it. So I built a benchmark: take a real chunk, have a model write a question that chunk answers, then check whether it comes back and at what rank. Raising the retrieval budget from three passages to eight took recall from 0.78 to 0.94.
The grader scores barely moved, which turned out to be a lesson of its own. The questions were generic definitions the books covered several times over, so three passages or eight came out the same. I rebuilt the set out of the material itself: sample a real chunk, have a model write a question only that passage can answer, then have a second model throw it out if general knowledge would have done. That runs per subject, so new PDFs get their own set without me writing one, and only then could the scores tell a good retrieval change from a bad one.
Where it is now
The tool has grown past a single book. A subject now holds a whole set of materials, several chats run against it with their own teaching styles, there's a lecture mode that pulls neighboring passages in reading order, follow-ups work, answers stream, and there's a test suite behind all of it.
The evals are what made that safe to build, and most of their value has been in what they talked me out of. Of the four retrieval upgrades I tried, three measured worse than what I already had. Hybrid keyword search dropped recall from 0.88 to 0.80, and two different rerankers came out neutral or below. All three were reasonable ideas, all three would have looked like improvements from the outside, and without the numbers I would have shipped every one of them.
That is the case for evaluating early rather than late. The scores aren't a report you write at the end, they're what tells you whether a change helped, and often the answer is that it didn't.
And that's the answer to the problem I started with. I built the tutor so I would stop checking every answer against the textbook by hand, and a fluent, cited answer was never going to let me do that. The scores are what let me. The checking I used to do by reading the source, I now do by reading the numbers, and that is what makes the tool trustworthy rather than just convincing.
