← Work

Oral Exam

A working AI oral exam, built to answer one question before engineering committed: can we run this for everyone? The team decided not to pursue it.

Year
2025
Track
built
Category
prototype

The idea was an AI oral exam. A student answers out loud, and the AI examines and grades them. Students wanted it and leadership liked it. Nobody knew whether we could run it at our scale.

The first question was not whether it was good. It was whether we could run it. I decided to answer that with a working version, before engineering committed to it.

I built a working proof of concept on real AI models, with AI as my main collaborator. A student could speak, be examined and get a grade. It adjusted the difficulty as the student answered. After each answer it showed the professor's reaction, so you could tell whether it was pushing you harder or giving you more support.

The screens here are from a later version of the prototype, which asks scripted questions instead of calling a model, so anyone can open it.

The progress pill

5 states

Session
The progress pill alone, a clock icon and 14:47 counting down.
Timed

A timed session counts down, and that is all it shows. There is no number of questions to draw.

Five questions

Counted in questions, it draws one segment a question and fills them as you go. The clock counts up instead.

Ten questions

Past eight, the segments would shrink to dashes, so it says the count in words.

No limit

An open conversation has no end to count toward, so it shows the time spent and nothing else.

A course you know

Once you have practised a course, it adds how many of its topics you have mastered so far.

It worked for one student. At our scale it did not hold. The hard part was speech. A student has to be able to cut in while the examiner is talking, so the system listens the whole time and streams audio back and forth. Around that sat preparing the right exam before it starts, and all the processing after it ends. The complexity of all of it, and the cost of running it, made it simply too expensive to build for the bet we were making.

I made the case to stop. Engineering reviewed the prototype and agreed, and the team decided not to pursue it. It cost a prototype.

I built it to check the experience, the complexity and the cost, and it did.

A prototype that kills an idea is worth as much as one that ships it.

Screens

  1. Step one of three, Choose your professor. Four cards: By-the-Book Professor, who wants the formal definition in the right words; Socratic Questioner, selected, who answers you with another question until you get there; Application-Focused Professor, who asks where you would actually use it; and Surprise me. A Continue button sits at the foot.
    Prototype. You pick who examines you.
  2. An exam on Microeconomics with the Socratic Questioner. Two questions and two typed answers run down the page, and the third question quotes the student's last answer back: so goods would go to whoever queues first. But is that a cause or a consequence? The progress pill reads 2 of 4 topics mastered, three of five segments filled and 00:20. A box to type the answer sits below.
    Prototype. The next question picks up the words of your last answer.
  3. The foot of the exam screen while it listens: a round microphone button, with Always listening switched on, Type instead and Hint, 1 of 3, in a row beneath it, over a soft blue glow.
    Prototype. While it listens, a light blooms under the microphone.
  4. The report after the exam. It opens with A solid start on Microeconomics and a sentence on the main gap, then the questions and answers again, each answer marked Held with one short note. At the foot, two topics to revise next, a Schedule 3 reviews button and Practise again.
    Prototype. The report is your own answers, with a note on each.

More prototypes

See all 31