Work

Catching reasoning errors inside a language model

0.916 AUROC predicting answer correctness

Question

Language models give confident wrong answers. We wanted to know if the signal for “this answer is wrong” shows up in the model’s internals before the answer does.

What we built

A 1,246-problem reasoning benchmark across 8 task families, and a pipeline that captures Gemma 2 2B’s hidden states and KV cache and trains linear probes on them. The same harness runs on Llama and Qwen.

What we found

The probes predict correctness with an AUROC of 0.916, and the signal sits in a 64 to 96 dimensional subspace. Then we intervened on that subspace to see if it drives the reasoning. It doesn’t. The signal tracks correctness without causing it, which matters for anyone hoping to use it as a safety control.

Spotting synthetic voices on a live call

5-10 ms to score each audio chunk on a GPU

Problem

Cloned voices are good enough to fool people in real time. Most detectors work on finished recordings, after the call is over.

What we built

The browser streams 16 kHz microphone audio over a WebSocket. Voice activity detection drops the silence, an AASIST neural network scores each chunk as it arrives, and the result goes straight back to the page while the person is still speaking. A Microsoft Teams bot version can join a call and post an alert.

One map from scattered property records

Problem

Property records live in county systems with their own column names, formats, and gaps. Comparing parcels means stitching them together first, then adding everything the record leaves out.

What we built

A pipeline that pulls county parcel data, maps it onto one standard schema, and enriches each parcel with FEMA flood zones, USGS elevation, Census TIGER roads, ZIP codes, and opportunity zones. It scores every parcel on value, liquidity, access, and risk. A desktop viewer puts the results on a map, so you can click any parcel and see why it scored the way it did. It started with Montgomery County, Maryland, and new counties are a config entry.

Sorting recyclables from a photo

2nd place at Booz Allen Hamilton AI Codefest 2025

What we built

A ResNet50 classifier fine-tuned with transfer learning on an 80/10/10 split, plus a desktop app: drop in photos and it says which bin each one goes in.

The bug

Our first accuracy was too high for how few epochs we had trained. Early stopping and regularization didn’t change it, and neither did a stratified split. The cause was a dataset indexing bug that put the same images in both the training and validation sets. We fixed the indexing and kept the number that held up.