PillWatch: 99 percent accurate, wrong every fifth box

This summer I have been running a small research program for four high school students, my son among them. Faculty mentorship comes from Prof. Tinoosh Mohsenin’s lab at Johns Hopkins. The project is PillWatch. A camera on a Raspberry Pi watches a weekly pill organizer and reports which cells still hold pills.

The organizer has 21 cells: seven days, with morning, noon, and night for each day. That number is the whole project.

The math that reframes the problem

A per-cell classifier that is 99 percent accurate sounds finished. Raise 0.99 to the 21st power and the whole box is read correctly about 81 percent of the time. One reading in five contains an error somewhere. Nobody checks their pills one cell at a time. The unit that matters is the box.

That is the research angle. Per-cell accuracy is the headline metric in most papers, and it quietly hides a product that fails weekly. The students’ actual contribution is engineering whole-box reliability on commodity hardware, and being honest about what that takes.

The two errors are not twins

A cell that reads full when it is empty produces a false alarm. Annoying, self-correcting, cheap.

A cell that reads empty when it still holds a pill is a different animal. The system believes the dose was taken. The person who relies on the system now misses a dose, and nothing anywhere disagrees. So the design centers the false-empty rate. Thresholds tune asymmetrically to keep that failure rare, at the cost of more false alarms.

One discipline rule came with this: thresholds are chosen on validation data and frozen before the test set is scored. Tuning the threshold on the numbers you report is how projects quietly cheat.

What the hardware and models say so far

The rig is a Raspberry Pi 5 with an Arducam camera, about $60 of hardware, running everything on device. The team compared four trained models against a classic difference-of-Gaussians baseline.

A small reference-conditioned CNN gives the best overall balance so far: macro-F1 of 0.81 with only 80,000 parameters. The camouflage results, where the models truly separate, live in the data essay. Translucent vitamin D capsules under glare beat everybody.

Temporal voting, reading the box several times and voting, helps with transient errors like a hand passing through the frame. It does not help when the condition itself fools the model, because the same wrong answer just wins the vote. Testing that distinction explicitly turned out to be a finding of its own.

The data is homemade, on purpose

The team collects its own data with a capture pipeline it built on the Pi, and everything pushes to a shared repository that people and training jobs pull from. The dataset, and what building it taught the team, has its own essay.

Why I run the program this way

The students work independently during the week and compare results at a Saturday sync. They are learning the part of machine learning that tutorials skip. The metric you report is a decision, not a given.

The dataset behind this, small, condition-stratified, and being prepared for public release, gets its own essay. The team is aiming the work at the MIT Undergraduate Research Technology Conference this fall.

Code and hardware notes live in the PillWatch repository.