Vision-AI Parcel Detection for Smart Locker Kiosks
A vision-LLM pipeline that lets self-managing delivery kiosks understand their scan table from a single webcam frame—replacing custom-trained computer vision models with prompted Gemini calls.
The Problem
Parcelist builds self-managing delivery hubs for residential buildings—smart parcel lockers where couriers drop packages 24/7 with no staff involved. For that to work, the kiosk has to understand what is happening at its scan table the moment a courier places a parcel: how many parcels are present, whether the label sign is clear or obstructed, and whether a person is still standing there.
The traditional computer vision route would mean collecting and labeling image datasets for every camera type—webcam, security camera, depth camera—and retraining models whenever hardware or lighting conditions change. At the prototyping stage of a physical product, that cycle is too slow and too costly, and it turns every hardware iteration into a machine learning project.
What We Built
We replaced custom-trained models with prompted vision-LLM calls: a resized webcam frame goes to Google Gemini with a detailed natural-language prompt and a fixed JSON response schema covering parcel counts, sign emptiness and obstruction, human presence, and per-field confidence scores. Business rules layer confidence thresholds over the parsed response to drive kiosk workflow decisions—so changing detection behavior is a prompt edit, not a retraining cycle.
The contract was validated research-first in a Jupyter notebook against real labeled kiosk imagery—empty scenes, parcels present, night vision, bad lighting—then hardened into a typed, fully test-covered Scala service with an image-quality gate that rejects black or underexposed frames before spending an LLM call, and OpenTelemetry tracing and per-kiosk latency histograms throughout. Alongside the vision layer, we also built the storefront application that runs on the kiosks themselves and the shared library serving the platform’s end-to-end delivery flow.
Prompted detection instead of trained models
One Gemini call on a single frame returns parcel counts, label-sign state, and human presence as structured JSON with confidence scores—no datasets, no labeling, no per-camera retraining.
Image-quality gating
Brightness and black-frame checks reject unusable frames before they reach the model, so the kiosk never spends an LLM call on an image that cannot be interpreted.
Confidence-threshold business rules
Workflow decisions like "is the scan area empty" combine several returned confidences against tuned thresholds rather than trusting any single boolean from the model.
Production observability per kiosk
OpenTelemetry metrics and distributed tracing capture Gemini latency and error histograms for each kiosk, with trace context propagated into every model call.
How It Works in Practice
A courier walks up to an unstaffed kiosk at night and places two parcels on the scan table.
- 1
Capture and gate
The kiosk grabs a webcam frame, and the quality gate checks brightness and exposure—a black night-vision misfire is rejected and retried instead of being sent to the model.
- 2
One call, full scene
The frame goes to Gemini with the structured prompt, returning parcel count, sign state, and human presence with per-field confidence scores in a single response.
- 3
Decide with thresholds
Business rules weigh the confidences—the kiosk waits while the courier is still detected at the table, then proceeds once the scene is stable.
- 4
Route and notify
The kiosk workflow moves the parcels into the delivery flow, with every model call traced and its latency recorded against that kiosk for monitoring.
Screenshots
Engineering Approach
Research first, production second
The prompt and JSON contract were iterated in a notebook against labeled real-world imagery before any service code was written—so the hard uncertainty was resolved cheaply, then engineered once.
Typed contracts over raw model output
Constrained JSON output with low temperature and a fixed seed is decoded into typed Scala case classes, turning a probabilistic model into a dependable, testable interface.
Built to merge, not to demo
The service mirrors the packages, constructors, and DTOs of the main kiosk controller, with injected fakes for all external calls—so reintegration into production is low-risk and no test touches a live API.
Have a similar problem?
Tell us what you're working on and we'll tell you honestly whether we can help.
