Real-Time Voice Assistant Prototype

About The Project

This prototype was an early exploration into building a lightweight voice assistant capable of driving real-time UI changes through natural conversation. Designed for a mobile vehicle configurator, the assistant allowed users to speak freely without rigid command structures and see their intent translated into in-app actions, such as selecting a vehicle grade or changing the paint colour. It was designed to feel more like a quick, casual chat than a traditional voice UI.

At its core, the system used OpenAI’s Whisper model for speech-to-text transcription. The transcript was then fed into the OpenAI Chat API, along with a carefully structured system prompt that provided instructions, guardrails, and embedded business logic. The assistant’s responses weren’t just conversational, they included embedded UI triggers formatted in a custom syntax that could be parsed and executed by the host application.

A major part of the assistant’s design was the use of prompt-based programming rather than model fine-tuning. I provided the API with a layered instruction set that defined personality (“Charlie”), interaction style (e.g. speaking aloud within 15 seconds), formatting constraints, and a full catalog of product-specific actions, options, and exclusions. Each response was expected to balance natural tone with structured action syntax, effectively turning freeform user speech into safe, deterministic UI events.

The result was a flexible and responsive assistant that could react to voice input in under two seconds, guide users through a multi-step journey, and maintain a personable tone throughout. It handled dynamic conditions like checking the current scene before triggering navigation, and redirected off-topic queries back toward product features. The architecture prioritized debuggability, adaptability, and speed offering a practical model for low-latency voice control in real-world interactive applications.