This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

I built OrderMind for my friend Ishan Kumar, who runs InTheBox, a packaging company that provides consultation, design and manufacturing.

Ishan's customers don't fill in structured order forms. They message him on WhatsApp and Instagram: a text, then a voice note, then a photo of a box they like, then "make it a little taller", then "actually 200 extra", then "same material as last time".

The real order lives across messages and media files, and Ishan has to reconstruct it himself.

In Ishan's words:

“A lot of order details are scattered across WhatsApp and Instagram, so I often have to go back through old conversations to figure out what the customer actually confirmed.”

OrderMind takes that mess and turns it into one order you can trust.

• Dump it in. Import WhatsApp conversations, voice notes, photos and PDFs.
• See what the customer actually asked for. Every extracted field keeps its source evidence and is marked CONFIRMED, INFERRED, MISSING or CONFLICTING.
• Catch problems early. "Make it 1 cm taller" becomes an inferred change that Ishan can confirm. "Same material as last time" is checked against the previous order, and a mismatch is surfaced as a conflict instead of silently guessed.
• Follow the real business workflow. Orders are handled according to InTheBox's service structure across consultation, design and manufacturing.
• Respect business boundaries. Requests outside the business scope, such as logo creation or marketing copy, are flagged rather than being treated as packaging orders.

The core principle behind OrderMind is simple:

AI must never silently turn a guess into business truth.

Demo

Watch the walkthrough

Code

How I Built It

Open-source AI at the core: Gemma

OrderMind uses Gemma as the core model for understanding customer conversations and extracting structured order information.

The model sits behind a small LLMProvider interface, which keeps the AI layer replaceable without coupling the rest of the application to one model provider.

For the deployed demo, Gemma is accessed through OpenRouter using google/gemma-3-27b-it. During development, the same interface can also work with local Ollama inference.

Extraction takes roughly 0.8 seconds per message in the current implementation.

The pipeline

The system is deliberately split into separate stages rather than relying on one giant prompt.

1. Normalize every source

WhatsApp exports, voice notes, images and PDFs are converted into a common internal message format.

1. Transcribe voice notes

Voice input is converted into text before entering the same extraction pipeline as normal messages.

1. Extract typed claims

Gemma extracts structured claims from each message and resolves references such as "same as last time".

1. Validate evidence

Every extracted claim must include an exact quote from the source message. If the evidence does not exist, the claim is rejected.

1. Build immutable order events

Claims become order events. The model does not directly write the current order state.

1. Reconstruct deterministic state

A deterministic reducer replays the events to calculate the current order state.

1. Detect changes and conflicts

Deterministic logic identifies changed values, missing fields, conflicting information and out-of-scope requests. Gemma can help phrase clarification questions, but it does not become the source of truth.

1. Human confirmation

When information is uncertain or conflicting, the user reviews and confirms it before it becomes part of the canonical order.

1. Production handoff

Once the required information and business conditions are satisfied, OrderMind can generate a structured production brief.

The important architectural decision

The most important design choice was keeping AI output separate from business state.

Instead of:

text
Customer message
→ LLM
→ update database