Are AI models running directly in the browser finally good enough to use in production? Let's find out! π
It's been a week since my talk at FrontK...
For further actions, you may consider blocking this person and/or reporting abuse
Awww, thanks, Lingfeng! πβ€οΈ I always say that the comments under my articles are better than the articles themselves! π
Love this! Great articles bring even better discussions π I think just quietly reading along is really fun sometimes.
Really interesting article! πΈ
I knew
WebGPUexisted, but I honestly wasnβt sure what kind of real-world scenarios it would be most useful for. Running AI models directly in the browser made it click for me, this is exactly the kind of workload where access to the GPU starts to make a lot of sense. πRight now, as you mentioned, browser AI probably works best as an optional enhancement because hardware and browser support still vary quite a bit.
But I can imagine a future where both machines and browsers become powerful enough that using several small, specialized models for different tasks becomes completely normal.
In that world,
WebGPUmight become a much more ordinary part of frontend development than it feels today. πΈExactly! Honestly, not that long ago, running LLMs in the browser felt a bit like forcing a technology into a place where it didn't really belong. But things are moving fast!
One thing I'm really curious about is your point about WebGPU becoming an ordinary part of frontend development. Because, in my opinion, WGSL (the shader language) is... hard! π It's not particularly intuitive for someone who's spent most of their career programming for the CPU.
So I wonder how we'll solve this. Will we get higher-level libraries and abstractions that make GPU programming more accessible? We already have TypeGPU, but personally, I don't think it simplifies things all that much.
Or will we just let LLMs write our shaders? π But that's not a perfect solution either. If we don't understand what's happening under the hood, AI might fix one problem while introducing three new ones.
I'm really curious to see how this evolves!
Great article, and a really interesting topic!
But I canβt help imagining a typical developer setup: multiple Chrome windows, 10+ tabs in each, and now some of those tabs running local AI models. And some customers are still using pretty old hardware, so Chrome could become a real PC performance killer (sometimes it already is, even without AI π). Of course, it depends on the hardware and how the models are used.
Iβm on my phone right now, but I definitely have to try the demos later!
Hmm, if thereβs a background remover, there must be a background adder too, right? So how do we know you didnβt just make up those FrontKon photos? π€£π€£π€£π€£ Just kidding, of course! π€£
Exactly! π These are definitely features that make sense in specific scenarios, and ideally as optional enhancements rather than something every user has to run.
But to be fair, this isn't even the first time we're doing something like this! Chrome is already doing it for us by downloading a ~4 GB Gemini Nano model. π So downloading mysterious little AI models isn't exactly something we're not already doing. π€£
And you're absolutely right about the photos! These days, a photo proves nothing. I could just as easily generate a picture of myself standing in front of the White House and claim I was there too! ππ
Hm.., now that I think about it, smaller models could actually be pretty useful for Chrome extensions or even browser games hm..π
Yeah, I remember seeing quite a few articles about Chrome silently downloading an AI model in the background. A lot of people were angry about it, or maybe just concerned about privacy and security. π
Oh, I'm definitely planning to write about Gemini Nano at some point too, because that's another really interesting one! π
And yes, Chrome extensions are a perfect use case! You don't even have to make users download a model every time they visit a website.
As for games, more and more things are being built with WASM + WebGL/WebGPU! And apparently, someone even managed to get Cyberpunk 2077 (or what is that exact number xD) running in a browser! π We're living in interesting times!
Fantastic article! This really resonates with me. π I asked the same question at the beginning of the year: Has Anyone Actually Shipped a Free, Offline Mobile LLM?. Back then, there werenβt many answers, but weβve come a long way since!
All my recent hackathon submissions now include an offline AI fallback because itβs cheap, handy, and surprisingly useful. Itβs exciting to see how much more we can do with local AI now! π
Oh, absolutely! π When I was preparing my recent WebMCP talk, I also played around with local models and experimented with Gemini Nano. Really interesting model, and it even supports tool calling! π Things are definitely getting exciting in the local AI space! π
The semantic search one surprised me the most. A 23 MB model doing that well is kind of wild. Also loved that you were honest about the ticket creator being harder than expected and the phone timing out, way more believable than the usual βAI can do everythingβ posts π Just followed you on LinkedIn too!
Thanks, Maame! π Hahaha, exactly! Browser AI is still pretty far from being able to do everything! π And thanks for connecting on LinkedIn too! β€οΈ
Really interesting experiments, especially the contrast between semantic search and structured ticket generation.
I recently ran a somewhat related experiment on a Raspberry Pi 4, comparing two small local LLMs against deterministic baselines for tasks like order-ID extraction, classification, and document Q&A. One of the more amusing results was that regex extracted 30/30 order IDs correctly, while the two language models managed 28/30 and 8/30. It reinforced something I've been thinking about: before asking where we can run a model, it's worth asking whether the task needs that kind of model in the first place.
Your semantic-search example clearly illustrates the next step. Sometimes a small, specialized model is exactly the right tool, and running it in the browser offers some compelling privacy and deployment advantages. The ticket generator illustrates the other side: even with a model capable of interpreting natural language, producing reliable structured output still requires validation and an explicit review boundary.
I also appreciate that you included hardware failures and download costs, not just the successful demos. Local inference moves computation and some costs onto the user's device; it doesn't make those costs disappear.
One thing I'm curious about: if you were extending these experiments, would you consider benchmarking the model-assisted features against non-AI implementations of the same tasks? I'd be particularly interested in where the added model complexity starts to pay off.
This is such a great comment! I especially love the regex example! π It's a perfect reminder that sometimes the simplest solution is actually the best one, and we really don't need an LLM for everything.
And I love your idea of benchmarking AI-powered solutions against traditional approaches! I'd be particularly interested in comparing semantic search with more traditional search methods. I think the results could be really interesting!
I have to admit, though, I've got quite a lot on my plate lately, so I'm not sure when I'll find the time for another round of experiments. π But if you're interested in exploring this yourself, please go ahead! I'd genuinely love to read about your findings!
And one more thing: I absolutely love how you put it, that running LLMs in the browser changes who pays the costs rather than eliminating them. That's such a good way to describe it! Because, of course, local AI isn't magically free. Someone still has to pay for all that computation. π
Thanks again for such a thoughtful comment!
Hello Sylwia, thank you for this wonderful and highly practical article! π»
Your focus on user privacy, offline capabilities, and your honest take on the real-world limits of local models is truly refreshing. The demos are brilliant and inspiring.
I visited the repository β thanks for sharing the link. Starred it βπΈ
Looking forward to your future posts π
Wishing you continued success π
Thank you so much! β€οΈ I'm really happy you found it useful! And thanks for starring the repo! π₯°
The embedding-first approach makes sense, especially for an offline product. One thing I'd watch closely is how the retrieval pipeline behaves as the local index grows. Even if generating embeddings is cheap, storing them, updating stale vectors, and ranking results can become the real bottlenecks.
For a support-ticket system, I'd also benchmark retrieval quality against actual queries, including cases where users describe the same bug with completely different wording. A lightweight model that consistently retrieves the right ticket can be far more useful than a larger model that looks impressive in a demo.
That's a really interesting point! And I completely agree, there are definitely questions around scalability, incremental updates, retrieval quality, and so on.
Of course, my demo is nowhere near production-ready (otherwise I'd be selling it instead of giving it away as a demo! π). There's still a lot to test and optimize.
But I genuinely think this is a very promising direction worth exploring. Especially because, as you pointed out, the embedding model is so small! The relatively low cost of running it locally makes the whole approach particularly interesting. π
One thing I find interesting here is the offline-first use case. Imagine a field-service app with thousands of repair manuals cached locally. You could generate embeddings on-device and search them without sending queries or documents to a server.
The tricky part is updating that local index when the embedding model changes. You'd need to keep the old index usable while rebuilding the new one, especially on devices with limited storage. That's where the small model size becomes a real architectural advantage, not just a nice demo feature.
Exactly! Offline-first is becoming an increasingly common approach, and I think this is a great example of where local AI could really shine!
As for rebuilding the index when the embedding model changes, that's definitely an interesting challenge. But I think it's worth distinguishing between updating documents and updating the model itself. While document embeddings may need to be refreshed regularly, replacing the entire embedding model would probably happen much less frequently.
In a real-world application, you could stick with a specific model version for quite a long time and only rebuild the entire index when a new model offers a meaningful improvement in retrieval quality.
Still, plenty of interesting architectural challenges to explore here!
That distinction matters. Imagine a field-service app where technicians store repair logs offline, then sync them days later. A model upgrade could leave the local index with vectors generated by the old model while new queries use the new one. Even if the vector dimensions match, the embedding spaces might not, so retrieval quality could quietly degrade.
I'd version the model, tokenizer, and preprocessing pipeline together, then build the new index alongside the old one and run a retrieval evaluation before switching. For example, test whether a query like βhydraulic pressure drops after 20 minutesβ still retrieves tickets describing the same fault as βpump loses pressure once the system heats up.β That kind of semantic mismatch is where a small offline model needs to prove itself beyond the demo.
Thanks, Sylwia, for the code and the live demo. I will be testing the demos soon. I would ship the embedding model before the ticket model. Item 4βs embedding model is about 23 MB, nearly zero cost to the user, whereas the ticket model weighs in at 786 MB, with much higher cost.
Mobile data is not free for many users. It costs real money. I havenβt been able to test models in a browser yet, but file size is important for my project (StacksNG), which is completely offline.
Which two of your demos timed out on your phone? Did the semantic search work? Iβll let you know what I find out.
Hey Daniel, thanks for the comment! And you're absolutely right! Mobile data isn't free, and that's a really good point. One possible safeguard would be to only allow model downloads over Wi-Fi.
That's also why I think we're not quite at the point where we should rely on these models for mission-critical tasks.
On my phone, two demos didn't work: the ticket creation demo and the last one using the camera. The others worked, although they were noticeably slower than on desktop.
The exception was the embeddings model, which actually ran almost as fast as on my desktop! Although, to be fair, I had a WebAssembly fallback there. π
Looking forward to hearing how your tests go!
Saved for research future purposes.ποΈ
That's awesome! Hope it comes in handy! πβ€οΈ
Running heavy processes directly in the browser is fascinating. I completely agree with the point in the comments about using Web Workers so the main thread doesn't freeze. I've been focusing heavily on pure Vanilla JS lately to keep real-time text processing blazing fast without relying on bloated frameworks. For these intense in-browser features, do you think sticking to Vanilla JS is necessary for maximum performance, or is the overhead of React negligible compared to the WebGPU load?
I'd say the overhead of React is negligible in most cases. If inference runs on the GPU and React is mainly responsible for the UI, the framework itself usually won't be the bottleneck. Avoiding main thread blocking and unnecessary re-renders matters much more. So I definitely wouldn't give up React just to use WebGPU efficiently!
the small specialized model beating the general one is the lesson that stuck with me here. i keep hitting the opposite problem β trying to squeeze a big model onto hardware that was never meant for it, and the quality drop shows up fast. the semantic search demo is the one i would actually ship. embedding a support ticket base locally and never sending it to a server is the privacy win that makes the whole local-inference pitch worth it. the webcam pause-and-ask is fun but i agree, not reliable enough for accessibility yet. transformers.js getting to this point in a year is the real story. the gap between "fun curiosity" and "actually useful" closed a lot.
Exactly! I was also genuinely surprised by how well semantic search works with these local models! And it's not just about privacy. We don't always have internet access, after all. Some people spend hours using apps out in the field, often with little or no connectivity. So having these features work entirely offline can be a huge advantage!
Although most of these are pretty cool, I would actually prefer your initial bug report in number three.
It has way more context, way more little side details that may or may not matter, and generally just more. Yeah, 90% of the bugs you submit will be read, immediately recognized, and pushed into a Bug#1234 bucket. But for those last 10%, the details can matter.
I totally get your point! π Unfortunately, you know how it is with support tickets. You have to fill in all those fields and end up spending half an hour figuring out what goes where. π But hey, that stream of consciousness could easily be stored in an additional field too! Anyway, I dream of a feature like this every single time I have to fill out a support ticket! π
The download sizes make the optional-enhancement argument very concrete. For the ticket creator, Iβd want a visible download size and cancel button before loading nearly 800 MB, with the normal ticket form usable throughout. Cold-load UX seems just as important here as inference speed.
That's a fair point! π I think it's especially important on mobile devices, but on desktop I'm not entirely convinced a separate confirmation step is always necessary. I'd definitely show the download size and progress, though, and keep the regular form fully functional while the model loads. And of course, I wouldn't download 800 MB automatically without the user choosing to use the feature! π Cold-load UX absolutely matters, but I think the exact approach depends on the device and use case.
The WebGPU adoption number tracks with what I've seen β I ran a quick check last month across 12k visitors on a side project and only ~8% had WebGPU enabled in their browser. Most of those were Chrome on desktop. So yeah, it's fast when it works, but you're shipping a fallback path for 90%+ of users and that's the part nobody talks about. Would love to see the actual production numbers if you've shipped this to real traffic.
Thanks for sharing these numbers! That's a really interesting point, and I absolutely agree that browser support is something we need to consider. But I also think it depends a lot on the project and its target audience. For example, in the applications I work on, most users have WebGPU enabled, simply because they're primarily desktop applications. And in some scenarios, we can even make WebGPU support a requirement.
Also, WebGPU adoption is progressing, and this isn't exactly a new problem in web development. Remember when we had to transpile everything because browsers didn't fully support ES6? π We're seeing similar challenges with newer technologies today. WebMCP is another interesting example: there's a lot of excitement around it, but people sometimes forget that it's still experimental!
That said, your point is absolutely valid. We need to consider actual adoption numbers, especially when building applications for a broad audience. And that's exactly why I keep emphasizing that, at least for now, browser AI shouldn't be relied on for mission-critical functionality without a reliable fallback.
Thanks again for bringing some real-world numbers into the discussion!
RMBG-1.4 at about 170 MB is smaller than I expected for background removal that clean, and running inference in a Web Worker so the UI stays responsive is the detail I'd copy first. I've mostly treated in-browser models as toys, so calling them optional enhancements, like Florence-2 drafting alt text, makes sense to me. How long does that first model download take on a mid-range phone?
On a phone, it actually takes quite a while, around a minute! π The other models are pretty quick, except for the semantic search one. But I expect things to improve, especially now that WebGPU is becoming available in mobile browsers, including on iPhones!
This is a refreshingly honest and practical look at browser AI.
Most pieces either hype local models as βthe futureβ or dismiss them as toys. You did the useful middle path: built five real features, measured what actually works, and was transparent about the gaps (model sizes, mobile WebGPU, inconsistent quality on structured output and contextual alt text).
Semantic search with the tiny embedding model and the background remover feel especially production-viable right now as optional enhancements. The Smart Ticket Creator is the one that best illustrates both the promise and the friction β getting reliable structured output from a 0.5B model is still more prompt + validation work than people expect.
Appreciate the clear βoptional enhancement, not critical pathβ stance. Thatβs the right framing for 2026.
Thank you so much, Anne! β€οΈ That's exactly what I wanted to show with this article! I think browser-based AI has reached a point where we can start building genuinely useful features with it, but we still need to be realistic about its limitations. And yes, the Smart Ticket Creator was definitely a good reminder that getting a model to generate something is one thing, but getting reliable, structured output is a whole different challenge! π I'm really excited to see where we'll be a year from now!
This looks good. What can I use it for? Is it only for web applications, or is it possible to use it for other things as well?
Unfortunately, this particular implementation is web-only, but the same concept can definitely be applied to native apps too!
Thats not good.
I'll be honest β I had never looked into browser-based AI before this, and I didn't know any of it was possible. Reading this was a genuine surprise.
What made it click for me is the semantic search example. A 23 MB model doing something more useful than keyword search is a much lower barrier than I assumed this kind of thing had. It made me think about things I could actually try, not just read about.
My background is mostly backend and ordinary frontend work, so I'm starting from zero on WebGPU and Transformers.js. If someone wanted to learn this properly, what would you point them to first β the Transformers.js docs, or something else you'd recommend before that?
Hey, thanks for your comment! π If you're interested in running AI models in the browser, the Transformers.js docs are definitely enough to get started! If you want to dive deeper into WebGPU itself, that's a whole different story involving shaders and GPU-specific code. But honestly, I don't think you need to go that deep unless you have a specific reason to. It's worth understanding what WebGPU is and how it works, but you don't need to master it to start building cool things with browser-based AI! π
The WebGPU to WASM fallback is doing more work than the headline. Most tutorials ship WebGPU only β graceful degradation is what makes this production viable on devices without GPU access yet.
Also the Web Workers architecture is the unglamorous detail nobody demos. Inference on the main thread means your UI freezes halfway through a sentence. Offloading that before you add features rather than retrofitting it later is the call most teams get backwards.
Curious if you hit cold start latency on the first inference call? Loading 44M params even compressed has a perceptible pause on slower connections. Did you add any loading UX or let it be?
Exactly! I keep saying that WebGPU still needs a fallback! And yes, of course I added a loading state so users know the model is being loaded. It's pretty basic, since this is just a demo and not a production app, but informative enough! π And I totally agree about Web Workers. They're an absolute must here!
I donβt know why, but looking at these images makes me happy π
Aww it's so sweet! Indeed, I was very happy there βΊοΈ
You can spend quite some time with that background removal, this used to be a painful process in Photoshop, back in the old days.
Hahaha, oh yes, I remember those days! π And the result often looked so obviously cut out, you could spot it immediately! π
It was an interesting article.
Thanks a lot βΊοΈ
Can Do everthing i want.
Ah, clever!!!
What a fantastic article Sylwiaπ I knew about WebGPU and Transformers.js but I always assumed browser AI was still a fun toy. Your article changed that view completely.
The Smart Ticket Creator demo is what sold me. Watching a 500M parameter model turn messy text into structured JSON, all locally without an API call, feels like a glimpse of where things are going.
Curious: did you find any specific trick that helped with JSON consistency, or was it mostly prompt engineering plus Zod validation?
Wow A live demo? You got my star!
This is seriously impressive!
That's something I would actually use!
By the way, currently it shows "removed" on the original side and "original" on the "removed" side.
Hahaha, you're right! π I'll have to fix that later! But come on, you have to admit it's ridiculously fun to play with! π€£