Are AI models running directly in the browser finally good enough to use in production? Let's find out! π
It's been a week since my talk at FrontKon in Prague. As I mentioned before, it was fantastic! I'm adding a few photos below, just to prove that I actually speak at conferences and don't just make these things up. π
 |
 |
 |
 |
 |
 |
My talk was titled "Your Browser Is a Compute Platform (And You're Using It for Forms)", and I showed how we can use WebAssembly and WebGPU to perform advanced computations directly in the browser. I was a little concerned to discover that, out of everyone in the audience, maybe three or four people had even heard of WebGPU! And that's a shame, because WebGPU can accelerate certain types of parallel computations by tens of times compared to CPU-based implementations.
But okay, at least the people at FrontKon know what WebGPU is now. π And if you'd like a talk about this at your conference, feel free to reach out! Especially if your conference happens to take place in a particularly attractive tourist destination. π
Why I'm Revisiting Browser AI
After a conference, I traditionally prepare a blog post related to my talk. However, since I've already written about WebAssembly and WebGPU many times, today I'd like to approach the topic from a different angle. Namely, let's take a look at the current state of AI models running directly in the browser, which, of course, make extensive use of these technologies. π
Browser-based AI models are a fascinating concept. No inference API calls. No network round trips to an AI backend. No worrying that your users will consume too many tokens, because you're not paying for them! But of course, local inference still takes time, and downloading the model in the first place can take a while.
Another advantage is that these models can work offline once they've been downloaded and cached by the browser, provided the application itself supports offline operation.
And that also brings a significant privacy benefit: inference happens directly on the user's device, so their data doesn't need to be sent to an external AI service.
However, as you can probably imagine, these are relatively small models. We're not going to run a massive frontier-scale model in a typical browser tab.
So the question is: how useful are these smaller models, really?
Some of you may remember that I experimented with browser-based AI models about a year ago. Back then, we were mostly laughing that they were a fun curiosity, but not much more than that.
But I've been following people from Hugging Face who work on Transformers.js on LinkedIn, and every now and then, their demos would appear in my feed. At first glance, some of them looked genuinely impressive. You know, practically spaceships running inside a browser. π
So before FrontKon, I decided to take another look at what they had to offer. And it turned out that after upgrading Transformers.js to version 4.3.1, which I used for these experiments, and trying some newer models, the results were genuinely impressive!
Of course, Transformers.js isn't the only option for running AI models in the browser. There are other libraries and runtimes, but I really like Transformers.js, and I think it's a pretty representative example of what this technology can achieve today.
At the same time, I don't work for Hugging Face, and nobody is paying me to sell you a spaceship in your browser. So I'll tell you what browser-based AI models can actually do today, what works well, and what still doesn't.
And even better, you can explore the code and try everything yourself!
GitHub: https://github.com/sylwia-lask/practical-browser-ai
Live demo: https://sylwia-lask.github.io/practical-browser-ai
If you like the project, please give it a star on GitHub. I'm collecting them! π
Can We Actually Use This in Production?
For these demos, I wanted to focus on practical applications. More specifically, I wanted to find out how we could use browser-based AI today, right now, in real production applications.
So is this technology worth paying attention to? In my opinion, absolutely!
But!
I probably wouldn't build a mission-critical system that depends entirely on these models working perfectly. π Instead, I'd treat them as interesting, optional enhancements to existing application features, especially in areas like e-commerce, accessibility, and customer support.
The application itself is nothing particularly exotic, by the way. It's a regular React application written in TypeScript, using Web Workers to run model inference without blocking the main UI thread.
The models are loaded on demand, so you don't need to download all five of them just to open the application. The inference runs locally through Transformers.js, using WebGPU where supported and WebAssembly-based CPU execution where a fallback is available.
No AI backend involved!
Let's see what I managed to build. And at the end, I'll share my thoughts on the limitations of these models.
One more thing: these aren't browser-based AI agents yet. They're simply applications using local AI models for specific tasks. But give me a few more weeks, and you'll see agents running in the browser on my blog too!
1. Background Remover
One of the things I particularly appreciate about browser-based AI is that it's not limited to text generation. I love working with multimodal AI!
In this demo, a small model called RMBG-1.4, with approximately 44 million parameters, does a really impressive job of separating an object from its background. π The model itself is only about 170 MB to download in the configuration I'm using.
As you can see, we can then save the result as a transparent PNG, and it works surprisingly well!
I can easily imagine this as an additional feature on auction websites or e-commerce platforms.
Imagine someone trying to sell a product but only having a photo like my example. Instead of requiring them to use an external image-editing service, we could simply offer a background removal button directly in the application.
And the photo doesn't even need to leave their device. Pretty cool, isn't it?
One practical note: RMBG-1.4 comes with licensing conditions that should be checked before commercial deployment. So while the technical concept is very promising for e-commerce, you should always verify the model's license before shipping it.
2. Alt Text Generator
I've always felt that AI could become another useful layer of accessibility, if we use it properly.
This demo uses Florence-2-base-ft, a vision model with approximately 230 million parameters, to generate descriptions of images.
Imagine that you're uploading an image to DEV, for example. Instead of having to write an alt text description from scratch, you could use a model like this to generate a suggestion automatically. Especially when you really don't feel like describing what's in the picture!
Of course, the generated description should still be reviewed and edited by a human. Good alt text isn't just about identifying objects in an image. It also depends on the context and purpose of that image.
The model doesn't necessarily understand why you're using the picture, and it can occasionally miss or invent details. Still, as an accessibility assistant, I think this is a very promising approach.
There's also an option to provide additional context, but don't expect too much from it. In my implementation, the extra text is simply appended to the generated description rather than genuinely influencing what the model sees or understands. I'll explain why I wasn't satisfied with this approach later in the article.
For reference, the model download is roughly 350β450 MB, depending on the precision supported by the device.
3. Smart Ticket Creator
Oh, this one is my dream feature! I describe what's wrong in completely natural, messy language, and the model turns it into a structured ticket that I can send to support. π
The model I ended up using is Qwen2.5-0.5B-Instruct, a small language model with approximately 500 million parameters. In my configuration, its quantized weights require a download of around 786 MB, plus tokenizer files.
And I have to admit something. When I started implementing this feature, I thought it would be the easiest one! Seriously. How difficult could it be for a language model to turn a complaint into a structured ticket?
Well... Apparently, quite difficult! I spent much more time than expected trying different models and getting them to produce the output I actually needed.
The main challenge wasn't just understanding the complaint. It was producing a consistent, structured response with the correct fields, without inventing information or returning malformed JSON. Eventually, I got it working using a carefully designed prompt, output parsing, and schema validation with Zod.
The generated ticket is editable, so the user can review everything before copying or submitting it. And I think that's exactly the kind of approach that makes sense for production: let the AI do the boring initial work, but keep the human in control.
4. Semantic Search
Now, this is where small models really shine! For this demo, I implemented semantic search over a collection of support tickets.
Instead of looking for matching keywords, the application searches for tickets with a similar meaning. And, of course, I also included a traditional keyword search so you can compare the results yourself.
I'm using all-MiniLM-L6-v2, a small embedding model with approximately 22 million parameters. Depending on the backend, its model weights are only about 23 MB for the quantized CPU version or 90 MB for the WebGPU version.
And it works really well! The model is fast, and the results are genuinely useful.
Under the hood, it converts the tickets and the search query into numerical vectors, then compares them using cosine similarity. No generative LLM is needed here.
In fact, that's one of the most interesting lessons from these experiments: sometimes a small, specialized model is a much better choice than a general-purpose language model. This is particularly good news for applications where privacy matters and you don't want to send potentially sensitive information to an external server.
For a small collection of documents, this approach is definitely worth testing. Of course, for larger datasets you'd need to think about indexing, storage, and performance. In my demo, the embeddings are kept in memory, which is perfectly sufficient for the example. But the basic idea works surprisingly well.
5. Pause & Ask
Okay, this one might be a little less practical, but it's definitely fancy! π And I think it also has interesting accessibility potential. For this demo, I'm using LFM2.5-VL-450M, a vision-language model with approximately 450 million parameters. The download is roughly 350β400 MB, depending on the configuration.
There are actually two features here. The first is live visual description. I turn on my webcam, and the model periodically describes what it sees. Yes, that's me:
But then it gets even better! I can pause the camera on a particular frame and ask the model questions about what it sees.
And here's the example showing that the model isn't exactly perfect. π Although, well, I'm certainly flattered!
I wouldn't treat this as a reliable accessibility tool just yet, but as an experiment in local visual question answering, it's genuinely fascinating.
And that's all five demos!
So What's the Catch?
Now, as promised, let's talk about the disadvantages. Because, of course, there are quite a few!
WebGPU Support and Hardware Limitations
First of all, these models run as well as they do largely thanks to WebGPU. This API allows applications to use GPU computing capabilities directly in the browser, which can make a huge difference for machine learning inference.
The problem is that WebGPU availability and performance aren't the same on every device. Support has improved significantly across modern browsers, but availability still depends on the browser version, operating system, GPU, drivers, etc. And even if WebGPU is supported, that doesn't mean everyone has an equally powerful computer.
Mobile devices are a particularly interesting problem. Some mobile browsers and devices still lack usable WebGPU support for certain workloads. In those cases, some Transformers.js models can fall back to WebAssembly-based CPU inference, potentially using SIMD or multithreading depending on the environment and configuration.
Unfortunately, this can be significantly slower. On my phone, two of the examples from my demo didn't work at all because they timed out. And it's not like I'm using some ancient phone from a museum! It's a Samsung Galaxy S24 FE. Not the newest flagship anymore, but definitely not a dinosaur either.
So yes, WebGPU support is improving, and the models are getting better. But I want to talk about what we can actually use right now, not what might be possible in a few years.
Model Downloads Aren't Exactly Tiny
There's also another important limitation: the models themselves have to be downloaded. And while some are relatively small, others can require hundreds of megabytes. As you've seen, the models in my demo range from around 23 MB for the quantized semantic search model to nearly 800 MB for the Smart Ticket Creator. That's quite a lot if you're visiting a website on a mobile connection!
Fortunately, browsers can cache the downloaded model files, so subsequent visits usually don't require downloading everything again. But the first load still matters, and browser storage isn't guaranteed to keep those files forever.
That's another reason why I'd treat these features as optional enhancements rather than something every user must load before they can use the application.
Some Things Still Don't Work Well Enough
I also couldn't implement everything I originally wanted, even though some ideas seemed perfectly reasonable. For example, I tried adding text recognition from images. And despite experimenting with different models and configurations, I couldn't get results that were reliable enough for my demo. The approaches I tested didn't give me the quality I wanted, so I decided not to include that feature.
Another thing I wanted to improve was the contextual information in the alt text generator. Imagine uploading a photo of a dog and adding some additional context, such as: "This dog is available for adoption at a local animal shelter." Ideally, the model would take that information into account and generate a beautiful, context-aware description.
I tried using an additional local language model for this, hoping it could combine the visual description with the extra information. But no matter what I tried, the results were very unsatisfactory. So I ended up with the simpler approach I mentioned earlier, where the context is added separately rather than genuinely interpreted by the image-captioning model.
And that's another good example of why I wouldn't blindly trust these models in production.
What About Voice Recognition?
On the other hand, I've heard that small browser-based models can be surprisingly good at speech recognition. Transformers.js supports audio-related tasks too, so that's definitely another interesting area to explore.
I didn't test it this time, though. Mostly because I was building these demos in the evenings, and my family would have killed me if I'd spent hours talking to my computer. π Maybe next time!
My Verdict: Is Browser AI Production-Ready?
So, to sum up: browser-based AI models are getting better, and I think it's already worth exploring how to use them in production applications.
Not necessarily as a replacement for every AI backend. And definitely not as a critical dependency without a fallback. But as optional features that improve the user experience? Absolutely.
What do you think? Have you already experimented with running AI models directly in the browser?
And if you enjoy reading my articles, feel free to follow me on LinkedIn too! π
Top comments (58)
Awww, thanks, Lingfeng! πβ€οΈ I always say that the comments under my articles are better than the articles themselves! π
Love this! Great articles bring even better discussions π I think just quietly reading along is really fun sometimes.
Really interesting article! πΈ
I knew
WebGPUexisted, but I honestly wasnβt sure what kind of real-world scenarios it would be most useful for. Running AI models directly in the browser made it click for me, this is exactly the kind of workload where access to the GPU starts to make a lot of sense. πRight now, as you mentioned, browser AI probably works best as an optional enhancement because hardware and browser support still vary quite a bit.
But I can imagine a future where both machines and browsers become powerful enough that using several small, specialized models for different tasks becomes completely normal.
In that world,
WebGPUmight become a much more ordinary part of frontend development than it feels today. πΈExactly! Honestly, not that long ago, running LLMs in the browser felt a bit like forcing a technology into a place where it didn't really belong. But things are moving fast!
One thing I'm really curious about is your point about WebGPU becoming an ordinary part of frontend development. Because, in my opinion, WGSL (the shader language) is... hard! π It's not particularly intuitive for someone who's spent most of their career programming for the CPU.
So I wonder how we'll solve this. Will we get higher-level libraries and abstractions that make GPU programming more accessible? We already have TypeGPU, but personally, I don't think it simplifies things all that much.
Or will we just let LLMs write our shaders? π But that's not a perfect solution either. If we don't understand what's happening under the hood, AI might fix one problem while introducing three new ones.
I'm really curious to see how this evolves!
Great article, and a really interesting topic!
But I canβt help imagining a typical developer setup: multiple Chrome windows, 10+ tabs in each, and now some of those tabs running local AI models. And some customers are still using pretty old hardware, so Chrome could become a real PC performance killer (sometimes it already is, even without AI π). Of course, it depends on the hardware and how the models are used.
Iβm on my phone right now, but I definitely have to try the demos later!
Hmm, if thereβs a background remover, there must be a background adder too, right? So how do we know you didnβt just make up those FrontKon photos? π€£π€£π€£π€£ Just kidding, of course! π€£
Exactly! π These are definitely features that make sense in specific scenarios, and ideally as optional enhancements rather than something every user has to run.
But to be fair, this isn't even the first time we're doing something like this! Chrome is already doing it for us by downloading a ~4 GB Gemini Nano model. π So downloading mysterious little AI models isn't exactly something we're not already doing. π€£
And you're absolutely right about the photos! These days, a photo proves nothing. I could just as easily generate a picture of myself standing in front of the White House and claim I was there too! ππ
Hm.., now that I think about it, smaller models could actually be pretty useful for Chrome extensions or even browser games hm..π
Yeah, I remember seeing quite a few articles about Chrome silently downloading an AI model in the background. A lot of people were angry about it, or maybe just concerned about privacy and security. π
Oh, I'm definitely planning to write about Gemini Nano at some point too, because that's another really interesting one! π
And yes, Chrome extensions are a perfect use case! You don't even have to make users download a model every time they visit a website.
As for games, more and more things are being built with WASM + WebGL/WebGPU! And apparently, someone even managed to get Cyberpunk 2077 (or what is that exact number xD) running in a browser! π We're living in interesting times!
Fantastic article! This really resonates with me. π I asked the same question at the beginning of the year: Has Anyone Actually Shipped a Free, Offline Mobile LLM?. Back then, there werenβt many answers, but weβve come a long way since!
All my recent hackathon submissions now include an offline AI fallback because itβs cheap, handy, and surprisingly useful. Itβs exciting to see how much more we can do with local AI now! π
Oh, absolutely! π When I was preparing my recent WebMCP talk, I also played around with local models and experimented with Gemini Nano. Really interesting model, and it even supports tool calling! π Things are definitely getting exciting in the local AI space! π
Hello Sylwia, thank you for this wonderful and highly practical article! π»
Your focus on user privacy, offline capabilities, and your honest take on the real-world limits of local models is truly refreshing. The demos are brilliant and inspiring.
I visited the repository β thanks for sharing the link. Starred it βπΈ
Looking forward to your future posts π
Wishing you continued success π
Thank you so much! β€οΈ I'm really happy you found it useful! And thanks for starring the repo! π₯°
The semantic search one surprised me the most. A 23 MB model doing that well is kind of wild. Also loved that you were honest about the ticket creator being harder than expected and the phone timing out, way more believable than the usual βAI can do everythingβ posts π Just followed you on LinkedIn too!
Thanks, Maame! π Hahaha, exactly! Browser AI is still pretty far from being able to do everything! π And thanks for connecting on LinkedIn too! β€οΈ
Really interesting experiments, especially the contrast between semantic search and structured ticket generation.
I recently ran a somewhat related experiment on a Raspberry Pi 4, comparing two small local LLMs against deterministic baselines for tasks like order-ID extraction, classification, and document Q&A. One of the more amusing results was that regex extracted 30/30 order IDs correctly, while the two language models managed 28/30 and 8/30. It reinforced something I've been thinking about: before asking where we can run a model, it's worth asking whether the task needs that kind of model in the first place.
Your semantic-search example clearly illustrates the next step. Sometimes a small, specialized model is exactly the right tool, and running it in the browser offers some compelling privacy and deployment advantages. The ticket generator illustrates the other side: even with a model capable of interpreting natural language, producing reliable structured output still requires validation and an explicit review boundary.
I also appreciate that you included hardware failures and download costs, not just the successful demos. Local inference moves computation and some costs onto the user's device; it doesn't make those costs disappear.
One thing I'm curious about: if you were extending these experiments, would you consider benchmarking the model-assisted features against non-AI implementations of the same tasks? I'd be particularly interested in where the added model complexity starts to pay off.
This is such a great comment! I especially love the regex example! π It's a perfect reminder that sometimes the simplest solution is actually the best one, and we really don't need an LLM for everything.
And I love your idea of benchmarking AI-powered solutions against traditional approaches! I'd be particularly interested in comparing semantic search with more traditional search methods. I think the results could be really interesting!
I have to admit, though, I've got quite a lot on my plate lately, so I'm not sure when I'll find the time for another round of experiments. π But if you're interested in exploring this yourself, please go ahead! I'd genuinely love to read about your findings!
And one more thing: I absolutely love how you put it, that running LLMs in the browser changes who pays the costs rather than eliminating them. That's such a good way to describe it! Because, of course, local AI isn't magically free. Someone still has to pay for all that computation. π
Thanks again for such a thoughtful comment!
The embedding-first approach makes sense, especially for an offline product. One thing I'd watch closely is how the retrieval pipeline behaves as the local index grows. Even if generating embeddings is cheap, storing them, updating stale vectors, and ranking results can become the real bottlenecks.
For a support-ticket system, I'd also benchmark retrieval quality against actual queries, including cases where users describe the same bug with completely different wording. A lightweight model that consistently retrieves the right ticket can be far more useful than a larger model that looks impressive in a demo.
That's a really interesting point! And I completely agree, there are definitely questions around scalability, incremental updates, retrieval quality, and so on.
Of course, my demo is nowhere near production-ready (otherwise I'd be selling it instead of giving it away as a demo! π). There's still a lot to test and optimize.
But I genuinely think this is a very promising direction worth exploring. Especially because, as you pointed out, the embedding model is so small! The relatively low cost of running it locally makes the whole approach particularly interesting. π
One thing I find interesting here is the offline-first use case. Imagine a field-service app with thousands of repair manuals cached locally. You could generate embeddings on-device and search them without sending queries or documents to a server.
The tricky part is updating that local index when the embedding model changes. You'd need to keep the old index usable while rebuilding the new one, especially on devices with limited storage. That's where the small model size becomes a real architectural advantage, not just a nice demo feature.
Exactly! Offline-first is becoming an increasingly common approach, and I think this is a great example of where local AI could really shine!
As for rebuilding the index when the embedding model changes, that's definitely an interesting challenge. But I think it's worth distinguishing between updating documents and updating the model itself. While document embeddings may need to be refreshed regularly, replacing the entire embedding model would probably happen much less frequently.
In a real-world application, you could stick with a specific model version for quite a long time and only rebuild the entire index when a new model offers a meaningful improvement in retrieval quality.
Still, plenty of interesting architectural challenges to explore here!
That distinction matters. Imagine a field-service app where technicians store repair logs offline, then sync them days later. A model upgrade could leave the local index with vectors generated by the old model while new queries use the new one. Even if the vector dimensions match, the embedding spaces might not, so retrieval quality could quietly degrade.
I'd version the model, tokenizer, and preprocessing pipeline together, then build the new index alongside the old one and run a retrieval evaluation before switching. For example, test whether a query like βhydraulic pressure drops after 20 minutesβ still retrieves tickets describing the same fault as βpump loses pressure once the system heats up.β That kind of semantic mismatch is where a small offline model needs to prove itself beyond the demo.
Saved for research future purposes.ποΈ
That's awesome! Hope it comes in handy! πβ€οΈ
Thanks, Sylwia, for the code and the live demo. I will be testing the demos soon. I would ship the embedding model before the ticket model. Item 4βs embedding model is about 23 MB, nearly zero cost to the user, whereas the ticket model weighs in at 786 MB, with much higher cost.
Mobile data is not free for many users. It costs real money. I havenβt been able to test models in a browser yet, but file size is important for my project (StacksNG), which is completely offline.
Which two of your demos timed out on your phone? Did the semantic search work? Iβll let you know what I find out.
Hey Daniel, thanks for the comment! And you're absolutely right! Mobile data isn't free, and that's a really good point. One possible safeguard would be to only allow model downloads over Wi-Fi.
That's also why I think we're not quite at the point where we should rely on these models for mission-critical tasks.
On my phone, two demos didn't work: the ticket creation demo and the last one using the camera. The others worked, although they were noticeably slower than on desktop.
The exception was the embeddings model, which actually ran almost as fast as on my desktop! Although, to be fair, I had a WebAssembly fallback there. π
Looking forward to hearing how your tests go!