--- title: "08 — Local and Federated Inference: Ollama, LM Studio, and OpenRouter" uid: local-and-federated-inference tags: ["openrouter", "ollama", "tools", "inference", "lm-studio", "roadmap:ai-engineer", "local-llm"] excerpt: "Locked to one closed provider? There are two escapes, at opposite ends of the spectrum: run open models on your own machine, or federate many providers behind one API." date: 2026-08-13T03:28:38+0000 source: https://www.aveshina.my.id/en/blog/local-and-federated-inference --- Being locked to one closed provider used to feel like a fact of life — you call the API, you pay the bill, you accept the limits. The realization that broke it: **there are two ways to escape "I'm locked to one closed provider," and they sit at opposite ends.** [1] On one end, Ollama and LM Studio run open models on my own machine — no network, no per-token bill, total privacy. On the other end, OpenRouter federates many providers behind one API — I get the breadth of the whole model market without managing a dozen API keys. They solve different problems, and confusing them leads to the wrong tool. The framing that finally landed is a spectrum of _where the compute happens_. At one extreme, the model runs on my laptop — maximum control, zero dependency on a vendor, capped by my hardware. At the other, the model runs in a vendor's cloud — maximum capability, zero local burden, dependency on the vendor. OpenRouter is the interesting middle: the compute still happens in vendors' clouds, but a single proxy in front of all of them means I'm no longer coupled to any one. ```figure ``` ## Ollama: local inference, made boring **Ollama** is the tool that made running an LLM on my own laptop feel routine. It packages open-weight models — Llama, Qwen, Gemma, Mistral, and many others — into a single CLI that handles downloading, quantizing, and serving the model on my machine [1]. I install it, run ollama run llama3, and I have a local model answering prompts with no network call. The SDK lets me integrate it into applications the same way I would a cloud API, except the endpoint is localhost. What Ollama really gives me is _privacy and predictability_. The data never leaves my machine, there is no per-token cost, and the model behaves the same tomorrow as today because no vendor can change it. The honest constraint is hardware: a laptop runs small-to-medium models well, and big models are slow or impossible locally. Ollama is the right call for development, for privacy-sensitive features, for offline scenarios, and for anything where I want zero ongoing inference cost. ## LM Studio: local with a GUI **LM Studio** is the desktop-application sibling of Ollama [2]. Same core idea — discover, download, and run open-source LLMs locally — but wrapped in a graphical interface designed for experimentation rather than scripting. I reach for it when I want to browse models, chat with several of them side by side, and compare outputs without writing code. For production wiring I prefer Ollama's CLI and SDK; for evaluation and model-shopping, LM Studio's UI earns its place. They're complements, not competitors — both run the same open weights on the same machine, just with different ergonomics. ## OpenRouter: one API, many providers **OpenRouter** solves the opposite problem. Instead of running models locally, it sits in front of many cloud providers — OpenAI, Anthropic, Google, Mistral, and others — and exposes them all through a single, OpenAI-compatible API [3]. One API key, one request format, one billing relationship, and behind it the entire model market. The value isn't just convenience. OpenRouter changes what I can do at the architecture level: - **Model portability.** I can switch the model in one line, compare providers on the same task, and avoid vendor lock-in. - **Fallback and resilience.** If one provider has an outage, I can route to another. The proxy makes provider-switching a config change, not a rewrite. - **Price and capability shopping.** I can pick the cheapest model that meets my quality bar per request, or route different query types to different models. The tradeoff is an extra hop — my request goes through OpenRouter before reaching the provider — which adds a small amount of latency and introduces one more dependency. For most features that trade is overwhelmingly worth it; for latency-critical paths I might call the provider directly. ## Hugging Face Tasks, Hub, and Transformers.js: the local/web angle The roadmap groups three Hugging Face pieces next to the local tools, and they round out the picture. **Hugging Face Tasks** is the catalog of things pre-trained models can do — classification, summarization, translation, visual question answering — each backed by many models on the Hub [4]. The **Hub** is the registry those models live on. And **Transformers.js** is the library that lets me run transformer models directly in the browser or Node.js using WebAssembly, with no cloud service required [5]. Transformers.js is the piece that genuinely surprised me. Running a model _in the browser_ means the user's device does the inference — full privacy, zero server cost, and offline capability. The constraint is the same as Ollama's, just sharper: browsers can only run small models well. For lightweight tasks — sentiment classification on a form field, simple classification in a UI — it's a genuinely viable architecture. For anything heavy, the server (or a cloud provider) still wins. ## How I use this I split my inference strategy by need. For development and any privacy-sensitive feature, I run Ollama locally and develop against localhost — it's free, private, and stable. For model shopping and comparison I use LM Studio's GUI to browse and benchmark before committing. For production features that need the breadth of the model market or resilience against a single provider, I put OpenRouter in front and treat the model choice as a config value, not a code change. And for tiny, privacy-first in-browser tasks, Transformers.js is now on my radar. The unifying discipline is never assuming "call the cloud API" is the only option — the right place for compute depends on privacy, cost, latency, and how locked-in I'm willing to be. ## References [1] Ollama, 2024. [Online]. Available: [https://ollama.com/](https://ollama.com/) [2] LM Studio, 2024. [Online]. Available: [https://lmstudio.ai/](https://lmstudio.ai/) [3] Codecademy, "What is OpenRouter? A Guide with Practical Examples," 2024. [Online]. Available: [https://www.codecademy.com/article/what-is-openrouter](https://www.codecademy.com/article/what-is-openrouter) [4] Hugging Face, "Task Summary," 2024. [Online]. Available: [https://huggingface.co/docs/transformers/v4.14.1/en/task_summary](https://huggingface.co/docs/transformers/v4.14.1/en/task_summary) [5] Hugging Face, "Transformers.js," 2024. [Online]. Available: [https://huggingface.co/docs/transformers.js/en/index](https://huggingface.co/docs/transformers.js/en/index) ```quiz Q: Ollama's main value proposition is… - it gives access to the largest closed models - it runs open-weight models locally on your machine, with no network and no per-token cost correct: 1 explain: Ollama packages open models for local inference. Privacy and predictable cost are the payoff; hardware capability is the constraint. Q: OpenRouter sits in front of many providers to give you… - a single model owned by one company - one API key and one request format that can route to many providers, enabling portability and fallback correct: 1 explain: OpenRouter is a federating proxy. Behind one contract you get the whole model market, which makes switching providers a config change. Q: Ollama and LM Studio differ mainly in… - which models they can run - ergonomics — Ollama is CLI/SDK-first, LM Studio is a GUI for browsing and comparing correct: 1 explain: Both run the same open weights on the same machine. Ollama suits production wiring and scripting; LM Studio suits experimentation and model shopping. Q: Transformers.js is notable because it… - replaces cloud providers for large models - runs transformer models in the browser via WebAssembly, giving privacy and offline use for small models correct: 1 explain: Browser inference works well for lightweight tasks on the user's device. Big models still need a server or cloud. Q: A latency-critical production path would likely… - always add OpenRouter for flexibility regardless of latency - weigh the extra proxy hop against the flexibility, and possibly call the provider directly correct: 1 explain: OpenRouter adds a hop. For most features that trade is worth it; for strict latency budgets, calling the provider directly can be the better call. ```