LLMs Run Locally

For engineers evaluating where and how to use large language models, the practical landscape is split between remote inference providers and local execution, with frameworks bridging the two. The choice matters for latency, data control, cost, and security posture.

LLMs: A New Tool for Inference

LLMs can be used for inference with various providers [1]. That provider model has become the default entry point for software development teams: a client sends a prompt to a managed endpoint and receives completions.

Some LLMs can be run and used locally [1]. Local execution moves the model weights and inference onto hardware you control. The trade-off is operational responsibility versus data residency.

A minimal local inference sketch shows the flow:

# pseudo client for local vs remote provider selection
def infer(prompt, mode="local"):
  if mode == "local":
  # model runs on owned hardware
  return local_ollama.generate(prompt)
  else:
  # model runs via managed provider
  return remote_provider.generate(prompt)

The decision surface is not just technical. Market dynamics shape longevity.

Many firms will go bust when the AI bubble pops, but not all [2]. When the dot-com bubble burst, it killed pets.com, it killed Webvan… but it did not kill Amazon [2].

For developers, that means preferring composable patterns and open interfaces over vendor-specific lock-in.

Frameworks for Building Data Applications

There are various frameworks for building LLM-powered data applications [1]. Frameworks abstract provider differences and add data plumbing.

Framework	Languages	Typical Use
LangChain	JS/TS, Python	Use AI Gateway with LangChain for unified provider access [1]
Ollama	JS/TS	Run and use large language models locally [1]

Use AI Gateway with LangChain for unified provider access [1]. This kind of abstraction lets teams swap remote and local backends without rewriting application logic. The unified provider access pattern is relevant because LLMs can be used for inference with various providers [1], and some LLMs can be run and used locally [1].

LLMs are increasingly used as coding agents. Large language model coding agents increasingly perform open-ended data modeling and analysis [3]. That capability raises evaluation challenges.

The autonomous model discovery behavior of LLMs cannot be adequately characterized by a single benchmark run [3]. These agents are stochastic and adaptive, and therefore their autonomous model discovery behavior cannot be adequately characterized by a single benchmark run [3].

In this work, we propose an experimental design and analysis framework for systematically evaluating this discovery process, quantifying its variability, and identifying important factors [3]. Systematic evaluation is essential when agents generate code that touches production data.

Practical prompting advice also emerges from current practice. One of the consequences of this is that we should always consider asking the LLM the same question more than once, perhaps with some variation in the wording. Then we can compare answers, indeed perhaps ask the LLM to compare answers for us [2].

Security Implications

Security implications of LLM adoption are distinct from traditional AI systems. LLMs create a huge increase in the attack surface of software systems [2]. Simon Willison described the The Lethal Trifecta for AI agents: an agent that combines access to your private data, exposure to untrusted content, and a way to externally communicate (“exfiltration”) [2].

Local deployment changes the risk profile. With local models, untrusted content still reaches the model, but exfiltration paths are reduced if network egress is controlled. With remote providers, data leaves the perimeter for inference, creating new confidentiality considerations.

Key mitigations for developers include:

• Input validation and output sanitization before any tool call.
• Scoped credentials for agents; no broad API keys in prompts.
• Logging of prompts, completions, and tool invocations for auditability.
• Preferring local execution for sensitive data, while acknowledging operational cost.

When comparing deployment options, the trade-offs are clear.

Dimension	Local LLM	Remote Provider
Data residency	On-prem, full control	Data sent to third party
Latency	Hardware dependent	Network dependent
Model choice	Limited by local hardware	Broad catalog
Update cadence	Manual	Automatic

Local runtime is not a silver bullet. Local models still hallucinate and can be steered by adversarial prompts. Security remains about architecture, not just location.

Conclusion

LLMs are a new tool for inference that can be accessed via providers or run locally [1][1]. Frameworks such as LangChain provide unified access patterns for building data applications [1]. Agents built on LLMs perform open-ended data modeling and analysis [3], but their autonomous model discovery behavior cannot be adequately characterized by a single benchmark run [3], requiring systematic evaluation frameworks [3].

Security must be designed in from the start. LLMs create a huge increase in the attack surface of software systems [2], and teams should expect to ask the same question multiple times with variation to improve reliability [2].

For software developers, the pragmatic path is hybrid: use local models where data sensitivity and control dominate, use remote providers where scale and model breadth dominate, and keep orchestration portable.

References

1. Helicone/helicone — github.com (31 August 2026)
2. Some thoughts on LLMs and Software Development — martinfowler.com (28 August 2025)
3. An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery — arxiv.org (7 July 2026)