Running AI Locally

Running models locally means no API costs, no data leaving your machine, and no rate limits. The recipe: pick a quantized GGUF model sized for your RAM/VRAM, and run it with llama.cpp, Ollama, LM Studio, or text-generation-webui.

Rough sizing: 7–8B models run on 8–16 GB RAM; 70B models need ~40 GB at 4-bit quantization (high-end GPUs or Apple Silicon with unified memory). Speed is measured in tokens/second — 10+ t/s feels interactive.

Beyond chat: local embeddings power private RAG, and local image models (Stable Diffusion, Flux) run well on consumer GPUs. Start small, verify quality on your tasks, then scale up.