AI Development Run LLMs Locally - spinningideas/resources GitHub Wiki

Recipe: Run LLMs Locally for Software Development

This guide covers running Machine Learning (ML) and Large Language Models (LLMs) locally as of July 2026, with a focus on coding, autonomous agentic software development, and IDE integration (VS Code, OpenCode, and Aider) on consumer hardware with a 16GB VRAM NVIDIA GPU and 64GB system RAM.

For specific model recommendations, see AI-Local-Development-Models.

Quick Decision Guide

Path Recommended setup Choose this when
Best Ollama or Unsloth + OpenCode or Cline + a separate autocomplete model You want the strongest agentic coding workflow, repeatable configuration, and the best use of a 16GB GPU.
Better LM Studio + VS Code Continue or Cline You prefer a desktop interface, visual model discovery, and simple hardware offload controls.
Good Jan AI or Ollama + the official VS Code Ollama extension, or Aider from the CLI You want a straightforward local setup with minimal configuration.

The Best/Better/Good labels are practical setup recommendations, not universal model-quality rankings. The best choice depends on operating system, GPU backend, model format, context length, and whether the workload is chat, coding, autocomplete, or autonomous tool use.

Hardware Criteria: 16GB GPU + 64GB RAM

Windows

A machine with a 16GB VRAM NVIDIA GPU (for example, an RTX 4080, RTX 5080, RTX 4070 Ti Super, or RTX 3090) and 64GB system RAM is somewhat workable workstation tier for local AI coding. The important criteria are:

  • VRAM capacity: Determines how much of the model and KV cache can remain on the GPU. More VRAM generally improves latency and context capacity.
  • System RAM: 64GB provides room for model files, CPU layers, KV cache, operating-system overhead, and quantized offload when a model does not fit entirely in VRAM.
  • GPU acceleration: NVIDIA CUDA support provides the broadest compatibility with local inference tools. Other GPU backends can work, but compatibility and performance vary by runtime.
  • Storage: Use fast SSD storage for model files and leave enough space for multiple quantizations. Models can consume many gigabytes each.
  • Thermals and power: Sustained inference is a long-running workload. Adequate cooling and power delivery help maintain consistent performance.
  • Context budget: Larger context windows consume additional VRAM through the KV cache. Start conservatively and increase the context only after confirming that the model remains fully usable.

Mac

The closest Mac equivalent is an Apple Silicon Mac with 64GB of unified memory, such as a Mac Studio or MacBook Pro with an M4 Max, 16-core CPU, 40-core GPU, and 64GB unified memory. Apple’s unified memory is shared by the CPU and GPU, so it does not map directly to a separate 16GB VRAM plus 64GB system RAM configuration. See Apple’s Mac Studio technical specifications and MLX for Apple Silicon.

This is a somewhat workable equivalent for local LLM development, especially when using MLX, MLX-LM, Ollama, or LM Studio with Metal acceleration. A 64GB unified-memory Mac can run larger quantized models than a 16GB discrete GPU, but it generally generates tokens more slowly than a comparable NVIDIA CUDA workstation because Apple Silicon has lower memory bandwidth and does not support CUDA-specific runtimes.

For a Mac local-AI setup, prioritize:

  • 64GB unified memory: Treat this as the main equivalent to the Windows machine’s combined GPU and system memory. Leave headroom for macOS, the IDE, and the KV cache; do not assume all 64GB is available to the model.
  • Max-class Apple Silicon: Prefer an M4 Max or newer Max-class chip with the highest available GPU core count and memory bandwidth. Ultra-class chips are preferable when running multiple models or larger models is more important than cost.
  • Fast internal SSD: Use at least 1TB when possible because quantized model files can consume many gigabytes.
  • Native Apple Silicon software: Prefer MLX/MLX-LM for maximum Apple hardware efficiency, or Ollama and LM Studio for easier model management and OpenAI-compatible endpoints.
  • Thermals and sustained power: A Mac Studio is generally better for long-running inference than a fanless MacBook Air. A MacBook Pro is useful when portability matters.

What Fits in This Hardware Tier

Memory allocation and performance vary by model architecture and quantization:

  • Fully in-VRAM models (8B–16B at Q4/Q8): Models such as Qwen 2.5/3.0 Coder 14B, DeepSeek-Coder-V2 Lite 16B MoE, Qwythos-9B, Gemma 4 12B, and LFM 2.5-8B occupy approximately 6–10GB of VRAM. This can leave 6GB or more for the KV cache, enabling 32k–64k context windows and fast generation (>60–120+ tokens/sec) for autocomplete and refactoring.
  • Dense 22B–24B models: Devstral Small 2 (24B) uses approximately 14.5GB at Q4_K_M and Codestral 22B uses approximately 12.5GB at Q4_K_M. These use most of the 16GB VRAM, so keep context limits conservative (2k–8k tokens) to avoid spilling into system memory.
  • MoE and quantized-offload models (27B–33B total parameters): Models such as Laguna XS 2.1 (33B total / 3B active) and Qwen 3.6 27B A3B Coder execute only approximately 2.4B–3B active parameters per token. They can keep active weights in VRAM while using 64GB system RAM for layer offloading and larger context buffers without catastrophic speed drops.

Recommended Dual-Model Strategy

Use different models for different latency and reasoning requirements:

  • Agent/chat model: Use Laguna XS 2.1 (33B MoE) or Devstral Small 2 (24B) for architectural planning, issue resolution, and multi-file changes.
  • Inline autocomplete model (FIM): Use a fast 1.5B–8B model, such as LFM 2.5 8B or Qwen 2.5-Coder 1.5B, exclusively for tab autocomplete and sub-50ms latency.

Step 1: Choose a Local Runtime

Choose one runtime or inference engine before configuring your editor. All of the primary options below can expose a local or OpenAI-compatible endpoint.

Best: Full-Control and Agent-Ready Options

  • Ollama - De facto CLI and background engine for local LLMs; provides a native REST API and OpenAI-compatible server at http://localhost:11434/v1.
  • Unsloth Studio & Engine (GitHub) - Advanced local AI environment with dynamic 4-bit quantization, hardware acceleration, and the unsloth start [agent] CLI for connecting local LLMs directly to coding agents.
  • llama.cpp Server - Raw C++ inference engine (llama-server) for maximum hardware control, custom layer offloading, and low memory overhead.
  • vLLM - High-throughput and memory-efficient inference and serving engine for LLMs.
  • OpenWebUI (GitHub) - Feature-rich web interface with deep Ollama integration, multi-model chat, RAG, and web search.

Better: Desktop and Flexible Developer Options

  • LM Studio - Desktop interface and server at http://localhost:1234/v1 for GGUF model discovery, visual context monitoring, and hardware offload configuration.
  • Jan AI (GitHub) - Fully offline, open-source Tauri desktop ChatGPT alternative with a local OpenAI-compatible API server at http://localhost:1337/v1 and Model Context Protocol (MCP) support.
  • Text-Generation-WebUI - Flexible engine supporting GGUF, EXL2, Hugging Face Transformers, and extensive plugin ecosystems.
  • LocalAI - Developer-focused drop-in OpenAI API replacement for text, audio, and vision.
  • AnythingLLM / UseAnything - All-in-one desktop application with document RAG and local model support.
  • GPT4All - Cross-platform local LLM application with LocalDocs RAG.
  • MindMac - Native desktop application supporting local endpoints from Ollama, LM Studio, and Jan.
  • TypingMind - Advanced web interface for local OpenAI-compatible endpoints.
  • NVIDIA ChatRTX - Hardware-accelerated local RAG for RTX GPUs.
  • LocalGPT - Document chat powered by local models.
  • Streamlit and Gradio - Prototyping frameworks for custom AI applications.
  • Secret Llama - Browser-based private local LLM chatbot.
  • LibreChat - Multi-model chat interface supporting custom local backend endpoints and agent plugins.

Good: Lightweight and Specialized Options

  • KoboldCpp - Lightweight single-file GGUF execution server.
  • LobeChat - Modern chat UI with extension and MCP support.
  • Lollms-WebUI - Multimodal web UI with PDF and web search capabilities.
  • H2O GPT - Document ingestion and query framework.
  • SillyTavern - Persona and character interaction interface.
  • ChainFury - Prompt chaining and web search framework.
  • PrivateGPT - Offline document RAG stack.
  • ExUI - VRAM-efficient GUI for EXL2 models.
  • AI-Messenger - Desktop interface supporting EXLv2 and LLaVA.

Mac-Only Options

  • FreeChat - Native macOS chat application.
  • Sanctum - Private local AI workspace for macOS.

Step 2: Choose a Local Coding Model

Information about specific models is maintained on AI-Local-Development-Models. Match the model to the role described in the hardware section rather than choosing one model for every task.

  • Use a larger agent/chat model for planning, tool use, issue resolution, and multi-file refactoring.
  • Use a smaller FIM model for inline completion and low-latency suggestions.
  • Use a quantized or MoE model when you need more capability than fits fully in 16GB VRAM.

Step 3: Connect the Runtime to Your Development Tools

Connect local models running in Ollama, LM Studio, Jan AI, or Unsloth to developer tools through their local or OpenAI-compatible REST endpoints.

OpenCode Setup

OpenCode connects to local models via OpenAI-compatible REST endpoints.

Option A: Automatic Launch via Unsloth CLI

If using Unsloth, launch OpenCode with pre-configured endpoints in a single command:

unsloth start opencode

Option B: Ollama and Manual Configuration

  1. Set the Ollama context window:

    cat > Modelfile <<EOF
FROM laguna-xs-2.1:q4_K_M
PARAMETER num_ctx 32768
EOF
ollama create my-laguna-32k -f Modelfile

  2. Configure the OpenCode config file (~/.config/opencode/opencode.json):

    {
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama Local",
      "options": {
        "baseURL": "http://localhost:11434/v1"
      },
      "models": {
        "my-laguna-32k": { "name": "Laguna XS 2.1 (33B MoE 32k)" },
        "devstral-small-2": { "name": "Devstral Small 2 (24B)" },
        "mannix/qwen3.6-27b-a3b-coder:CD-IQ4_K_M": {
          "name": "Qwen 3.6 27B Coder"
        }
      }
    },
    "lmstudio": {
      "name": "LM Studio Local",
      "options": {
        "baseURL": "http://localhost:1234/v1"
      },
      "models": {
        "local-model": { "name": "LM Studio Model" }
      }
    }
  }
}

  3. Launch OpenCode:

    opencode
# Or launch directly with Ollama:
ollama launch opencode --model devstral-small-2

VS Code Integration

Local models can be connected to VS Code through native features or popular extensions.

Method A: VS Code Native Copilot BYOK (Bring Your Own Key/Model)

VS Code supports native local model connections without cloud subscriptions:

  1. Open VS Code Settings (Ctrl+, / Cmd+,).
  2. Navigate to GitHub Copilot / Language Models.
  3. Set the custom model provider base URL to http://localhost:11434/v1 (Ollama) or http://localhost:1234/v1 (LM Studio).
  4. Select your local model directly inside the VS Code Copilot Chat panel.

Method B: Cline Extension (Recommended Agentic Tool)

(Note: Roo Code was archived in May 2026; Cline is the active community standard.)

  1. Install Cline from the VS Code Marketplace.
  2. Open Cline settings (gear icon):
    • API Provider: Select Ollama or OpenAI Compatible.
    • Base URL: Set http://localhost:11434 (Ollama), http://localhost:1234/v1 (LM Studio), or http://localhost:1337/v1 (Jan AI).
    • Model ID: Enter laguna-xs-2.1:q4_K_M or devstral-small-2.
    • Enable Use Compact Prompt to preserve context budget.

Method C: Continue Extension

  1. Install Continue from the VS Code Marketplace.

  2. Configure ~/.continue/config.yaml (or config.json):

    schema: v1
models:
  - name: Laguna XS 2.1 (Ollama)
    provider: ollama
    model: laguna-xs-2.1:q4_K_M
    apiBase: http://localhost:11434
    roles: [chat, edit, apply]
  - name: Devstral Small 2 (Ollama)
    provider: ollama
    model: devstral-small-2
    apiBase: http://localhost:11434
    roles: [chat, edit]
  - name: LM Studio Local Server
    provider: openai
    model: local-model
    apiBase: http://localhost:1234/v1
tabAutocompleteModel:
  name: LFM 2.5 Fast Autocomplete
  provider: ollama
  model: lfm2.5:8b-a1b-q8_0
  apiBase: http://localhost:11434

Method D: Official Microsoft / Ollama Extension

Install the official Ollama Extension from the VS Code Marketplace for direct one-click sidebar integration.

Aider CLI Pair Programmer Setup

Aider is a git-native CLI pair programmer that auto-commits code edits:

  1. Install Aider:

    uv tool install aider-chat

  2. Configure the local endpoint:

    # Linux / macOS
export OLLAMA_API_BASE=http://127.0.0.1:11434

# Windows PowerShell
$env:OLLAMA_API_BASE="http://127.0.0.1:11434"

  3. Run Aider with the Ollama chat prefix:

    aider --model ollama_chat/devstral-small-2

Goat

  • local-llm - Amazing guide to running SOTA LLMs locally.