Ollama: Local Large Language Model Runner & Serving Daemon

What is Ollama?

Ollama is an open-source framework that packages model weights, configuration, and hardware acceleration into a single self-contained runtime. It abstracts the complexity of model loading, memory management, and cross-platform GPU driver detection, allowing developers to run models like Llama 3.3, DeepSeek-R1, Qwen 2.5, and Mistral with a single command.

Key Architectural Advantages

• Zero-Config Hardware Offload: Automatically detects Apple Silicon (Metal Unified Memory), NVIDIA (CUDA), and AMD (ROCm) hardware.
• Unified Modelfile System: Create reproducible model configurations combining base weights, custom system instructions, stop sequences, and runtime parameters.
• OpenAI-Compatible Endpoints: Built-in drop-in replacement for OpenAI API clients listening on http://localhost:11434/v1.
• Lightweight Memory Footprint: Unloads models from VRAM after a configurable idle timeout (default 5 minutes), freeing GPU resources for other tasks.

Installation & Setup

macOS & Windows

Download the native installer from ollama.com or install via package managers:

# macOS with Homebrew
brew install ollama

# Windows with Winget
winget install Ollama.Ollama

Linux Automated Installation

curl -fsSL https://ollama.com/install.sh | sh

Terminal Quickstart & Model Operations

1. Interactive Chat Session

To download and run a model interactively:

# General purpose 8B instruction model
ollama run llama3.1

# High-precision coding assistant
ollama run qwen2.5-coder:7b

# DeepSeek reasoning engine
ollama run deepseek-r1:8b

2. Modelfile Customization

Create a file named Modelfile to define custom behavioral parameters:

FROM llama3.1

# Set larger context window
PARAMETER num_ctx 16384

# Control generation creativity
PARAMETER temperature 0.2

# Custom system instructions
SYSTEM """
You are a senior systems engineer. You provide direct, concise terminal commands
and TypeScript code without conversational filler or disclaimers.
"""

Compile and run your custom persona:

ollama create dev-assistant -f ./Modelfile
ollama run dev-assistant

3. OpenAI SDK Integration

Ollama provides full compatibility with the OpenAI SDK. Point your base URL to http://localhost:11434/v1:

import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama', // Required by SDK client, ignored by Ollama
});

async function main() {
  const completion = await openai.chat.completions.create({
    model: 'llama3.1',
    messages: [
      { role: 'system', content: 'You are a code review agent.' },
      { role: 'user', content: 'Explain the benefits of TypeScript utility types.' },
    ],
    temperature: 0.3,
  });

  console.log(completion.choices[0].message.content);
}

main();