Local LLM with Ollama - up1/workshop-ai-with-technical-team GitHub Wiki

Local LLM with Ollama

Install Ollama

Download model

$ollama run llama3.2:1b

List of models

$ollama list
NAME           	ID          	SIZE  	MODIFIED    
llama3.2:1b    baf6a787fdff    1.3 GB    12 minutes ago

Access to Ollama API

Configuration

$export SYSTEMD_EDITOR=vim
$systemctl edit ollama.service

Edit

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Reload ans restart

$systemctl daemon-reload
$systemctl restart ollama

$ollama serve

$ollama ps
NAME           	ID          	SIZE  	PROCESSOR	UNTIL              
llama3.1:latest	91ab477bec9d	6.7 GB	100% GPU 	4 minutes from now

Stop Ollama server

$pgrep ollama
$kill -9 <pid>

Start LiteLLM and Open Web UI with Docker compose

File docker-compose.yml

services:
  webui:
    image: ghcr.io/open-webui/open-webui:main
    restart: unless-stopped
    ports:
      - '8080:8080'
    environment:
      - OPENAI_API_KEY=dummy
      - OPENAI_API_BASE_URL=http://litellm:4000/v1
    volumes:
      - open-webui:/app/backend/data
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    restart: unless-stopped
    command:
      - '--config=/litellm_config.yaml'
      - '--detailed_debug'
    ports:
      - '127.0.0.1:33372:4000'
    environment:
      #- LITELLM_MASTER_KEY=dummy
      - OPENAI_API_KEY
      - GEMINI_API_KEY
      - ANTHROPIC_API_KEY
    volumes:
      - ./litellm_config.yaml:/litellm_config.yaml
volumes:
  open-webui:

Config for LiteLLM with models

  • litellm_config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20240620
  - model_name: gemini-1.5-pro-latest
    litellm_params:
      model: gemini/gemini-1.5-pro-latest
  - model_name: "ollama3.1"
    litellm_params:
      model: "ollama/llama3.1"
      api_base: "http://159.223.53.44:11434"

Run

$docker compose up -d

$docker compose ps
NAME               IMAGE                                 COMMAND                  SERVICE   CREATED          STATUS                    PORTS
ollama-litellm-1   ghcr.io/berriai/litellm:main-latest   "litellm --config=/l…"   litellm   45 minutes ago   Up 37 minutes             127.0.0.1:33372->4000/tcp
ollama-webui-1     ghcr.io/open-webui/open-webui:main    "bash start.sh"          webui     45 minutes ago   Up 37 minutes (healthy)   127.0.0.1:33371->8080/tcp

Access to Open web ui

Write code

Example with Python

$pip install ollama

Hello ollama

import ollama

# Create server
server = ollama.Client(host='http://128.199.88.214:11434')

# Send question
response = server.chat(
    model='llama3.2:1b', 
    messages=[
  {
    'role': 'user',
    'content': 'Why is the sky blue?',
  },
])
print(response['message']['content'])
⚠️ **GitHub.com Fallback** ⚠️