Ollama in Docker ‐ isolated - slgrobotics/articubot_one GitHub Wiki

Here is how I ran Ollama on a Desktop machine

Contents:

The approach involves running Ollama and Open Web UI within composed Docker containers, without granting access to the host machine or networks. Furthermore, all data is intended to be stored in a single local directory, while allowing select folders to be shared between the host and containers.

The machine

I used my old-but-good machine for the experiment:

sudo lshw -short
bus            Asus P8Z68-V PRO (~2014)
processor      Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (4 core 8 threads, Sandy Bridge)
memory         16GiB System Memory  (1600 MHz)
display        GA104 [GeForce RTX 3060 Ti Lite Hash Rate]

nvidia-smi
| NVIDIA-SMI 535.288.01             Driver Version: 535.288.01   CUDA Version: 12.2     |
|   0  NVIDIA GeForce RTX 3060 Ti     Off | 00000000:01:00.0  On |                  N/A |
|  0%   43C    P5              24W / 200W |   1458MiB /  8192MiB |      0%      Default |

It is running Ubuntu 24.04 Desktop with the usual development tools installed.

Check NVIDIA Drivers: Open "Software & Updates" -> "Additional Drivers" and select the latest proprietary NVIDIA driver (e.g., version 550+).

The installation process

1. Install Docker

Follow this guide.

Install NVIDIA Container Toolkit to give Docker access to the GPU:

sudo apt-get install -y nvidia-container-toolkit

Note: docker commands should run under sudo, unless you do the following:

sudo groupadd docker
sudo usermod -aG docker $USER

At this point it is better to log out or reboot, or you will have to use sudo for all docker commands.

2. Create folders

mkdir -p ~/ollama/ollama_data
mkdir -p ~/ollama/ollama_share
mkdir -p ~/ollama/ollama_share_rw
mkdir -p ~/ollama/openwebui_data

3. Create ~/ollama/docker-compose.yaml

Important: replace "/home/sergei" with your home directory name

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    environment:
      - OLLAMA_HOST=0.0.0.0
    volumes:
      # A host folder mapped to Ollama work folder to preserve models and data between runs:
      - /home/sergei/ollama/ollama_data:/root/.ollama
      # Read-Only mount for sharing existing models or data
      - /home/sergei/ollama/ollama_share:/root/shared_data:ro
      # Read-Write mount for RAG (documents the AI can read, edit, delete)
      - /home/sergei/ollama/ollama_share_rw:/root/shared_data_rw
    networks:
      - private_ai_net
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    volumes:
      - /home/sergei/ollama/openwebui_data:/app/backend/data
      # Read-Only mount for RAG (documents the AI can read but not edit)
      - /home/sergei/ollama/ollama_share:/app/backend/data/shared_docs_readonly:ro
      # Read-Write mount for RAG (documents the AI can read, edit, delete)
      - /home/sergei/ollama/ollama_share_rw:/app/backend/data/shared_docs_rw
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    networks:
      - private_ai_net
    restart: unless-stopped

networks:
  private_ai_net:
    driver: bridge

4. Run it

cd ~/ollama
docker compose up -d

# ----- it will take a while - Docker images are pulled from the Web: ----
[+] up 26/26
 ✔ Image ghcr.io/open-webui/open-webui:main Pulled      205.8s
 ✔ Image ollama/ollama:latest               Pulled      150.5s
 ✔ Network ollama_private_ai_net            Created       0.0s
 ✔ Container ollama                         Created       2.7s
 ✔ Container open-webui                     Created       2.7s

The containers come up empty (no models loaded yet). Manually pull a model so there's something to see:

docker exec -it ollama ollama pull llama3.2

You can see that the ~/ollama/ollama_data folder is now filled with a ~2 GB model:

1972112	./ollama_data/models

The Open Web UI should be available in the browser on the host machine:

http://localhost:3000/

Create an account as requested. There is one thing you might have to change in the admin panel:

In your Open Web UI settings (Settings > Connections), change the local Ollama API URL to:

http://ollama:11434

You can verify that connection and also make sure that OpenAI API Connection fails to respond (due to isolation).

Go back to "New Chat" (a.k.a. http://localhost:3000) and select a model from the drop-down list.

Now that your local genie is fully functional, retrieve it from its bottle. Ask it something truly meaningful:

Note:

  • By default, Ollama unloads the model after 5 minutes of inactivity. The first prompt after a break will always be slow because it has to reload the entire model into VRAM. You can set a longer timeout in your ollama environment variables: OLLAMA_KEEP_ALIVE=24h
  • keep in mind the size of containers on the host machine's disk and downloaded models (cd ~/ollama; du):
docker image ls
                                                                                                                                                                                                                                                        IMAGE                                 ID             DISK USAGE   CONTENT SIZE   EXTRA
ghcr.io/open-webui/open-webui:main    2b9d2bec40ff       6.54GB         1.68GB    U   
ollama/ollama:latest                  63f052826604       9.01GB         3.35GB    U

Useful commands

Monitor GPU usage (Intel/AMD machines):

nvidia-smi
nvidia-smi dmon
nvidia-smi pmon

Stop all running containers:

docker stop $(docker ps -q)

Run the composed containers again:

cd ~/ollama
docker compose up -d   # using space in "docker compose", new form

Bring up shell:

docker exec -it ollama bash
docker exec -it open-webui bash

Display all the container's mounts:

docker inspect ollama --format '{{ json .Mounts }}' | jq

For more Docker commands see this guide.

Useful Links

Back to Home