Ollama on Jetson Nano - slgrobotics/articubot_one GitHub Wiki

Note: Trying to run anything modern on a Jetson Nano is an exercise in futility, akin to reviving a PDP-11.

See "Core Technical Barriers" at the end.

You've been warned! ;-)

Here is how I tried to run Ollama on a Jetson Nano (not using GPU)

Contents:

The goal was to run Ollama and Open Web UI, either with or without Docker Compose containers. Isolation would be nice, but it is not a priority at the moment.

The machine

I have an official Jetson Nano DevKit V2 (with two camera connectors), which I used for this experiment.

sudo lshw -short
H/W path  Device  Class          Description
-----------------------------------------------------------------
                  system         NVIDIA Jetson Nano Developer Kit
/0/0              processor      cpu  ( quad-core Arm Cortex-A57 1.43GHz )
/0/8              memory         3956MiB System memory  (4GB)
/0/2/0    eth0    network        RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
/0/1/0    wlan0   network        Wireless 8265 / 8275 - external M.2 card

In object detection tests (YOLOv5), the Nano can achieve 30+ FPS on 1080p streams.

Note: You need a physical Ethernet connection to your LAN. Wi-Fi is too slow for downloading large Docker images and models.

The OS installation process

NVIDIA is notorious for dropping support for older products. As a result, the Jetson Nano can only run Ubuntu 18.04 (a.k.a. Jetson Linux 32.7.6) and supports JetPack 4.6.1 at best โ€” the latest available for this hardware.

NVIDIA SDK Manager no longer offers an image for the Nano, but the most recent SD card image is available here:

https://developer.nvidia.com/embedded/l4t/r32_release_v7.1/jp_4.6.1_b110_sd_card/jeston_nano/jetson-nano-jp461-sd-card-image.zip

The download size is 6.6 GB; the extracted image is 13.8 GB. I used the Ubuntu Disks utility on my development machine to create a bootable SD card.

Tip: Use a "high endurance" SD card like this: https://www.amazon.com/dp/B07P3D6Y5B - it helps when swapping is enabled.

I used a 64 GB card since Ollama models are quite large (currently I see about 69% utilized), so a 32 GB card would not be sufficient for some experiments.

After the initial boot, I named the machine "jetson" and created a user account named "jetson". Choose "LXDE" Desktop on your first login - it is the lightest on resources.

Then you need to do the following:

1. Do the usual:

sudo apt update
sudo apt upgrade   # a lot of stuff will become a bit more recent.

I answered "Y" to all notifications about the configuration files updated by the vendor.

2. Optional: Reassign swap to SD card to free some RAM:

Jetson Nanoโ€™s default swap configuration uses ZRAM, not a traditional swap file or disk partition. This avoids thrashing the SD card. "zram" is a compressed RAM-based swap device. Instead of writing swapped memory pages to disk, the kernel:

  • Compresses the memory page
  • Stores it in RAM inside a special block device

First, check swap status:

jetson@jetson:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G        1.0G        1.8G         37M        1.1G        2.7G
Swap:          1.9G          0B        1.9G

jetson@jetson:~$ swapon --show
NAME       TYPE        SIZE USED PRIO
/dev/zram0 partition 494.5M   0B    5
/dev/zram1 partition 494.5M   0B    5
/dev/zram2 partition 494.5M   0B    5
/dev/zram3 partition 494.5M   0B    5

jetson@jetson:~$ cat /proc/swaps

jetson@jetson:~$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache     si   so    bi    bo   in   cs us sy id wa st
 0  0      0 1892784  42100 1062992    0    0   119    46  241  266  3  4 92  1  0
 0  0      0 1892644  42100 1063020    0    0     0     0  533  448  1  2 97  0  0
Note: si = swap in   so = swap out

jetson@jetson:~$ cat /proc/sys/vm/swappiness
60  # Swappiness controls how aggressively Linux uses swap. "10" is better for embedded systems

If changing swap:

sudo systemctl disable nvzramconfig
sudo swapoff -a
sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
---
sudo vi /etc/fstab  # - add the following line at the end:
/swapfile none swap sw 0 0
---

Changing swappiness - edit /etc/sysctl.conf:

vm.swappiness=10

3. Check if you need to expand file system to use full SD card space.

If df -k shows less than you expect, run the following:

sudo /usr/lib/nvidia/resizefs/nvresizefs.sh

4. Add yourself to "docker" group:

# sudo groupadd docker   # - this should already exist, docker is part of the image
sudo usermod -aG docker $USER

-- It is a good time to reboot now. --

The Nano boots OK; you should be able to log in remotely: ssh [email protected]

jetson@jetson:~/$ uname -a
Linux jetson 4.9.337-tegra #1 SMP PREEMPT Mon Nov 4 23:41:41 PST 2024 aarch64 aarch64 aarch64 GNU/Linux

jetson@jetson:~$ docker -v
Docker version 20.10.21, build 20.10.21-0ubuntu1~18.04.3

Free some space:

sudo apt autoremove

5. Install some updates and utilities:

sudo apt install htop python3-pip nvidia-cuda
pip3 install --upgrade pip
sudo -H pip3 install --upgrade setuptools
sudo -H pip3 install docker-compose
sudo -H pip3 install -U jetson-stats  # creates "jtop" group and adds you to it

# Check the monitoring tools:
htop
sudo tegrastats
sudo jtop         # after logout and login no need for "sudo"

The jtop is very useful when you want to see GPU utilization; 'tegrastats' is the most precise.

GUI Desktop vs. SSH - GPU utilization monitoring on Nano

The Jetson Nano normally boots into full GUI (desktop) mode. At the password screen, you can click the gear icon and select the desktop environment โ€” Unity (default), LXDE, Openbox, or Ubuntu. I suggest LXDE since it is less demanding on system resources.

You can work in GUI mode until you start monitoring the GPU (for example, if it is being used by Ollama or the GUI itself).

At that point, you may want to disable the desktop and work via SSH instead. The AI browser will still be available at http://jetson.local:8080 from other machines on your LAN.

sudo init 3  # Desktop off
sudo init 5  # Desktop on (you might need to select "LXDE" by clicking on the gear icon near the password entry field)

# Permanently switch Desktop off on reboots, "init 5" will still work:
sudo systemctl set-default multi-user.target   # boot without GUI (to "init 3" opposed to "init 5")
# If needed, you can reverse that:
sudo systemctl set-default graphical.target

Note: this completes the OS installation and configuration. Experiments with Ollama below. For ROS2 TCP/IP Server see this guide.

The Ollama installation process - both containers

1. Create folders

mkdir -p ~/ollama/ollama_data
mkdir -p ~/ollama/ollama_share
mkdir -p ~/ollama/ollama_share_rw
mkdir -p ~/ollama/openwebui_data

2. Create ~/ollama/docker-compose.yaml

Note the image name and the port 8080 mapping (open for LAN access)

services:
  ollama:
    # Default image is not compiled with support for NVIDIA's JetPack/L4T environment used by the Jetson Nano:
    image: ollama/ollama:latest
    # Community images at Docker Hub may or may not work, no exact match found:
    # See https://hub.docker.com/r/dustynv/ollama
    #image: dustynv/ollama:r32.7.1
    container_name: ollama
    runtime: nvidia
    environment:
      - OLLAMA_HOST=0.0.0.0
    volumes:
      # A host folder mapped to Ollama work folder to preserve models and data between runs: 
      - /home/jetson/ollama/ollama_data:/root/.ollama
      # Read-Only mount for sharing existing models or data:
      - /home/jetson/ollama/ollama_share:/root/shared_data:ro
      # Read-Write mount for RAG (documents the AI can read, edit, delete):
      - /home/jetson/ollama/ollama_share_rw:/root/shared_data_rw
    ports:
      - "11434:11434"
    networks:
      - private_ai_net
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    volumes:
      - /home/jetson/ollama/openwebui_data:/app/backend/data
      # Read-Only mount for RAG (documents the AI can read but not edit)
      - /home/jetson/ollama/ollama_share:/app/backend/data/shared_docs_readonly:ro
      # Read-Write mount for RAG (documents the AI can read, edit, delete)
      - /home/jetson/ollama/ollama_share_rw:/app/backend/data/shared_docs_rw
    ports:
      #- "3000:8080"
      - "8080:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    networks:
      - private_ai_net
    restart: unless-stopped

networks:
  private_ai_net:
    driver: bridge

4. Run it

cd ~/ollama
docker-compose up -d  # note the dash in "docker-compose", that's the old form

# ----- it will take a while the first time - Docker images are pulled from the Web: ----
docker image ls
REPOSITORY                      TAG       IMAGE ID       CREATED      SIZE
ghcr.io/open-webui/open-webui   main      dfab7cc0dc18   2 days ago   4.15GB
ollama/ollama                   latest    a9129bcdf8ea   2 days ago   5.54GB

The containers come up empty (no models loaded yet). Manually pull a model so there's something to see (the lighter qwen2:0.5b is good for now):

docker exec -it ollama ollama pull qwen2:0.5b
# docker exec -it ollama ollama pull llama3.2

You can see that the ~/ollama/ollama_data folder is now filled with a model file. It will persist between Docker runs.

The Open Web UI should be available in the browser on the host machine (or other machines on your LAN):

http://localhost:8080/ or http://jetson.local:8080

Create an account as requested. Check one thing in the admin panel:

In your Open Web UI settings (Settings > Connections), the local Ollama API URL should be: http://ollama:11434

You can verify that connection and also make sure that OpenAI API Connection fails to respond (due to isolation).

Go back to "New Chat" (a.k.a. http://localhost:3000) and select a model from the drop-down list.

Now that your local genie is fully functional, retrieve it from its bottle. Ask it something truly meaningful:

(please, don't follow this recipe. Don't ask me how I know ;-))

Note:

  • GPU use will be close to zero, while CPUs will be maxed out. No surprise, we used a non-GPU generic image.
  • By default, Ollama unloads the model after 5 minutes of inactivity. The first prompt after a break will always be slow because it has to reload the entire model into VRAM. You can set a longer timeout in your ollama environment variables: OLLAMA_KEEP_ALIVE=24h
  • keep in mind the size of containers on the host machine's disk (docker image ls) and downloaded models (cd ~/ollama; du)

Useful commands

Monitor GPU usage:

jtop
sudo tegrastats

Run the composed containers again (note the dash in "docker-compose", that's the old form):

cd ~/ollama
docker-compose up -d

For more Docker commands see this guide.

Useful Links

Core Technical Barriers

Why using anything modern with GPU support is practically impossible on Jetson Nano.

Hardware (Maxwell Architecture): The Jetson Nano uses the Maxwell GPU architecture, which is officially capped at CUDA 10.2. Modern AI frameworks and libraries (like PyTorch and newer versions of CUDA) now standardly require CUDA 11 or 12 to access optimized kernels and newer data types like bfloat16.

Software (Ubuntu 18.04): The device is tethered to JetPack 4, which relies on Ubuntu 18.04 LTS. This version reached its end-of-standard-support in 2023, leaving users stuck with:

  • Python 3.6.9: Most modern AI libraries (Transformers, PyTorch) now require at least Python 3.8+.
  • GCC 7.5.0: This outdated C++ compiler cannot build the latest high-performance C++ codebases used in modern inference engines.

Dependencies (llama.cpp & Ollama): While "hacks" exist to compile older versions of llama.cpp using manual GCC upgrades (e.g., to version 8.5), these versions lack the optimizations and compatibility needed for current models. Standard Ollama distributions often fail to recognize the Nano's GPU entirely due to the legacy CUDA driver.

Incompatibility (Llama 3.2+): New models like Llama 3.2 utilize architectural features and quantization formats (like GGUF versions or specific layouts) that legacy inference engines cannot interpret. Even if loaded, the Nano's 4GB LPDDR4 RAM is shared between the system and GPU, leaving insufficient memory for even the smallest modern quantized models to run at a usable speed.

See this guide for details.

Back to Home