NVIDIA + CUDA + OLLAMA on Windows 11

On windows, manually set these to ENV variables.

OLLAMA_MODELS, when pulled using ollama, downloads all models in that location OLLAMA_NUM_GPU_LAYERS - when running ollama tries to offload as much work as possible to NVIDIA GPU (for that install CUDA for your GPU from here: https://developer.nvidia.com/cuda-downloads)

ollama serve

nvidia-smi

ChatGPT like UI for interacting models on local machine

https://github.com/open-webui/open-webui