Unlock LM Studio on Any AMD GPU with ROCm Guide - likelovewant/ROCmLibs-for-gfx1103-AMD780M-APU GitHub Wiki

Using AMD Graphics Cards with LM Studio

This guide helps Windows users get their AMD graphics cards working with LM Studio. Linux users shouldn't face these issues.

[!IMPORTANT] This guide is currently outdated due to breaking changes introduced in Ollama versions 0.4.0 and beyond. Here's how to proceed:

  • If you need to continue using this method: Refer to the backup script provided on each Ollama release page. This script allows you to build the libraries with the appropriate Llama.cpp commit for compatibility.
  • For testing purposes: You can use pre-built libraries from versions of Ollama before 0.4.0. Please note that this guide may not be reliable for current Ollama versions Or LMS

Why is this necessary? Officially supported ROCm drivers for LM Studio are limited, meaning many AMD graphics cards can only use the OpenCL version.

The solution: We can compile ROCm ourselves to support your specific AMD graphics card.

The steps maybe changes due to LM Studio update . However,the basic rule are same . ( make sure grab llama.dll and ggml.dll from ollama for amd builded with rocm5.7 ,update , from v0.3.4 ,lmstudio update the library to rocm6.1.2 ) ( test work untill to latest 0.3.5)

Step-by-Step Guide:

  1. Download and Install:

  2. Edit the Backend Manifest:

    • LM Studio versions before v0.2.25 and v0.2.28 (Maybe later): Open this file: C:\Users\Username\AppData\Local\LM-Studio\app-0.2.24\resources\app\.webpack\main\build\Release\llama_rocm\backend-C:\Users\Username\AppData\Local\LM-Studio\app-0.2.24\esources\app.webpack\main\build\Release\llama_rocm\backend-manifest.json`( change the path for different version)
    • LM Studio versions v0.2.25 or later: Open this file: C:\Users\Username\.cache\lm-studio\extensions\backends\win-llama-rocm-lm\backend-manifest.json
    • LM Studio versions v0.3.9 or later: Open this file: C:\Users\Username\.lmstudio\extensions\backends\llama.cpp-win-x86_64-amd-rocm-avx2-1.10.0(name may changes due to update)\backend-manifest.json
    • Add your graphics card model to the list. For example, if you have an AMD780M, add "gfx1103". You can find your card's model number online or in device manager.

  3. Replace llama.dll and ggml.dll or ggml-hip.dll（ Notes For v0.3.5 or later ,no need to replace the llama.dll ,only need to rename the ggml.dll toggml_llamacpp.dll replace the origianl one, while from v0.3.9 , may only need replace ggml-hip.dll ):

    • Find the llama.dll and ggml.dll file: It's located either in C:\Users\Username\AppData\Local\Programs\Ollama\ollama_runners\rocm_v5.7 or v6.1 (if you have supported Ollama files) or download it from https://github.com/likelovewant/ollama-for-amd/releases as ollama-windows-amd64.zip.
    • Important: For LM Studio v0.2.25 and later, use the llama.dll from an Ollama version later than v0.1.46. Earlier versions may be more stable.
    • Replace the existing llama.dll and ggml.dll file in the same folder as your backend-manifest.json with the downloaded one.
  • For version 0.3.9 or later : you may get a ggml-hip.dll by self-build llama.cpp with your rocm support arches in llama.cpp\build\bin replace the same libs in rocm extension.make sure git checkout to the same commit in the lmstuido release note , eg , git checkout 90d987b , then build llama.cpp .

Troubleshooting Ollama Llama.cpp with LM Studio

Here's how to fix common issues you might encounter when using Ollama llama.cpp with LM Studio:

"GGGGGGGGG" Output: This error usually means there's a compatibility problem between your Ollama version and the LM Studio version.

  • Try these solutions:
    • Use an older Ollama version (v0.1.48 or earlier).
    • If you're using LM Studio v0.2.28, enable Flash Attention.

Ollama Versions After v0.2:

  • File Placement: If you're using an Ollama version newer than v0.2, move the llama.dll and ggml.dll files into your LM Studio extensions folder or llama_rocm folder.

    • Important: This only works with LM Studio v0.2.28 or later (maybe). Make sure not to rename any other files!

  • Other problems: Check the LM Studio forums and GitHub for solutions specific to your graphics card model and LM Studio version.

This section provides advanced instructions for Windows users to get their AMD graphics cards working with LM Studio.

Important: If you're already using a compatible version of LM Studio, skip this section.

2 Rocm Support Step-by-Step Guide:

  1. Replace rocblas.dll and Library:

    • LM Studio versions before v0.2.25: Find the rocblas.dll and library files in this folder: C:\Users\Username\AppData\LM-Studio\app-0.2.24\resources\app\.webpack\main\build\Release\llama_rocm\bin
    • LM Studio versions v0.2.25 or later: Find the files in this folder: C:\Users\Username\.cache\lm-studio\extensions\backends\vendor\win-llama-rocm-vendor
    • LM Studio versions v0.3.9 or later: Find the files in this folder: C:\Users\Username\.lmstudio\extensions\backends\vendor\win-llama-rocm-vendor-v3 (name may changes due to update, currently need rocm6.1.2 rocmlibs)

    If you don't have them, download them from this repo or build your own rocblas.dll following the instructions on the wiki. Make sure to replace the existing files with the new ones for your GPU models.

  2. (Optional) Test CPU Support:

    • For LM Studio versions before v0.2.24 only: You can try replacing the llama.dll in this folder: C:\Users\Username\AppData\LM-Studio\app-0.2.24\resources\app\.webpack\main\build\Release\llama_cpu.
    • Download llama.dll from C:\Users\Username\AppData\Local\Programs\Ollama\ollama_runners\cpu_avx2 (if available) or download it online. You can also use the files from C:\Users\Username\AppData\Local\Programs\Ollama\ollama_runners\cpu_avx if your CPU uses the AVX series.

  3. Test ROCm:

    • Right-click within LM Studio and select "Switch to ROCm."
    • Load a model. If it loads successfully, you've set everything up correctly! Remember to close the program completely and restart it for changes to take effect.

Troubleshooting:

  • Still not working? Check the LM-Studio-0.2.24-full.nupkg file in your LM Studio installation directory (e.g., C:\Users\Username\AppData\Local\LM-Studio\packages).
  • Consult the LM Studio forums and GitHub for solutions specific to your graphics card model and LM Studio version.

The location would be in C:\Users\Username\.cache\lm-studio\extensions\backends\vendor\win-llama-rocm-vendor

This section provides advanced troubleshooting steps for using AMD GPUs with LM Studio versions before v0.2.25.

Caution: Modifying .nupkg files can potentially break your installation. Proceed with caution and back up your LM Studio folder before making changes.

###(Don't do this ,unless you had try the previous steps many times) Modifying the .nupkg File :

  1. Open the Package: Use VS Code (or any suitable program) to open the LM-Studio-0.2.24-full.nupkg file located in your LM Studio installation directory (e.g., C:\Users\Username\AppData\Local\LM-Studio\packages).

  2. Search for "gfx1100": Within the .nupkg file, search for the string "gfx1100".

  3. Replace with Your Model: Replace all instances of "gfx1100" with your specific AMD graphics card model (e.g., "gfx1103").

  4. Rezip the File: After making the changes, re-archive the .nupkg file using a suitable tool (e.g., 7-Zip).

Important Notes:

  • Available Models: This modification is based on the assumption that your graphics card model is listed in this set: gfx803; gfx900; gfx902; gfx940; gfx941; gfx942; gfx90c; gfx906; gfx1010; gfx1011; gfx1012; gfx1030; gfx1031; gfx1032; gfx1033; gfx1034; gfx1035; gfx1036; gfx1100; gfx1101; gfx1102; gfx1103

  • gfx803 Warning: Please note that "gfx803" is reported to be unstable.

  • Performance Expectations: Even with this modification, performance may be significantly slower than using Ollama. This method is primarily for experimental purposes and exploring GPU utilization in LM Studio.