Installation on Windows

This page outlines the steps necessary to install and configure LLM-Distillery on a Windows system.

1. Clone the Repository

First, open Command Prompt (or PowerShell) and run:

git clone https://github.com/golololologol/LLM-Distillery
cd LLM-Distillery

2. Set Up the Virtual Environment

Inside the repository folder, run the batch script:

open_venv.bat

This command creates and activates a virtual environment automatically.

3. Install Prerequisites

Ensure you have the following installed on your system:

• Python 3.10+ (pipeline developed on 3.10.11)
• NVIDIA CUDA Toolkit 11.8+ (e.g., CUDA 11.8 or CUDA 12.1)

4. Install the Required Packages

Within your activated virtual environment, install these packages:

PyTorch

Install PyTorch 2.2.0+ (2.2.0 is recommended):

• PyTorch 2.2.0
• Latest version

Exllamav2

Install Exllamav2 0.0.19+ (0.2.7 is recommended). Choose the proper release based on your system configuration (CUDA Toolkit, PyTorch, Python, and Windows):

• Exllamav2 Releases

Flash Attention

For Flash Attention 2 2.4.2+ (2.5.2 is recommended):

• If your setup uses Python 3.10.x and torch 2.2.0, run:

  pip install https://github.com/bdashore3/flash-attention/releases/download/v2.5.2/flash_attn-2.5.2+cu122torch2.2.0cxx11abiFALSE-cp310-cp310-win_amd64.whl
  

• Otherwise, check this page for a wheel that matches your environment.

Final Step: Install Other Dependencies

Finally, install the remaining packages with:

pip install -r requirements.txt

Your Windows setup is now complete, and you’re ready to run LLM-Distillery.