Installation on Linux

This page provides step-by-step instructions to install and set up LLM-Distillery on a Linux system.

1. Clone the Repository

Open a terminal and run:

git clone https://github.com/golololologol/LLM-Distillery
cd LLM-Distillery

2. Set Up the Virtual Environment

Make the startup script executable and run it:

chmod +x open_venv.sh
./open_venv.sh

This script creates a virtual environment and opens a terminal with the activated environment.

3. Install Prerequisites

Ensure you have the following installed before proceeding:

• Python 3.10+ (pipeline developed on 3.10.11)
• NVIDIA CUDA Toolkit 11.8+ (e.g., CUDA 11.8 or CUDA 12.1)

4. Install the Required Packages

Within your activated virtual environment, manually install the following packages:

PyTorch

Install PyTorch 2.2.0+ (2.2.0 is recommended).

• PyTorch 2.2.0
• Latest version

Exllamav2

Install Exllamav2 0.0.19+ (0.2.7 is recommended). Choose the correct version based on your CUDA Toolkit, PyTorch, Python, and Linux distribution.

• Exllamav2 Releases

Flash Attention

For Flash Attention 2 2.4.2+ (2.5.2 is recommended):

• Visit the Flash Attention releases page and select the appropriate abiFALSE wheel for your system.

Then pip install the package you chose into the venv.

Final Step: Install Other Dependencies

Finally, install all remaining necessary packages by running:

pip install -r requirements.txt

Now your Linux environment is ready for running distillations with LLM-Distillery.