💫 Huawei Ascend（华为昇腾）ascend3.5

Based on Triton 3.5, aarch64
Available for 910B, 910C

1. Quick start

1.1 Use the image (910B|910C)

Tips: You can use apt install aria2; aria2c -c ... instead of wget ... to download large files with resume support.

# cann9.0.0, python3.11

## 910B
### 910B Plan A: docker pull (13.3GB)
IMAGE=harbor.baai.ac.cn/flagtree/flagtree-ascend3.5-910b-py311-cann9.0.0-ubuntu22.04-aarch64:202606-torch2.9.0-base
docker pull ${IMAGE}
### 910B Plan B: docker load (4.8GB)
IMAGE=flagtree-ascend3.5-910b-py311-cann9.0.0-ubuntu22.04-aarch64:202606-torch2.9.0-base
wget https://baai-cp-web.ks3-cn-beijing.ksyuncs.com/trans/flagtree-ascend3.5-910b-py311-cann9.0.0-ubuntu22.04-aarch64.202606-torch2.9.0-base.tar.gz
docker load -i flagtree-ascend3.5-910b-py311-cann9.0.0-ubuntu22.04-aarch64.202606-torch2.9.0-base.tar.gz

## 910C
### 910C Plan A: docker pull (19.2GB)
IMAGE=harbor.baai.ac.cn/flagtree/flagtree-ascend3.5-910c-py311-cann9.0.0-ubuntu22.04-aarch64:202608-torch2.10.0-vllm0.20.2
docker pull ${IMAGE}
### 910C Plan B: docker load (6.6GB)
IMAGE=flagtree-ascend3.5-910c-py311-cann9.0.0-ubuntu22.04-aarch64:202608-torch2.10.0-vllm0.20.2
wget https://baai-cp-web.ks3-cn-beijing.ksyuncs.com/trans/flagtree-ascend3.5-910c-py311-cann9.0.0-ubuntu22.04-aarch64.202608-torch2.10.0-vllm0.20.2.tar.gz
docker load -i flagtree-ascend3.5-910c-py311-cann9.0.0-ubuntu22.04-aarch64.202608-torch2.10.0-vllm0.20.2.tar.gz

# cann9.1.0, python3.11

## 910B
IMAGE=harbor.baai.ac.cn/flagtree/flagtree-ascend-910b-py311-cann9.1.0-ubuntu22.04-aarch64:202610-3.5-torch2.12.0-shmem-base
### 910B Plan A: docker pull (15.2GB)
docker pull ${IMAGE}
### 910B Plan B: docker load (5.2GB)
wget https://baai-cp-web.ks3-cn-beijing.ksyuncs.com/trans/flagtree-ascend-910b-py311-cann9.1.0-ubuntu22.04-aarch64.202610-3.5-torch2.12.0-shmem-base.tar.gz
docker load -i flagtree-ascend-910b-py311-cann9.1.0-ubuntu22.04-aarch64.202610-3.5-torch2.12.0-shmem-base.tar.gz

## 910C
IMAGE=harbor.baai.ac.cn/flagtree/flagtree-ascend-910c-py311-cann9.1.0-ubuntu22.04-aarch64:202610-3.5-torch2.12.0-shmem-base
### 910C Plan A: docker pull (15.1GB)
docker pull ${IMAGE}
### 910C Plan B: docker load (5.2GB)
wget https://baai-cp-web.ks3-cn-beijing.ksyuncs.com/trans/flagtree-ascend-910c-py311-cann9.1.0-ubuntu22.04-aarch64.202610-3.5-torch2.12.0-shmem-base.tar.gz
docker load -i flagtree-ascend-910c-py311-cann9.1.0-ubuntu22.04-aarch64.202610-3.5-torch2.12.0-shmem-base.tar.gz

CONTAINER=flagtree-dev-xxx
docker run -dit -u 0 --user=root \
    --network=host --pid=host --ipc=host --privileged \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/sbin/:/usr/local/sbin/ \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    --device=/dev/davinci0 --device=/dev/davinci1 \
    --device=/dev/davinci2 --device=/dev/davinci3 \
    --device=/dev/davinci4 --device=/dev/davinci5 \
    --device=/dev/davinci6 --device=/dev/davinci7 \
    --device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc \
    -v /etc/localtime:/etc/localtime:ro \
    -v /data:/data -v /home:/home \
    -w /root --name ${CONTAINER} ${IMAGE} bash
docker exec -it ${CONTAINER} /bin/bash

1.2 Source-free Installation

# Note: First install PyTorch, then execute the following commands
python3 -m pip uninstall -y triton  # Repeat the cmd until fully uninstalled
RES="--index-url=https://resource.flagos.net/repository/flagos-pypi-hosted/simple"
python3.11 -m pip install flagtree===0.7.0+ascend3.5 $RES

2. Build from Source

2.1 Manually download the FlagTree dependencies

If your network connection is available, you do not need to download the dependencies which will be fetched automatically during the build.

mkdir -p ~/.flagtree/ascend; cd ~/.flagtree/ascend
wget https://baai-cp-web.ks3-cn-beijing.ksyuncs.com/trans/llvm-7d5de303-ubuntu-aarch64-python311-compat_v0.6.0.tar.gz
tar zxvf llvm-7d5de303-ubuntu-aarch64-python311-compat_v0.6.0.tar.gz

To check out which commit of flir & FlagPrism, please refer to https://github.com/flagos-ai/FlagTree/blob/triton_v3.5.x/python/setup_tools/utils/__init__.py

cd ${YOUR_CODE_DIR}/FlagTree/third_party
git clone https://github.com/flagos-ai/flir.git
git clone https://github.com/flagos-ai/FlagPrism.git

To check out which commit of AscendNPU-IR, please refer to https://github.com/flagos-ai/FlagTree/blob/triton_v3.5.x/python/setup_tools/utils/ascend.py

cd ${YOUR_CODE_DIR}/FlagTree/third_party/ascend
git clone https://github.com/flagos-ai/FlagTree-AscendNPU-IR.git

2.2 Manually download the Triton dependencies

The Triton dependencies are already downloaded and installed in the image. If your network connection is available, you do not need to download the dependencies which will be fetched automatically during the build.

cd ${YOUR_CODE_DIR}/FlagTree
# For Triton 3.5 (aarch64)
wget https://baai-cp-web.ks3-cn-beijing.ksyuncs.com/trans/build-deps-triton_3.5.x-linux-aarch64.tar.gz
sh python/scripts/unpack_triton_build_deps.sh ./build-deps-triton_3.5.x-linux-aarch64.tar.gz

After executing the above script, the original ~/.triton directory will be renamed, and a new ~/.triton directory will be created to store the pre-downloaded packages. Note that the script will prompt for manual confirmation during execution.

2.3 Build Commands

cd ${YOUR_CODE_DIR}/FlagTree
git checkout -b triton_v3.5.x origin/triton_v3.5.x
export PATH=~/.flagtree/ascend/llvm-7d5de303-ubuntu-aarch64-python311-compat/bin/:${PATH}  # clang
export FLAGTREE_BACKEND=ascend
MAX_JOBS=32 python3 -m pip install . --no-build-isolation -v

3. Testing and validation

After installing flagtree, you can check it with:

python3 -m pip show flagtree

Before testing, you need to execute source /usr/local/Ascend/ascend-toolkit/set_env.sh

Refer to Tests of ascend3.5 backend

4. Run Qwen vLLM benchmark

4.1 Use the image (910C)

IMAGE=harbor.baai.ac.cn/flagtree/flagtree-ascend3.5-910c-py311-cann9.0.0-ubuntu22.04-aarch64:202608-torch2.10.0-vllm0.20.2

4.2 Install FlagOS

Install flagtree as described above.

Install flag_gems and vllm-plugin-fl as follows.

git clone https://github.com/flagos-ai/FlagGems.git
cd FlagGems; git checkout qwen-vllm_for_ascend
python3 -m pip install --no-build-isolation .

git clone https://github.com/flagos-ai/vllm-plugin-FL.git
cd vllm-plugin-FL
python3 -m pip install --no-build-isolation .

4.3 Download the model

python3 -m modelscope.cli.cli download --model Qwen/Qwen3.6-27B \
    --local_dir ${MODEL_DIR}/Qwen3.6-27B
python3 -m modelscope.cli.cli download --model Qwen/Qwen3.6-35B-A3B \
    --local_dir ${MODEL_DIR}/Qwen3.6-35B-A3B

4.4 Run the benchmark

export BENCH_SCRIPT_DIR="${GIT_DIR}/FlagTree/.github/workflows/benchmark/ascend/ascend3.5-qwen3.6"
cd ${MODEL_DIR}
bash ${BENCH_SCRIPT_DIR}/start.sh
bash ${BENCH_SCRIPT_DIR}/test.sh
bash ${BENCH_SCRIPT_DIR}/perf.sh
bash ${BENCH_SCRIPT_DIR}/stop.sh

Refer to Qwen Benchmark of ascend3.5 backend

💫 Huawei Ascend（华为昇腾）ascend3.2

Based on Triton 3.2, aarch64
Available for 910B, 910C

1. Quick start

1.1 Use the preinstalled image (910C)

If you use this preinstalled image, you do not need to perform the later installation steps for 910C, and for 910B you only need to perform step 2.1.

# Plan A: docker pull (26.2GB)
IMAGE=harbor.baai.ac.cn/flagtree/flagtree-ascend-910c-py311-torch2.6.0-cann8.5.0-ubuntu22.04-aarch64:202603
docker pull ${IMAGE}
# Plan B: docker load (8.8GB)
IMAGE=flagtree-ascend-910c-py311-torch2.6.0-cann8.5.0-ubuntu22.04-aarch64:202603
wget https://baai-cp-web.ks3-cn-beijing.ksyuncs.com/trans/flagtree-ascend-910c-py311-torch2.6.0-cann8.5.0-ubuntu22.04-aarch64.202603.tar.gz
docker load -i flagtree-ascend-910c-py311-torch2.6.0-cann8.5.0-ubuntu22.04-aarch64.202603.tar.gz

CONTAINER=flagtree-dev-xxx
docker run -dit -u 0 --user=root \
    --network=host --pid=host --ipc=host --privileged \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \
    -v /usr/local/sbin/:/usr/local/sbin/ \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    --device=/dev/davinci0 --device=/dev/davinci1 \
    --device=/dev/davinci2 --device=/dev/davinci3 \
    --device=/dev/davinci4 --device=/dev/davinci5 \
    --device=/dev/davinci6 --device=/dev/davinci7 \
    --device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc \
    -v /etc/localtime:/etc/localtime:ro \
    -v /data:/data -v /home:/home -v /tmp:/tmp \
    -w /root --name ${CONTAINER} ${IMAGE} bash
docker exec -it ${CONTAINER} /bin/bash

1.2 Source-free Installation

# Note: First install PyTorch, then execute the following commands
python3 -m pip uninstall -y triton  # Repeat the cmd until fully uninstalled
RES="--index-url=https://resource.flagos.net/repository/flagos-pypi-hosted/simple"
python3.11 -m pip install flagtree===0.6.0+ascend3.2 $RES

flagtree is already installed in the preinstalled image.

2. Build from Source

2.1 Install CANN

A3 CANN is already installed in the 910C image. For 910B, register an account at https://www.hiascend.com/developer/download/community/result?module=cann and download the corresponding cann-ops for your platform.

# cann-toolkit (A2|A3)
chmod +x Ascend-cann-toolkit_8.5.0_linux-aarch64.run
./Ascend-cann-toolkit_8.5.0_linux-aarch64.run --install
# cann-ops for 910B (A2)
chmod +x Ascend-cann-910b-ops_8.5.0_linux-aarch64.run
./Ascend-cann-910b-ops_8.5.0_linux-aarch64.run --install
# cann-ops for 910C (A3)
chmod +x Ascend-cann-A3-ops_8.5.0_linux-aarch64.run
./Ascend-cann-A3-ops_8.5.0_linux-aarch64.run --install

2.2 Manually download the FlagTree dependencies

If your network connection is available, you do not need to download the dependencies which will be fetched automatically during the build.

mkdir -p ~/.flagtree/ascend; cd ~/.flagtree/ascend
wget https://baai-cp-web.ks3-cn-beijing.ksyuncs.com/trans/llvm-a66376b0-ubuntu-aarch64-python311-compat_v0.3.0.tar.gz
tar zxvf llvm-a66376b0-ubuntu-aarch64-python311-compat_v0.3.0.tar.gz

cd ${YOUR_CODE_DIR}/FlagTree/third_party
git clone https://github.com/flagos-ai/flir.git
cd flir
git checkout -b triton_v3.3.x origin/triton_v3.3.x  # For flagtree triton_v3.2.x triton_v3.3.x

cd ${YOUR_CODE_DIR}/FlagTree/third_party/ascend
git clone https://gitcode.com/Ascend/AscendNPU-IR.git
git checkout 5a3921f8

2.3 Manually download the Triton dependencies

The Triton dependencies are already downloaded and installed in the preinstalled image. If your network connection is available, you do not need to download the dependencies which will be fetched automatically during the build.

cd ${YOUR_CODE_DIR}/FlagTree
# For Triton 3.2 (aarch64)
wget https://baai-cp-web.ks3-cn-beijing.ksyuncs.com/trans/build-deps-triton_3.2.x-linux-aarch64.tar.gz
sh python/scripts/unpack_triton_build_deps.sh ./build-deps-triton_3.2.x-linux-aarch64.tar.gz

2.4 Build Commands

cd ${YOUR_CODE_DIR}/FlagTree/python
git checkout -b triton_v3.2.x origin/triton_v3.2.x
export FLAGTREE_BACKEND=ascend
MAX_JOBS=32 python3 -m pip install . --no-build-isolation -v

3. Testing and validation

You can check the installed flagtree with:

python3 -m pip show flagtree

Before testing, you need to execute source /usr/local/Ascend/ascend-toolkit/set_env.sh

Refer to Tests of ascend3.2 backend

User manual for ascend - flagos-ai/FlagTree GitHub Wiki

💫 Huawei Ascend（华为昇腾）ascend3.5

1. Quick start

2. Build from Source

3. Testing and validation

4. Run Qwen vLLM benchmark

💫 Huawei Ascend（华为昇腾）ascend3.2

1. Quick start

2. Build from Source

3. Testing and validation

⚠️ GitHub.com Fallback ⚠️

User manual for ascend - flagos-ai/FlagTree GitHub Wiki

💫 Huawei Ascend（华为昇腾）ascend3.5

1. Quick start

2. Build from Source

3. Testing and validation

4. Run Qwen vLLM benchmark

💫 Huawei Ascend（华为昇腾）ascend3.2

1. Quick start

2. Build from Source

3. Testing and validation

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️