[HiCache] 测试脚本 - shun8686/sglang GitHub Wiki
脚本
先设置变量
HICACHE_TEST_URL={sglang推理url，如http://127.0.0.1:2000}
HICACHE_TEST_SERVER_URL={对于PD分离设置为prefill的url, 不是router的，对于PD混部无需设置}
HICACHE_TEST_MODEL={qwen3.6-hybrid-mamba、deepseek-v4、glm5.2-dsa、kimi-k3-kda}
测试L3:
python -m unittest test_hicache.TestNPUHiCache.test_npu_restores_from_l3
"""
NPU HiCache L2/L3 coverage
Server requirements (see launch script):
--max-total-tokens 65536 (L1 device KV pool: 65536 / 128 = 512 pages)
--chunked-prefill-size 65536
--hicache-ratio 2 (L2 host pool: 131200 tokens)
--enable-hierarchical-cache
--hicache-storage-backend npu_memcache
--hicache-write-policy write_through
--hicache-io-backend kernel_ascend
Select the model profile with the HICACHE_TEST_MODEL env var (default
"qwen3.6-hybrid-mamba"):
HICACHE_TEST_URL=http://127.0.0.1:8869 HICACHE_TEST_MODEL=deepseek-v4 python3 test_npu_hicache_npu_memcache.py
For PD-disaggregated runs behind a router, HICACHE_TEST_URL is the router
(/generate, /flush_cache) and HICACHE_TEST_SERVER_URL must point at a prefill
engine for the engine-only endpoints (/tokenize, /hicache/storage-backend/clear):
HICACHE_TEST_URL=http://127.0.0.1:2000 HICACHE_TEST_SERVER_URL=http://127.0.0.1:8869 ...
Everything model-specific lives in MODEL_PRESETS below; the shared server
knobs are the constants right above it. Per-model behaviours covered:
1. Cacheable-prefix alignment. The radix cache is a UnifiedRadixCache whose
effective cache length is floored by the stateful sidecar component:
- qwen3.6-hybrid-mamba: the Mamba checkpoint grid (mamba_track_interval)
floors a 3200-token prompt to 3072 tokens (24 device pages).
- deepseek-v4: the c128 sparse-attention sidecar groups 128 * c128_page_size
(= 2048) tokens, flooring a 3200-token prompt to 2048 tokens (16 pages).
`_prime_on_device` asserts the exact aligned length, so a wrong
`cache_align_tokens` in the profile fails fast.
2. Commit lag (mamba only). A node only matches when its FULL pages AND its
sidecar state are resident. Under the mamba profile the checkpoint of a
finished request is committed to the radix node only in that request's
closing event round, which the NEXT request's prefix match has already
passed, so a fresh prefix needs TWO priming requests and reports its first
device hit only on the THIRD request. Verified identical under
SGLANG_NPU_HICACHE_MAMBA_IO=sync and =async: that env var only selects the
device<->host state-copy operator, not the commit/match ordering.
Pure-attention profiles (deepseek-v4) commit synchronously: the repeat
already hits on the second request.
3. Prompt distribution matters for the equality checks (mamba only). Full
prefill and checkpoint replay are not bit-identical in bf16 on the Ascend
kernels, so greedy decoding of out-of-distribution random token ids
(where every token is a near-tie) frequently flips after a load-back. The
prompts here are therefore 3200-token windows of real natural language
tokenized through the server, on which greedy decoding is stable and every
load-back reproduces the resident output exactly. Pure-attention profiles
copy KV bit-exactly and do not need natural text, but all profiles use it
alike for uniformity.
4. Capacity. The L1 pool holds the base prefix plus N unique pressure peers.
Eviction is driven by the SMALLEST sub-pool, not the FULL pool. The mamba
profile only has the 512-page FULL pool (threshold: 20 peers; 22/17
measured on hardware). deepseek-v4 is a FULL+SWA+C4+C128 four-pool model
where SWA = full * swa_full_tokens_ratio (~0.2-0.25, i.e. ~102-128 pages)
is the bottleneck: every peer parks ~25 SWA pages (window >= prompt), so
eviction starts at just 3 peers. The deepseek-v4 counts below are
measured on hardware (65536 FULL pool); re-measure if SWA sizing or the
pool budget changes.
"""
import os
import time
import unittest
import requests
from sglang.test.kl_test_utils import (
_flush_cache,
_generate,
)
from sglang.test.test_utils import (
CustomTestCase,
)
# ---------------------------------------------------------------------------
# Shared server knobs — must match the launch script of the server under test.
# ---------------------------------------------------------------------------
URL_FOR_TEST = os.environ.get("HICACHE_TEST_URL", "http://127.0.0.1:8869")
# Engine URL for engine-only endpoints (/tokenize, /hicache/storage-backend/
# clear). In PD-disaggregated deployments HICACHE_TEST_URL points at the
# router, which only proxies /generate and /flush_cache — set this to any
# prefill engine's URL. Defaults to HICACHE_TEST_URL for single-server runs.
SERVER_URL_FOR_TEST = os.environ.get("HICACHE_TEST_SERVER_URL", URL_FOR_TEST)
# model name: qwen3.6-hybrid-mamba、deepseek-v4、glm5.2-dsa、kimi-k3-kda
MODEL_NAME = os.environ.get("HICACHE_TEST_MODEL", "qwen3.6-hybrid-mamba")
PAGE_SIZE = 128 # --page-size
PROMPT_LEN = 3200 # pressure prompt length in tokens
MAX_TOTAL_TOKENS = 65536 # --max-total-tokens (L1 device KV pool)
# ---------------------------------------------------------------------------
# Per-model profiles — the ONLY block to touch when porting to a new model.
#
# page_size --page-size of the service. The radix matcher floors
# the cacheable prefix to a multiple of this
# (radix_cache.page_aligned), so expected cached len =
# PROMPT_LEN // page_size * page_size unless a sidecar
# component floors further.
# cache_align_tokens effective floor granularity of the cacheable prefix:
# max(page_size, sidecar grid). For models without a
# stateful sidecar this equals page_size; for mamba
# it's the track grid; for c128 it's 128*c128_page_size.
# prime_requests requests sent before asserting a device hit:
# 3 for mamba (commit lag), 2 for pure attention.
# commit_lag True if the sidecar commit lags one event round
# (mamba); drives the warm-request assertions.
# pressure_peers_evict unique pressure prompts for the L2 eviction case;
# must push the base prefix out of the L1 pool.
# pressure_peers_l1_only unique pressure prompts for the L1-only case; must
# leave the base prefix inside the L1 pool.
# Both counts are measured on hardware per model: the
# eviction threshold is set by the smallest KV sub-pool
# (FULL for mamba, SWA for deepseek-v4), not by the
# FULL pool alone.
# ---------------------------------------------------------------------------
MODEL_PRESETS = {
"qwen3.6-hybrid-mamba": dict(
page_size=128,
cache_align_tokens=256, # mamba_track_interval grid
prime_requests=3,
commit_lag=True,
pressure_peers_evict=22, # measured: eviction starts at 20 peers
pressure_peers_l1_only=17, # measured: 19 peers stay on L1
),
"deepseek-v4": dict(
page_size=128,
cache_align_tokens=2048, # c128 sidecar: 128 * c128_page_size(16)
prime_requests=2,
commit_lag=False,
pressure_peers_evict=4, # measured: eviction starts at 3 peers (SWA pool)
pressure_peers_l1_only=2, # measured: 2 peers stay on L1 (no margin)
),
"glm5.2-dsa": dict(
# GlmMoeDsaForCausalLM (MLA+DSA): FULL component only (server.log
# confirms "Components: (FULL,)"). page_size=128 (NPU default). Measured
# cached=3072 for a 3200-token prompt — root cause unclear (not
# page_aligned, not FULL floor_cache_len, not sidecar); possibly DSA
# backend reports cached_tokens differently. cache_align_tokens=256
# matches the measured value. No commit lag, prime hits on 2nd request.
# Pool = 65536/128 = 512 pages; each peer = 3072/128 = 24 pages;
# threshold ~512/24 ≈ 21 -> 22/17 (not yet calibrated on hardware).
page_size=128,
cache_align_tokens=256, # measured, root cause TBD
prime_requests=2,
commit_lag=False,
pressure_peers_evict=22,
pressure_peers_l1_only=17,
),
"kimi-k3-kda": dict(
# KimiK3 (MLA + KDA linear attention): mambaish via kimi_linear_config
# (hybrid_arch.py), so the radix cache carries the MAMBA component —
# same commit-lag timing (hit on the 3rd request) and checkpoint-grid
# flooring as the Qwen hybrid profile. KDA chunk is the FLA 64, so
# grid = lcm(64->128, page 128, track_interval 256) = 256.
page_size=128,
cache_align_tokens=256,
prime_requests=3,
commit_lag=True,
# Computed: single FULL MLA pool (512 pages, 24-page peers ->
# threshold ~20); not yet calibrated on hardware.
pressure_peers_evict=22,
pressure_peers_l1_only=17,
),
}
# Natural-language corpus, sized dynamically in _build_prompts to cover
# base + pressure peers as disjoint PROMPT_LEN-token windows.
_CORPUS_PARAGRAPH = (
"The history of artificial intelligence began in antiquity, with myths "
"and stories of artificial beings endowed with intelligence by master "
"craftsmen. Modern artificial intelligence research began in the middle "
"of the twentieth century and has since transformed industry, science, "
"medicine, transportation, education, finance, agriculture, energy, "
"manufacturing, entertainment, communication, security, logistics, law, "
"public administration, scientific discovery, creative work, software "
"engineering, weather forecasting, space exploration, and everyday life. "
)
class TestNPUHiCache(CustomTestCase):
"""Exercise L2/L3 load-back on NPU across model profiles."""
page_size = MODEL_PRESETS[MODEL_NAME].get("page_size", PAGE_SIZE)
prompt_len = PROMPT_LEN
# L1 device KV pool size, must match the server's --max-total-tokens.
# 65536 tokens / page_size = device page count.
max_total_tokens = MAX_TOTAL_TOKENS
routed_dp_rank = 0
@classmethod
def _clear_storage_backend(cls):
requests.post(
f"{cls.admin_url}/hicache/storage-backend/clear", timeout=60
).raise_for_status()
@classmethod
def setUpClass(cls):
super().setUpClass()
cls.base_url = URL_FOR_TEST
cls.admin_url = SERVER_URL_FOR_TEST
preset = MODEL_PRESETS[MODEL_NAME]
cls.model_name = MODEL_NAME
cls.cache_align_tokens = preset["cache_align_tokens"]
cls.prime_requests = preset["prime_requests"]
cls.commit_lag = preset["commit_lag"]
cls.pressure_peers_evict = preset["pressure_peers_evict"]
cls.pressure_peers_l1_only = preset["pressure_peers_l1_only"]
# Cacheable prefix of one prompt after sidecar alignment, and the
# device pages each cached peer parks in the L1 pool.
cls.expected_cached_len = (
cls.prompt_len // cls.cache_align_tokens * cls.cache_align_tokens
)
cls.peer_pages = cls.expected_cached_len // cls.page_size
cls.prompts = cls._build_prompts()
def setUp(self):
# /flush_cache only clears L1/L2 (radix tree + host pool); the L3
# storage backend survives it. Clear both per test method so each
# case starts from a true cold state — otherwise write-through data
# left by a previous test makes the L3 cold request a storage hit.
super().setUp()
_flush_cache(self.base_url)
self._clear_storage_backend()
@classmethod
def _build_prompts(cls):
"""Tokenize one natural-language corpus and slice it into disjoint
prompt_len-token windows: prompts[0] is the base prefix, the rest are
unique pressure prompts. The corpus is sized dynamically to cover
base + pressure_peers_evict windows."""
probe = requests.post(
f"{cls.admin_url}/tokenize", json={"prompt": _CORPUS_PARAGRAPH},
timeout=60,
)
probe.raise_for_status()
tokens_per_para = len(probe.json()["tokens"])
need = 1 + cls.pressure_peers_evict
min_tokens = need * cls.prompt_len
# Tokenizer merges across paragraph boundaries, so the estimate can
# fall slightly short; grow the corpus until it covers the need.
repeats = -(-min_tokens // tokens_per_para)
while True:
resp = requests.post(
f"{cls.admin_url}/tokenize",
json={"prompt": _CORPUS_PARAGRAPH * repeats},
timeout=120,
)
resp.raise_for_status()
token_ids = resp.json()["tokens"]
if len(token_ids) >= min_tokens:
break
repeats += max(1, repeats // 16)
prompts = [
token_ids[i * cls.prompt_len : (i + 1) * cls.prompt_len]
for i in range(need)
]
assert all(len(p) == cls.prompt_len for p in prompts), (
f"Corpus too short: only {len(token_ids)} tokens tokenized, "
f"need {need * cls.prompt_len}"
)
return prompts
def _generate_one(self, input_ids, max_new_tokens, return_logprob=False):
results = _generate(
self.base_url,
[input_ids],
max_new_tokens=max_new_tokens,
return_logprob=return_logprob,
temperature=0,
routed_dp_rank=self.routed_dp_rank,
)
self.assertEqual(len(results), 1)
return results[0]
@staticmethod
def _tier_details(meta_info):
details = meta_info.get("cached_tokens_details") or {}
return (
int(meta_info.get("cached_tokens", 0) or 0),
int(details.get("device", 0) or 0),
int(details.get("host", 0) or 0),
int(details.get("storage", 0) or 0),
)
def _prime_on_device(self, base_prompt):
"""Make base_prompt device-resident and return the first device-hit
response (the deterministic-output reference).
Mamba profiles: the sidecar checkpoint of a finished request lands
one event round after the next request's prefix match, so
prime_requests (3) are required before the device hit shows up.
Pure-attention profiles commit synchronously and hit on the second
request (prime_requests = 2, one spare round).
"""
for _ in range(self.prime_requests - 1):
self._generate_one(base_prompt, max_new_tokens=16)
resident = self._generate_one(base_prompt, max_new_tokens=16)
cached, device, host, storage = self._tier_details(resident["meta_info"])
print(
f"[prime] prompt_tokens={len(base_prompt)} "
f"cached={cached} device={device} host={host} storage={storage}"
)
self.assertEqual(
cached,
self.expected_cached_len,
f"Cacheable prefix {cached} != expected {self.expected_cached_len} "
f"(prompt_len={self.prompt_len} aligned to "
f"{self.cache_align_tokens}); check cache_align_tokens of the "
f"'{self.model_name}' profile",
)
self.assertGreater(
device,
0,
"Base prefix did not become device-resident after the priming "
f"requests: {resident['meta_info']}",
)
self.assertEqual(host, 0, f"Unexpected L2 contribution while priming: {resident['meta_info']}")
self.assertEqual(storage, 0, f"Unexpected L3 contribution while priming: {resident['meta_info']}")
return resident
def test_npu_restores_from_l2(
self,
pressure_prompts_num=None,
expect_evict: bool = True,
):
"""L2 evict and restore test"""
if pressure_prompts_num is None:
pressure_prompts_num = self.pressure_peers_evict
base_prompt = self.prompts[0]
pressure_prompts = self.prompts[1 : 1 + pressure_prompts_num]
_flush_cache(self.base_url)
# The priming rounds absorb the mamba sidecar commit lag (profiles
# without it simply run one spare round); the final priming request
# is the reference served entirely from L1 device.
first = self._prime_on_device(base_prompt)
# Fill the L1 pool ({self.max_total_tokens // self.page_size} pages)
# with unique pressure prompts and evict the oldest prefix
# (base_prompt) to the host tier. After sidecar alignment each cached
# peer parks self.peer_pages device pages; the eviction pressure
# count comes from the model profile.
for prompt in pressure_prompts:
self._generate_one(prompt, max_new_tokens=1)
after_pressure = self._generate_one(
base_prompt, max_new_tokens=16, return_logprob=True
)
meta_info = after_pressure["meta_info"]
cached_tokens, device, host, storage = self._tier_details(meta_info)
print(
f"[L2] n={pressure_prompts_num} expect_evict={expect_evict} "
f"cached={cached_tokens} device={device} host={host} storage={storage}"
)
self.assertGreater(
cached_tokens,
0,
f"Expected cached_tokens to be greater than 0, got {cached_tokens}",
)
self.assertEqual(
storage,
0,
f"Expected no L3 contribution: {meta_info.get('cached_tokens_details')}",
)
if expect_evict:
self.assertEqual(
device,
0,
"Expected the base prefix gone from L1 after pressure; "
f"got cached_tokens_details={meta_info.get('cached_tokens_details')}. "
"If it stayed on L1, raise pressure_peers_evict for this profile.",
)
self.assertGreater(
host,
0,
"Expected the original prefix to be restored from the HiCache host tier; "
f"got cached_tokens={cached_tokens}, "
f"cached_tokens_details={meta_info.get('cached_tokens_details')}",
)
else:
self.assertGreater(
device,
0,
"Expected the base prefix to stay on L1 under the pressure; "
f"got cached_tokens_details={meta_info.get('cached_tokens_details')}. "
"If it was evicted, lower pressure_peers_l1_only for this profile.",
)
self.assertEqual(
host,
0,
"Expected the original prefix not to be restored from the HiCache host tier; "
f"got cached_tokens={cached_tokens}, "
f"cached_tokens_details={meta_info.get('cached_tokens_details')}",
)
self.assertEqual(
after_pressure["text"],
first["text"],
"L2 restore changed deterministic generation output.",
)
def test_npu_only_l1(self):
# pressure_peers_l1_only leaves the base prefix comfortably inside
# the L1 pool (per-profile value, see MODEL_PRESETS), so it is never
# demoted to the host tier.
self.test_npu_restores_from_l2(
pressure_prompts_num=self.pressure_peers_l1_only, expect_evict=False
)
def test_npu_restores_from_l3(self):
"""L3 restore test"""
_flush_cache(self.base_url)
base_prompt = self.prompts[0]
# First request is a cold miss.
cold = self._generate_one(base_prompt, max_new_tokens=16)
cold_meta = cold["meta_info"]
print(f"[L3][1] cold_meta.cached_tokens_details={cold_meta.get('cached_tokens_details')}")
self.assertEqual(int(cold_meta.get("cached_tokens", 0)), 0)
# The second request is the "warm" request that starts the
# L1 -> L2 -> L3 write-through. Under mamba profiles its checkpoint
# is not matchable yet (the commit lands one event round later), so
# it still reports zero cached tokens; under pure-attention profiles
# it is already a device hit.
warm = self._generate_one(base_prompt, max_new_tokens=16)
warm_cached, warm_device, _, _ = self._tier_details(warm["meta_info"])
print(f"[L3][2] warm_meta.cached_tokens_details={warm['meta_info'].get('cached_tokens_details')}")
if self.commit_lag:
self.assertEqual(
warm_cached,
0,
"Mamba profiles report the first device hit only on the "
f"third request; got cached_tokens={warm_cached}",
)
else:
self.assertGreater(
warm_device,
0,
"Pure-attention profiles should hit on the second request; "
f"got cached_tokens_details={warm['meta_info'].get('cached_tokens_details')}",
)
# The third request is a device hit for every profile and is used as
# the deterministic-output reference below (mamba full-prefill and
# checkpoint-replay paths differ slightly in bf16, so the cold output
# is not a valid reference).
first = self._generate_one(base_prompt, max_new_tokens=16)
first_meta = first["meta_info"]
_, device, host, storage = self._tier_details(first_meta)
print(f"[L3][3] resident cached_tokens_details={first_meta.get('cached_tokens_details')}")
self.assertGreater(device, 0, f"Expected an L1 device hit: {first_meta}")
self.assertEqual(host, 0, f"Expected no L2 contribution: {first_meta}")
self.assertEqual(storage, 0, f"Expected no L3 contribution: {first_meta}")
time.sleep(5)
# Remove L1/L2 residency without clearing the storage (L3) backend.
_flush_cache(self.base_url)
restored = self._generate_one(base_prompt, max_new_tokens=16)
restored_meta = restored["meta_info"]
restored_details = restored_meta.get("cached_tokens_details") or {}
print(f"[L3][4] restored_meta.cached_tokens_details={restored_details}")
device = int(restored_details.get("device", 0))
host = int(restored_details.get("host", 0))
storage = int(restored_details.get("storage", 0))
self.assertEqual(
device,
0,
f"Expected no L1 device contribution: {restored_details}",
)
self.assertEqual(
host,
0,
f"Expected no L2 contribution: {restored_details}",
)
self.assertGreater(
storage,
0,
f"Expected an L3 storage hit; got meta_info={restored_meta}",
)
self.assertEqual(
restored["text"],
first["text"],
"L3 restore changed deterministic generation output.",
)
# The load-back repopulates L1, so the next repeat is a device hit.
restored = self._generate_one(base_prompt, max_new_tokens=16)
restored_meta = restored["meta_info"]
details = restored_meta.get("cached_tokens_details") or {}
print(f"[L3][5] restored_meta.cached_tokens_details={details}")
self.assertGreater(
int(details.get("device", 0) or 0),
0,
f"Expected an L1 device hit; got meta_info={restored_meta}",
)
if __name__ == "__main__":
unittest.main()
NPU HiCache 分层缓存测试设计经验总结
来源：
test_design/test_npu_hicache_npu_memcache.py的编写与调通过程。 覆盖模型：Qwen3.6-35B-A3B（FULL+MAMBA 混合 MoE）、DeepSeek V4（MLA + SWA 稀疏四池）。 服务：SGLang NPU（Ascend），--enable-hierarchical-cache+--hicache-storage-backend npu_memcache。
1. 三层缓存模型与观测手段
| 层 | 位置 | 触达条件 |
|---|---|---|
| L1 device | NPU KV 池（--max-total-tokens） |
前缀匹配直接命中 |
| L2 host | CPU 侧镜像池（--hicache-ratio × L1） |
L1 淘汰后（write_through 下 insert 时即备份） |
| L3 storage | npu_memcache（SDMA 到外部存储） | L2 也淘汰后，或手动 flush |
观测方法（最重要）：/generate 返回的 meta_info.cached_tokens_details 含
device / host / storage 三元组，测试的核心断言全部建立在这三个数上：
def _tier_details(meta_info):
d = meta_info.get("cached_tokens_details") or {}
return (meta_info["cached_tokens"], d.get("device", 0),
d.get("host", 0), d.get("storage", 0))
注意 /generate 的 JSON 是 list（取 r.json()[0]），不是 dict。
2. 核心机制发现（均为实测，勿凭直觉假设）
2.1 命中时序：有 sidecar 状态的模型要 prime 多次
- 混合 Mamba（
SGLANG_NPU_HICACHE_MAMBA_IO=sync/async均一致）： 请求结束时prepare_for_caching_req才把 mamba checkpoint 挂到 radix 节点 （unified_cache/components/mamba.py），与下一请求的前缀匹配在调度器 event loop 里错开一轮 → 新前缀要 3 次请求才报 device 命中（#1/#2 cached=0）。 - 纯注意力（DSV4）：KV 同步入树，第 2 次即命中。
SGLANG_NPU_HICACHE_MAMBA_IO只选 device↔host 拷贝算子（torch fallback vs sgl-kernel-npu 直传 kernel），不影响可见性时序。- sleep 无效：这是轮次顺序问题，不是时间延迟问题。
测试用 _prime_on_device 连发 prime_requests 次，取最后一次为
resident 基准输出。
2.2 cache_align_tokens：可缓存前缀会被向下取整
带状态 sidecar 的组件会把有效缓存长度 floor 到自己的粒度：
| 模型 | 公式 | 3200-token prompt 实际 cached |
|---|---|---|
| Qwen3.6（mamba） | lcm(max(64, page), page, mamba_track_interval) = lcm(128,128,256) = 256 |
3072（24 页） |
| Kimi K3（KDA，mambaish 路径） | 同上公式（KDA chunk=FLA 64）= 256 | 3072（24 页） |
| DeepSeek V4（c128 sidecar） | 128 × c128_page_size(16) = 2048 |
2048（16 页） |
| GLM5.2（GlmMoeDsa） | 实测 cached=3072，page_size=128（NPU 默认），FULL 组件无 floor，根因未定 = 256 | 3072（24 页） |
mamba_track_interval默认 256（不设也是它）；c128_page_size默认 16。- sidecar 组件注册条件见
mem_cache/registry.py：mambaish → MAMBA （mambaish_config含 KDA/KimiLinear/GDN/glm5_next 等，见hybrid_arch.py），req_to_token_pool.req_to_c128_sidecar存在 → C128（DSV4 专属），其余模型只有 FULL。 - 定新模型值：实测最可靠——同一 prompt 发两次看
cached_tokens； 测试里对 prime 的断言就是cached == PROMPT_LEN // align * align， 填错会直接报出实际值。
2.3 淘汰阈值：由最小子池决定，而非 FULL 池
- 淘汰触发：某子池分配失败 →
evict_for_alloc从 LRU 逐出 radix 节点 （write_through 下节点早已备份到 host → 表现为 host 命中）。 - Qwen3.6：只有 FULL 池（65536/128 = 512 页），每 peer 占 24 页 → 阈值 20 个 peer（19 留 L1，20 淘汰）。
- DSV4：FULL+SWA+C4+C128 四池（
pool_configurator.py）：swa = full × swa_full_tokens_ratio（~0.2-0.25）≈ 102-128 页是瓶颈， 每 peer 占 ~25 SWA 页 → 阈值仅 3 个 peer！ - GLM5.2（GlmMoeDsa）：FULL 组件 only（server.log 确认）， page_size=128（NPU 默认），每 peer 24 页，池 512 页 → 推算阈值 ~21 （profile 暂用 22/17，待实测校准）。
- Kimi K3（KDA）：mambaish 路径但 FULL 池与 Qwen 同为 512 页、 每 peer 24 页 → 推算阈值 ~20（profile 暂用 22/17，待实测校准； KDA state 槽位可能额外压低 max_running_requests，不影响本用例）。
- 教训：
--max-total-tokens只定 FULL 池；多池模型的阈值可能比 按 FULL 池推算小一个数量级。所有压力数必须实测 sweep，并留 1~2 的裕量。
2.4 文本保真：load-back 不总是位精确
- mamba（bf16 checkpoint 重放）：随机 token id prompt 的输出在 load-back 后高频翻转（6/8 seed 在第 4~19 token 分歧，L2/L3 分歧位置相同 → 是 checkpoint 重放路径的通病，非某层存储的缺陷）。原因是 OOD 随机 token 上每步 logprob 近平局，重放的微小数值差被放大。
- 对策：用
/tokenize切真实自然语言语料做 prompt——自然文本下 8/8 严格一致。 - 纯注意力（DSV4）：load-back 是 KV 位拷贝（无重放），理论上随机 prompt 也可，统一用自然语料保持一致。
- 对比基准：cold 全 prefill 输出与 checkpoint 重放本就不同（混合模型 路径差异），不能拿第一次 cold 输出当期望值；必须用 resident （device 命中那次）输出做基准。
2.5 清理接口的坑
POST /flush_cache不清 L3 storage；L3 清空要用POST /clear_hicache_storage。- 原
_clear_storage_backend有f" {url}"（URL 前导空格）bug → 请求直接 404 且静默失败。URL 拼接要 strip。
3. 可复用的测试结构
setUpClass
├─ 读 MODEL_PRESETS[profile]（环境变量 HICACHE_TEST_MODEL 切换）
├─ 派生 expected_cached_len / peer_pages
└─ /tokenize 生成互不相交的自然语料 prompt（数量按 peer 数动态补）
test_npu_only_l1 → 调 restores_from_l2(少, expect_evict=False)
test_npu_restores_from_l2 → prime → 压力 → 断言 host>0 / device=0 → 文本=resident
test_npu_restores_from_l3 → prime → /flush_cache → 断言 storage>0/device=host=0
→ 再请求 → 断言回到 device>0，文本=resident
断言写法要点：
- 每个断言携带完整三元组：
assertGreater(cached, 0, f"cached={c} device={d} host={h} storage={s}")—— 失败时一眼看出 淘汰过头/不足。 - 淘汰边界断言成对（evict=阈值+裕量 / l1_only=阈值-裕量）， 并把实测阈值写进注释。
- profile 参数化：
cache_align_tokens / prime_requests / commit_lag / pressure_peers_*是唯一换模型要动的地方；常量必须有"实测 or 推算"标注。
4. 调通过程中的误判记录（避免重蹈）
| 误判 | 排除手段 |
|---|---|
| 以为不命中是容量不够 | 极小 peer 数复现 → 与容量无关 |
| 怀疑 routed_dp_rank 影响匹配 | 带/不带 rank 矩阵实验 → 行为一致 |
| 怀疑 async mamba IO 导致 3 次命中 | sync 下完全一致 → 是提交滞后一轮 |
| 按 FULL 池推 DSV4 阈值 ~32 | 实测 n=3 就淘汰 → SWA 池才是瓶颈 |
| 用 cold 输出做文本基准 | resident 与 L2/L3 恢复严格一致、cold 不一致 → 换基准 |
| 随机 token 导致文本断言随机失败 | 换自然语料 → 8/8 稳定 |
5. 新模型适配清单（checklist）
- 确定
cache_align_tokens（实测两次请求的 cached_tokens，或按 §2.2 公式推）。 - 确定
prime_requests/commit_lag（有状态 sidecar → 3/True）。 - 实测淘汰阈值：从 n=1 起步逐个加压，记录
[cached device host]由 device→host 的拐点；注意先查 server.log 里各子池大小 （DSV4 找DSV4 SWA sizing行），判断哪个池最小。 - 检查输出保真路径（位拷贝 or 状态重放），决定 prompt 语料与断言严格度。
- 确认
--page-size、--max-total-tokens、--hicache-ratio与共享常量一致。 - 切换服务/模型不改代码：
HICACHE_TEST_URL（服务地址，默认 127.0.0.1:8869）+HICACHE_TEST_MODEL（profile）。