Part 2 of 3. Part 1: from help pages to an index · Part 3: writing the answer, and learning from misses · Every design question, answered

A question arrives. In about a fifth of a second the assistant must decide which of 722 help sections answer it. It does that twice, two different ways, then merges the two:

• Word search - BM25, the classic search-engine scoring. Free, instant, no AI, exact about names.
• Meaning - the question's vector against every section's vector. Finds help that shares no words with the question.
• The merge - reciprocal rank fusion, words at half weight.

Neither alone is as good as both. This page goes through each with the real numbers from one question, then shows how the system decides how sure it is.

The running example, from prtest, as an administrator in English:

how do I create a ticketing department

function helpTokens(string $text): array
{
    $words = preg_split('/[^\p{L}\p{N}]+/u', mb_strtolower($text), -1, PREG_SPLIT_NO_EMPTY) ?: [];
    // ...drop words under 2 letters and common English ones (how, do, i, a, the, ...), then:
    $out[] = helpStem($w);
}

helpTokens('How do I create a ticketing department?') → ["creat", "ticket", "department"].

helpStem() is a cut-down Porter stemmer. It does not need to produce real words - only to make the question and the help agree, because both go through it:

TRAP: a stemmer's rules must chain. The first version returned after the first rule, so settings became setting and stopped, never meeting set; and adding lost its ing and then a doubled d, becoming ad. Now: plurals, then -ing/-ed (undoubling a final consonant only when more than three letters remain), then a silent e.

Non-English words are only lower-cased - that is the reason meaning, below, matters so much for other languages.

The first build scored whole sections, and the very first test question - this one - missed its answer. Tickets → Settings is about 10,000 characters of cards (the average section is 1,500), and BM25 rightly stops a long text winning just by being long, so its one Departments card was buried. Now each section is cut into passages of about 700 characters, made of whole blocks (helpPassages()), passages are scored, and a section scores as its best passage - which also becomes its preview.

On prtest that is 1,853 passages for 722 sections.

For each question word t in a passage d:

$$\text{score}(d) = \sum_{t} \text{idf}(t)\cdot\frac{f(t,d),(k_1+1)}{f(t,d)+k_1\left(1-b+b,\frac{|d|}{\overline{|d|}}\right)} \qquad \text{idf}(t)=\ln!\left(1+\frac{N-n_t+0.5}{n_t+0.5}\right)$$

k₁ = 1.2 and b = 0.75 (the textbook values). N is the number of passages, nₜ how many contain the word, f how often it appears in this passage, |d| the passage's length in words against the average. Title words are counted three times (HELP_TITLE_WEIGHT), and a section from the module the asker is in gets ×1.15 (HELP_MODULE_BOOST) - a tie-breaker, never a filter.

The real numbers for our question:

That is the whole intuition of BM25: a word everyone uses barely counts, a rare word counts a lot, repetition helps with diminishing returns, and length is evened out.

The top of the word ranking:

Good, not perfect: orphaned tickets talks about deleting departments - all three words, wrong meaning. Words cannot tell "create" from "delete" near the same nouns.

helpRank() is pure - arrays in, arrays out - so all of this is tested without a database. Over 722 sections it takes about 80 ms in PHP.

An embedding model maps a text to a point in a space with hundreds of dimensions, placing texts that mean similar things near each other - whatever words they use, in whatever language. The question is embedded when it is asked (OpenAI's text-embedding-3-small, 512 dimensions, about 0.4 s), and compared with every candidate section's stored vector by cosine similarity - the angle between them:

$$\cos\theta=\frac{a\cdot b}{\lVert a\rVert,\lVert b\rVert}$$

function aiCosineSimilarity(array $a, array $b): float
{
    $dot = 0.0; $na = 0.0; $nb = 0.0;
    for ($i = 0, $len = min(count($a), count($b)); $i < $len; $i++) {
        $dot += $a[$i] * $b[$i]; $na += $a[$i] * $a[$i]; $nb += $b[$i] * $b[$i];
    }
    return ($na == 0.0 || $nb == 0.0) ? 0.0 : $dot / (sqrt($na) * sqrt($nb));
}

1 is the same direction, 0 unrelated. helpVectorRank() loads only the candidates' vectors (the analyst's language and English - not all 1,381), compares them all (about 16 ms), and ranks:

Meaning puts Departments second, and pushes the orphaned section (the wrong meaning) down. Its own first choice, the Tickets overview, is close but generic.

Every question gets a nearest section - even nonsense. So similarities under 0.32 are discarded (HELP_VECTORS_FLOOR):

Without the floor, "xyzzy" got five confident-looking help cards. With it, meaning has nothing to add and the reply is honestly "I couldn't find anything".

This is where meaning earns its keep. Word search cannot match "Hogyan hozzak létre jegykezelő részleget?" to English help - there are no shared words. Vectors place it near "Create departments like IT, HR and Finance" anyway, because the model learned both languages. (Part 3: the AI covers the rest by searching in English itself.)

Two rankings, on different scales (BM25 scores against similarities). Rather than try to compare the numbers, reciprocal rank fusion compares positions: each list adds weight ÷ (60 + position) to each section it ranks, and the sums are sorted.

foreach ($word as $i => $r)          $fused[$r['section_key']] += HELP_MERGE_WORD_WEIGHT / (60 + $i + 1);   // 0.5
foreach ($vec['ranked'] as $i => $r) $fused[$r['section_key']] += 1 / (60 + $i + 1);
arsort($fused);

The 60 is the standard constant: it stops the very top of one list swamping everything. Words count half - the value the question set chose (below).

Our question, worked:

The section that wins was first in neither list. It was strong in both - which is exactly the signal that two independent methods agree. The word ranking's false friend (orphaned tickets) drops to third; meaning's generic first choice (the overview) drops off the top five entirely. Both right answers are now first and second.

A section only meaning found still makes the list - with the start of its text as the preview, since there is no matched passage.

Each merge was run over the cached vectors of the question set (no API cost):

The search returns the five best. Without a chat AI, the window replies with a short canned line and those sections as cards - and the line must never sound surer than the search is. helpAssistantReplyKind():

Every threshold came from the question set's real scores, not a guess. With vectors on, clear winner came up 37 times: a right section first 32 times, in the top five the other 5 - never wrong. Our department question is several - correctly: its two right answers are close together, and so is a wrong one.

The lines live in the language files (help_assistant.phrase.<kind>.N), as many variants as wanted, picked at random - and none claims an AI read anything.

Matching by meaning needs the embedding model right now, for the question. Every way that can fail falls back to words alone, silently:

TRAP: embed the question with the model IN USE, not the model chosen. Once an admin saves a new model, the saved settings describe the new one - but the vectors answering were made by the old one. The first version embedded questions with the saved settings, so choosing a model silently dropped the assistant to words until the rebuild finished. The model in use now keeps its own settings (help_vectors_active_cfg, its key encrypted), written at the moment a rebuild switches over; helpVectorsReady() embeds questions with those.

tests/help-search-questions.php holds 65 realistic questions, each with every section that would be a right answer, in four kinds: plain ("how do I create a ticketing department"), exact names ("Warbot", "MoSCoW"), vague - sharing no words with the answer ("my boss wants a weekly PDF of how we are doing"), and other languages (German, Spanish, Hungarian, Polish). Every step of the build was measured against the same set:

A right section in the first five; for the AI row, the answer cites a right section.

TRAP: help that quotes the test questions flatters the measurement. The help assistant's own System help page uses "my boss wants a weekly PDF" as an example - so word search now ranks that help page first for the very question meant to test meaning. The merge still finds Report packs (meaning ranks it second), but a vague question's word score is no longer honest. When writing help for the assistant itself, use examples that are not in the question set - or add the question set's wording to the list of things help pages avoid.

Next: Part 3 - writing the answer, and learning from misses