Retrieval source audit 06: preparing 2,400 distinct questions

한국어 · Aggregate results

This is not a retrieval performance evaluation. Twenty-four questions with 100 candidates each are still 24 questions. Preserve the failed CoSQA classification results while auditing a separate source for candidate-finding experiments. No raw questions, code or weights are published here.

Observation	Count
Function/documentation pairs	14,291
Repositories	213
Unique lowercased/whitespace-normalized descriptions	11,678
Unique Go token sequences excluding comments/layout	14,291
Descriptions containing the function name as a word	12,908
Entire normalized description occurring verbatim in code	2
Descriptions shorter than three words	104
Generated-code hints in paths/function excerpts	34
Components connected by repository/equal query/equal code	205

Name matching ignores case and uses letter/digit/underscore word boundaries. Name presence is not automatically improper leakage, but named lookup and discovery without a known function name are different capabilities. Only 1,383 rows lack the function name: this source alone cannot supply 2,400 name-unknown questions. Rewriting descriptions to inflate the count would not resolve this limitation.

Go scanner removes comments/layout while preserving string literal contents and token boundaries. It does not detect all renamed or semantic clones. Repository membership and equal query/code identities are joined transitively to inform future leakage-resistant splits. The largest component contains 2,579 rows from one repository; meeting the size target with this one group would not establish diversity. No split or exclusion policy has yet been frozen.

Generated code and source terms

The source is the pinned CodeSearchNet distribution, Parquet SHA256 cb78d8951e9623bc2460d9b29c3c4152724f28c4f685fc019e0aafa071d2b18f, 9,751,451 bytes. Python performs maintainer-only Parquet projection; audit calculations run in Go.

Path hints cannot establish the actual generated-code count. chromedp/cdproto contributes 2,579 rows across 114 paths. One checked source-commit file, domstorage.go, contains the header Generated by the cdproto-gen command. Its SHA256 is 86521cac6e076b84c700a6236787d5be52fb211ab06de518db919fe76d48efa0. This does not verify the other 113 files. The 34 path/excerpt hints are not the total generated-code population.

Zero vendor paths does not prove absence of third-party code. Previous root-license checks cover only selected repositories. Per-file notices, generated code, duplicates and query provenance still require review; the report retains license_cleared=false. Documentation-derived queries are proxies, not actual user requests.

Next evaluation contract

1. Count at least 2,400 distinct questions, not candidate comparisons, repeated runs, translations or trivial rewrites.
2. Natural-language discovery claims require independent questions without a known function name. Keep documentation-based evaluation a separate proxy. Report name-present/absent strata; label name-removal tests as artificial diagnostics.
3. Split repository/duplicate-connected groups and report source-level results and concentration. Adopt sources only after terms are established. Do not select easy examples using model scores.
4. Measure known-target rank, Recall@k/MRR, candidates inspected, preparation time, search p50/p95 and full RSS. Other unjudged candidates are not automatically wrong.
5. Actual agent success, end-to-end time and LLM usage require separate evaluation. This source cannot establish savings or production readiness. Both CoSQA reserves remain unscored.

Reproduction

Prepare the pinned Parquet and a maintainer PyArrow environment. The projection contains raw rows and must stay local under .cache; use a new output path.

python scripts/training/project_retrieval_audit.py \
  --input .cache/codesearchnet-go/go/test-00000-of-00001.parquet \
  --output .cache/retrieval-audit-new.jsonl
go run ./cmd/riido-retrievalaudit --input .cache/retrieval-audit-new.jsonl

The Go CLI bounds input to 64MiB total, 2MiB per row and 100,000 rows. It rejects unknown fields, trailing JSON and empty input. The projection tool checks the pinned source hash; the Go report records the projection hash. The Go CLI alone does not authenticate provenance. Maps are used in this offline deduplication tool, outside inference paths.

On Apple M4 Pro/Go1.27.1 the compiled audit replay took 0.66s wall/0.30s user CPU and 63,143,936 bytes peak RSS. Output matched the first run byte-for-byte. These figures exclude Parquet projection/download and do not measure model inference.