TokenExplorer - Gnurro/FinetuneReFormatter GitHub Wiki

TokenExplorer Mode

Find tokens containing a specified string.
(More things like split-checking and ambiguity assessment might get added later, but this mode is low-priority for now.
Check the aptly-named CLI scripts in the project for shareable tokenizations.)