tokenize
Tokenizes query: lowercases, strips diacritics (NFKD), and splits on any run of whitespace or punctuation. Empty tokens are dropped. Result preserves left-to-right order for stability, but downstream partitioning is token-order independent.
Tokenizes query: lowercases, strips diacritics (NFKD), and splits on any run of whitespace or punctuation. Empty tokens are dropped. Result preserves left-to-right order for stability, but downstream partitioning is token-order independent.