Speed up a byte tokenizer and estimate token counts

You are given a slow but correct reference tokenizer over bytes. Explain precisely what it does, then write a faster version that returns identical output. Finally, estimate the token count of a large input within a fixed sampling budget without tokenizing all of it.