We have reverse-engineered the Claude 3 tokenizer by inspecting the generation stream. This is the worst (but only!) Claude 3 tokenizer. Check our blog post, code and Twitter thread.