Package index • rtiktoken

All functions

decode_tokens(): Decodes tokens back to text

get_token_count(): Returns the number of tokens in a text

get_tokens(): Converts text to tokens

model_to_tokenizer(): Gets the name of the tokenizer used by a model