Function chat_flame_backend::llm::loader::create_tokenizer

source · 
pub fn create_tokenizer(model: Models) -> Result<Tokenizer, Box<dyn Error>>
Expand description

Creates and loads a tokenizer from the Hugging Face Hub.

Arguments

  • model - The model enum specifying the tokenizer to load.

Returns

Returns a result containing the Tokenizer, or an error if loading fails.