Struct chat_flame_backend::llm::token_generator::TokenGenerator

pub struct TokenGenerator { /* private fields */ }

A token generator that generates tokens based on provided parameters, model processor, and sampler.

This struct implements the TokenGeneratorTrait and provides functionality to generate tokens for text generation tasks.

Implementations

impl TokenGenerator

pub fn new( stop_tokens: HashSet<u32>, parameter: GenerateParameter, model: Box<dyn ModelProcessor>, sampler: Box<dyn Sampler> ) -> Self

Creates a new TokenGenerator with the specified parameters.

Arguments

• stop_tokens - A set of token IDs that signal the end of token generation.
• parameter - The parameters to use for token generation.
• model - A model processor to generate logits.
• sampler - A sampler to sample tokens from logits.

Returns

A new instance of TokenGenerator.

Trait Implementations

impl TokenGeneratorTrait for TokenGenerator

fn init(&mut self, prompt_tokens: Vec<u32>) -> Result<()>

Initializes the token generator with a given set of prompt tokens.

fn next(&mut self) -> Result<TokenGeneratorResult>

Retrieves the next token from the generator.

impl Send for TokenGenerator