Skip to content

Commit

Permalink
added docs explaining the rollback op
Browse files Browse the repository at this point in the history
  • Loading branch information
l3utterfly committed Jan 31, 2024
1 parent d5f4dec commit 7fe612d
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions common/sampling.h
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,10 @@ void llama_sampling_accept(
llama_token id,
bool apply_grammar);

// this performs rollback of the latest sampling operation by "rollback_num" tokens;
// it simply strikes the latest "rollback_num" tokens from the "prev" vector
// in general, the rollback is "imperfect", meaning the "forgotten tokens" which were dropped when the length of "prev" exceeded "n_prev" cannot be recalled after rollback
// however, if `sampling_params.n_prev` >= `sampling_params.penalty_last_n` + `rollback_num`, then it becomes "perfect" rollback
void llama_sampling_rollback(
struct llama_sampling_context * ctx_sampling,
int rollback_num);

0 comments on commit 7fe612d

Please sign in to comment.