Customizable bad_token_ids policies

[dxoigmn]

llmart has the capability of banning "bad" tokens from the adversarial optimization.

Right now bad_token_ids implements a static policy for what is considered a "bad" token (non-printability, ascii-only):

LLMart/src/llmart/tokenizer.py

Lines 428 to 444 in c7bbef3

	def bad_token_ids(self) -> torch.Tensor:
	added_tokens = self.added_tokens_encoder.keys()
	tokens = [
	self.convert_tokens_to_string([token])
	for token in self.convert_ids_to_tokens(list(range(self.__vocab_size)))
	]
	printable_tokens = torch.tensor(
	[
	token.isprintable()
	and token.isascii()
	and token not in added_tokens
	and len(token.strip()) > 0
	for token in tokens
	],
	)
	return torch.where(~printable_tokens)[0]

Being able to add configurable policies would help with non-ascii languages. Additionally, being able to ban a set of tokens would also be beneficial.