notebooks/W6T_Transformers_Generation.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/gsarti/ik-nlp-tutorials/blob/main/notebooks/W6T_Transformers_Generation.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run in Colab to install local packages\n",
    "!pip install transformers"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural Language Generation with 🤗 Transformers\n",
    "\n",
    "*This notebook is adapted from the [Transformers documentation](https://huggingface.co/docs/transformers/generation_strategies) and the [How to generate](https://huggingface.co/blog/how-to-generate) blogpost.*\n",
    "\n",
    "This week we will learn how to use the 🤗 Transformers library to generate text. In particular, we will see how to customize the `generate` method associated to decoder-only and encoder-decoder models to improve the quality of the generated text by choosing the appropriate parameters and decoding strategies. Since the use of language models has become pervasive for all sort of tasks in a prompt-based setting, understanding notions like *temperature*, *top-k* and *top-p* will be useful even when generating text using company APIs (e.g. [OpenAI](https://beta.openai.com/) and [Cohere](https://cohere.ai/)).\n",
    "\n",
    "The [model.generate](https://huggingface.co/docs/transformers/en/main_classes/text_generation#transformers.GenerationMixin.generate) function can be used with any generative model available in the 🤗 Transformers library. Concretely, it is a utility taking source tokens as inputs and iteratively generating a sequence of target tokens by repeatedly calling the `model.forward()` method from PyTorch, obtaining the next predicted token according to the current **decoding strategy** and adding it to the model inputs before the next `forward` call."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoding strategies\n",
    "\n",
    "Decoding strategies are a fundamental component of generation pipelines, since they tell the language model how to select tokens from the predicted distribution at every generation step. Multiple decoding strategies are available in the 🤗 Transformers library. We will noew briefly see how some of them work.\n",
    "\n",
    "### Greedy search\n",
    "\n",
    "Greedy search simply selects the word with the highest probability as its next word: $w_t = argmax_{w}P(w | w_{1:t-1})$ at each timestep $t$. The following sketch shows an example of greedy search.\n",
    "\n",
    "<img src=\"https://hf.co/blog/assets/02_how-to-generate/greedy_search.png\" alt=\"greedy search\" style=\"margin: auto; display: block;\">\n",
    "\n",
    "Starting from the word `\"The\"`, the algorithm greedily chooses the next word of highest probability `\"nice\"` and so on, so that the final generated word sequence is `(\"The\",\"nice\",\"woman\")` having an overall probability of `0.5 × 0.4 = 0.2`.\n",
    "\n",
    "In the following we will generate word sequences using GPT-2 on the context `(\"I\",\"enjoy\",\"walking\",\"with\",\"my\",\"cute\",\"dog\")`. Let's see how greedy search can be used in transformers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/lsickert/Repositories/ik-nlp-tutorials/.venv/lib/python3.10/site-packages/transformers/generation/utils.py:1186: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with my dog. I'm not sure if I'll ever be able to walk with my dog.\n",
      "\n",
      "I'm not sure if I'll\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
    "\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n",
    "# add the EOS token as PAD token to avoid warnings\n",
    "model = AutoModelForCausalLM.from_pretrained(\"gpt2\", pad_token_id=tokenizer.eos_token_id)\n",
    "\n",
    "# encode context the generation is conditioned on\n",
    "input_ids = tokenizer.encode('I enjoy walking with my cute dog', return_tensors='pt')\n",
    "\n",
    "# generate text until the output length (which includes the context length) reaches 50\n",
    "greedy_output = model.generate(input_ids, max_length=50)\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode(greedy_output[0], skip_special_tokens=True))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright! We have generated our first short text with GPT2 😊. The generated words following the context are reasonable, but the model quickly starts repeating itself! This is a very common problem in language generation in general and seems to be even more so in greedy and beam search (see [Vijayakumar et al. 2016](https://arxiv.org/abs/1610.02424), [Shao et al. 2017](https://arxiv.org/abs/1701.03185))\n",
    "\n",
    "The major drawback of greedy search though is that it misses high probability words hidden behind a low probability word as can be seen in our sketch above:\n",
    "\n",
    "The word `\"has\"` with its high conditional probability of `0.9` is hidden behind the word `\"dog\"`, which has only the second-highest conditional probability, so that greedy search misses the word sequence `\"The\",\"dog\",\"has\"`.\n",
    "\n",
    "Thankfully, we have beam search to alleviate this problem!\n",
    "\n",
    "### Beam search\n",
    "\n",
    "Beam search reduces the risk of missing hidden high probability word sequences by keeping the most likely num_beams of hypotheses at each time step and eventually choosing the hypothesis that has the overall highest probability. Let's illustrate with num_beams=2:\n",
    "\n",
    "<img src=\"https://hf.co/blog/assets/02_how-to-generate/beam_search.png\" alt=\"beam search\" style=\"margin: auto; display: block;\">\n",
    "\n",
    "Beam search will always find an output sequence with higher probability than greedy search, but is not guaranteed to find the most likely output.\n",
    "\n",
    "Let's see how beam search can be used in transformers. We set `num_beams > 1` and `early_stopping=True` so that generation is finished when all beam hypotheses reached the `EOS` token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.\n",
      "\n",
      "I'm not sure if I'll ever be able to walk with him again. I'm not sure if I'll\n"
     ]
    }
   ],
   "source": [
    "# activate beam search and early_stopping\n",
    "beam_output = model.generate(\n",
    "    input_ids, \n",
    "    max_length=50, \n",
    "    num_beams=5, \n",
    "    early_stopping=True\n",
    ")\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode(beam_output[0], skip_special_tokens=True))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the result is arguably more fluent, the output still includes repetitions of the same word sequences. A simple remedy is to introduce n-gram penalties. The most common n-grams penalty makes sure that no n-gram appears twice by manually setting the probability of next words that could create an already seen n-gram to 0. Let's try it out by setting `no_repeat_ngram_size = 2` so that no 2-gram appears twice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.\n",
      "\n",
      "I've been thinking about this for a while now, and I think it's time for me to take a break\n"
     ]
    }
   ],
   "source": [
    "# set no_repeat_ngram_size to 2\n",
    "beam_output = model.generate(\n",
    "    input_ids, \n",
    "    max_length=50, \n",
    "    num_beams=5, \n",
    "    no_repeat_ngram_size=2, \n",
    "    early_stopping=True\n",
    ")\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode(beam_output[0], skip_special_tokens=True))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that repetition does not appear anymore. Nevertheless, n-gram penalties have to be used with care. An article generated about the city New York should not use a 2-gram penalty or otherwise, the name of the city could appear only once in the whole text!\n",
    "\n",
    "If we want to examine the generations produced by beam search in more detail, we can set `num_return_sequences > 1` to get the `num_return_sequences` best hypotheses. Importantly, `num_return_sequences` needs to be smaller or equal than `num_beams`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "0: I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.\n",
      "\n",
      "I've been thinking about this for a while now, and I think it's time for me to take a break\n",
      "1: I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.\n",
      "\n",
      "I've been thinking about this for a while now, and I think it's time for me to get back to\n",
      "2: I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with her again.\n",
      "\n",
      "I've been thinking about this for a while now, and I think it's time for me to take a break\n",
      "3: I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with her again.\n",
      "\n",
      "I've been thinking about this for a while now, and I think it's time for me to get back to\n",
      "4: I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.\n",
      "\n",
      "I've been thinking about this for a while now, and I think it's time for me to take a step\n"
     ]
    }
   ],
   "source": [
    "# set return_num_sequences > 1\n",
    "beam_outputs = model.generate(\n",
    "    input_ids, \n",
    "    max_length=50, \n",
    "    num_beams=5, \n",
    "    no_repeat_ngram_size=2, \n",
    "    num_return_sequences=5, \n",
    "    early_stopping=True\n",
    ")\n",
    "\n",
    "# now we have 3 output sequences\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "for i, beam_output in enumerate(beam_outputs):\n",
    "  print(\"{}: {}\".format(i, tokenizer.decode(beam_output, skip_special_tokens=True)))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen, the five beam hypotheses are only marginally different to each other - which should not be too surprising when using only 5 beams.\n",
    "\n",
    "In open-ended generation, a couple of reasons have recently been brought forward why beam search might not be the best possible option:\n",
    "\n",
    "- Beam search can work very well in tasks where the length of the desired generation is more or less predictable as in machine translation or summarization. But this is not the case for open-ended generation where the desired output length can vary greatly, e.g. dialog and story generation.\n",
    "- We have seen that beam search heavily suffers from repetitive generation. This is especially hard to control with n-gram- or other penalties in story generation since finding a good trade-off between forced \"no-repetition\" and repeating cycles of identical n-grams requires a lot of finetuning.\n",
    "- As argued in [Holtzman et al. (2019)](https://arxiv.org/abs/1904.09751), high quality human language does not follow a distribution of high probability next words. In other words, as humans, we want generated text to surprise us and not to be boring/predictable. The authors show this nicely by plotting the probability, a model would give to human text vs. what beam search does.\n",
    "\n",
    "<img src=\"https://blog.fastforwardlabs.com/images/2019/05/Screen_Shot_2019_05_08_at_3_06_36_PM-1557342561886.png\" alt=\"beam search vs human\" style=\"margin: auto; display: block;\">\n",
    "\n",
    "We will now see how we can introduce some randomness to improve the selection process.\n",
    "\n",
    "### Sampling\n",
    "\n",
    "In its most basic form, sampling means randomly picking the next word $w_t$ according to its conditional probability distribution:\n",
    "\n",
    "$$w_t∼P(w∣w_{1:t−1})$$\n",
    "\n",
    "Taking the example from above, the following graphic visualizes language generation when sampling.\n",
    "\n",
    "<img src=\"https://huggingface.co/blog/assets/02_how-to-generate/sampling_search.png\" alt=\"sampling\" style=\"margin: auto; display: block;\">\n",
    "\n",
    "In transformers, we set `do_sample=True` and deactivate Top-K sampling (more on this later) via `top_k=0`. In the following, we will fix `random_seed=0` for illustration purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "I enjoy walking with my cute dog, Daddy,\" a barter in the popular underground Twitter service Tweston tells Gladstone. \"Like it or not, I've always wanted to dogfeed and cooperate with several others. Being an American cost me quite\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "# Needed to reproduce results\n",
    "torch.manual_seed(42)\n",
    "\n",
    "# activate sampling and deactivate top_k by setting top_k sampling to 0\n",
    "sample_output = model.generate(\n",
    "    input_ids, \n",
    "    do_sample=True, \n",
    "    max_length=50, \n",
    "    top_k=0\n",
    ")\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode(sample_output[0], skip_special_tokens=True))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that when sampling, the model is at risk of generating incoherent gibberish. A trick is to make the distribution $P(w∣w_{1:t−1})$ sharper (increasing the likelihood of high probability words and decreasing the likelihood of low probability words) by lowering the **temperature** of the softmax. Softmax with temperature corresponds to:\n",
    "\n",
    "$$p_i = \\frac{\\exp(x_i/\\tau)}{\\sum_{j=1}^{N}\\exp(x_j/\\tau)}$$\n",
    "\n",
    "$\\tau$ is the temperature, and it is used to control the softness of the probability distribution. When $\\tau$ gets lower, the biggest values get more probability, and when it gets larger, the probability will be split more evenly on different elements. Consider the extreme cases where $\\tau$ approaches zero, the probability for the largest element will approach 1 (i.e. sampling converges to greedy decoding), while when $\\tau$ approaches infinity, the probability for each element will be the same.\n",
    "\n",
    "Let's see how we can cool down the distribution in the library by setting `temperature=0.7`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "I enjoy walking with my cute dog, and I was delighted to have him.\"\n",
      "\n",
      "The dog, also known as Bear, was born in March 2010. He was adopted from the Thomas and Catherine Salisbury Animal Shelter in Parkville, S.\n"
     ]
    }
   ],
   "source": [
    "# Needed to reproduce results\n",
    "torch.manual_seed(42)\n",
    "\n",
    "# use temperature to decrease the sensitivity to low probability candidates\n",
    "sample_output = model.generate(\n",
    "    input_ids, \n",
    "    do_sample=True, \n",
    "    max_length=50, \n",
    "    top_k=0, \n",
    "    temperature=0.7\n",
    ")\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode(sample_output[0], skip_special_tokens=True))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Top-K and Top-p sampling\n",
    "\n",
    "Top-K and Top-p sampling are two methods for sampling the next word that belong to the larger class of techniques that go under the umbrella term \"diverse beam search\" ([Fan et al. 2018](https://arxiv.org/abs/1805.04833)).\n",
    "\n",
    "In Top-K sampling, the K most likely next words are filtered and the probability mass is redistributed among only those K next words. GPT2 adopted this sampling scheme, which was one of the reasons for its success in story generation. We extend the range of words used for both sampling steps in the example above from 3 words to 10 words to better illustrate Top-K sampling.\n",
    "\n",
    "<img src=\"https://huggingface.co/blog/assets/02_how-to-generate/top_k_sampling.png\" alt=\"top-k sampling\" style=\"margin: auto; display: block;\" style=\"width:50%\">\n",
    "\n",
    "Let's see how we can use Top-K sampling in the library by setting `top_k=50`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "I enjoy walking with my cute dog, which is a little unusual in this part of our family. He's a friendly, calm kind of dog, and I've always wanted to have him around, and I always wanted to go with him when he\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "# set top_k to 50\n",
    "sample_output = model.generate(\n",
    "    input_ids, \n",
    "    do_sample=True, \n",
    "    max_length=50, \n",
    "    top_k=50\n",
    ")\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode(sample_output[0], skip_special_tokens=True))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the generated text looks good, one concern though with Top-K sampling is that it does not dynamically adapt the number of words that are filtered from the next word probability distribution. This can be problematic as some words might be sampled from a very sharp distribution, whereas others from a much more flat distribution. **Top-p sampling** (also called **nucleus** sampling) addresses this issue by filtering the next words such that the cumulative probability mass of the filtered words is greater than or equal to the selected portion of the probability mass of the next word.\n",
    "\n",
    "<img src=\"https://huggingface.co/blog/assets/02_how-to-generate/top_p_sampling.png\" alt=\"top-p sampling\" style=\"margin: auto; display: block;\"  style=\"width:50%\">\n",
    "\n",
    "In the example above, having set $p=0.92$, Top-p sampling picks the minimum number of words to exceed together 92% of the probability mass​. In the first example, this included the 9 most likely words, whereas it only has to pick the top 3 words in the second example to exceed 92%. It can be seen that it keeps a wide range of words where the next word is arguably less predictable, e.g. `P(w∣\"The”)`, and only a few words when the next word seems more predictable, e.g. `P(w∣\"The\",\"car\")`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "I enjoy walking with my cute dog, which is a little weird because I'm so small and he's like a little bit smaller than me. He's also kind of like a dog's friend, and I think that's why he's such a\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "# deactivate top_k sampling and sample only from 92% most likely words\n",
    "sample_output = model.generate(\n",
    "    input_ids, \n",
    "    do_sample=True, \n",
    "    max_length=50, \n",
    "    top_p=0.4, \n",
    "    top_k=0\n",
    ")\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode(sample_output[0], skip_special_tokens=True))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Contrastive search\n",
    "\n",
    "The contrastive search decoding strategy was proposed in the 2022 paper [A Contrastive Framework for Neural Text Generation](https://arxiv.org/abs/2202.06417). It demonstrates superior results for generating non-repetitive yet coherent long outputs. The method simply adds a degeneration penalty term to the regular top-K sampling process. The degeneration penalty is defined as the maximum cosine similarity between the token representation $x_t$ at generation step $t$​, and that of all tokens in the context $x_{<t}$​. Intuitively, a larger degeneration penalty means it is more similar (in the representation space) to the context, therefore more likely leading to the problem of model degeneration. The hyperparameter $\\alpha$ regulates the importance of this penalty term. When $\\alpha=0$, contrastive search degenerates to regular greedy search. More details can be found in [this blog post](https://huggingface.co/blog/introducing-csearch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "I enjoy walking with my cute dog, but I don't like to be alone.\n",
      "\n",
      "I'm going to be a little more adventurous with my dog, but I'm not going to be afraid to go out and play with him.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# deactivate top_k sampling and sample only from 92% most likely words\n",
    "sample_output = model.generate(\n",
    "    input_ids, \n",
    "    max_length=50,\n",
    "    penalty_alpha=0.6,\n",
    "    top_k=4,\n",
    ")\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode(sample_output[0], skip_special_tokens=True))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other Generation Parameters\n",
    "\n",
    "Besides the parameters we have seen so far, there are a couple of other parameters that are worth mentioning:\n",
    "\n",
    "- `max_length`: The maximum length of the sequence to be generated. If the model reaches the maximum length, it will stop there. Importantly, this parameter also accounts for the length of the original input (also called **prompt**) for language generation tasks that involve continuation of a given input sequence. The new parameter `max_new_tokens` is preferable in this context, since it only accounts for the number of tokens to be generated. `min_length` and `min_new_tokens` also exist.\n",
    "\n",
    "- `max_time`: The maximum time to wait for the generation to complete. This is useful when the generation is conducted in time-bound settings.\n",
    "\n",
    "- `repetition_penalty`: This parameter is used to penalize words that have already been generated. This is useful when we want to avoid repetition in the generation. The parameter is set to 1.0 by default, which means that no penalty is applied. To encourage the model to not generate the same word twice, we can set the parameter to a value greater than 1.0. To encourage the model to generate the same word twice, we can set the parameter to a value less than 1.0.\n",
    "\n",
    "- `length_penalty`: This parameter is used to penalize long generated sequences. This is useful when we want to encourage the model to generate shorter sequences. The parameter is set to 1.0 by default, which means that no penalty is applied. To encourage the model to generate shorter sequences, we can set the parameter to a value less than 1.0. To encourage the model to generate longer sequences, we can set the parameter to a value greater than 1.0.\n",
    "\n",
    "- `bad_words_ids`: Ids of words that should not appear in the generated text. This is useful when we want to filter out profanity, for example. The parameter is set to `None` by default, which means that no bad words are filtered out. To obtain ids from a tokenizer, you can use `tokenizer(bad_words, add_prefix_space=True, add_special_tokens=False).input_ids`.\n",
    "\n",
    "The full list of parameters is available in [the documentation](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationConfig).\n",
    "\n",
    "Since transformers 4.26, the set of parameters applied during the generation process has been centralized in a `generation_config` that can be easily inspected. Missing parameters will assume their default values in the library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GenerationConfig {\n",
       "  \"_from_model_config\": true,\n",
       "  \"bos_token_id\": 50256,\n",
       "  \"eos_token_id\": 50256,\n",
       "  \"pad_token_id\": 50256,\n",
       "  \"transformers_version\": \"4.26.0\"\n",
       "}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.generation_config"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to tokenizers, a custom `GenerationConfig` can be defined and saved alongside your model weights, locally or on the 🤗 Hub. This allows to specify configurations that work best for a given model, and allow model users to adopt them out-of-the-box:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I enjoy walking with my cute dog because as far as how his behavior and body can go I am quite a bit different. He is also very friendly and loving.\"\\n\\n\"That\\'s true. Sometimes I make him sit on our side of the couch with his hand on the back']\n"
     ]
    }
   ],
   "source": [
    "from transformers import GenerationConfig\n",
    "\n",
    "generation_config = GenerationConfig(\n",
    "    max_new_tokens=50, do_sample=True, top_k=50, eos_token_id=model.config.eos_token_id\n",
    ")\n",
    "generation_config.save_pretrained(\"my_model\")\n",
    "\n",
    "outputs = model.generate(input_ids, generation_config=generation_config)\n",
    "print(tokenizer.batch_decode(outputs, skip_special_tokens=True))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transition Scores\n",
    "\n",
    "The `compute_transition_scores` function is a utility associated to generation models to extract log probabilities of generated tokens at each step. These can be used to understand which steps in the generation process were deemed more or less likely by the model, and possibly identify problematic outputs. The procedure requires setting the `output_scores=True` flag in the `generate` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GreedySearchDecoderOnlyOutput(sequences=tensor([[8888,  318,  262, 1110,  618,  356,  460]]), scores=(tensor([[-122.8355, -122.5403, -127.6362,  ..., -133.4905, -131.9769,\n",
       "         -125.4615]]), tensor([[-113.1373, -111.5629, -114.7529,  ..., -117.6532, -119.1982,\n",
       "         -111.6793]]), tensor([[ -93.0146,  -94.7014, -102.5397,  ..., -106.0070, -106.3985,\n",
       "          -97.7504]]), tensor([[ -96.4015,  -96.7880, -101.3824,  ..., -102.0652, -103.4673,\n",
       "          -97.6960]]), tensor([[-136.1203, -136.8161, -141.1829,  ..., -147.3172, -140.2046,\n",
       "         -137.6747]])), attentions=None, hidden_states=None)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs = tokenizer([\"Today is\"], return_tensors=\"pt\")\n",
    "outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True)\n",
    "outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Today is the day when we can\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.decode(outputs.sequences[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   262 |  the     | -1.414 | 24.33%\n",
      "|  1110 |  day     | -2.609 | 7.36%\n",
      "|   618 |  when    | -2.010 | 13.41%\n",
      "|   356 |  we      | -1.859 | 15.58%\n",
      "|   460 |  can     | -2.508 | 8.14%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/lsickert/Repositories/ik-nlp-tutorials/.venv/lib/python3.10/site-packages/transformers/generation/utils.py:1029: UserWarning: Use of index_put_ on expanded tensors is deprecated. Please clone() the tensor before performing this operation. This also applies to advanced indexing e.g. tensor[indices] = tensor (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/TensorAdvancedIndexing.cpp:634.)\n",
      "  beam_indices[beam_indices_mask] = 0\n",
      "/Users/lsickert/Repositories/ik-nlp-tutorials/.venv/lib/python3.10/site-packages/transformers/generation/utils.py:1029: UserWarning: Use of masked_fill_ on expanded tensors is deprecated. Please clone() the tensor before performing this operation. This also applies to advanced indexing e.g. tensor[mask] = scalar (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1654.)\n",
      "  beam_indices[beam_indices_mask] = 0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "transition_scores = model.compute_transition_scores(outputs.sequences, outputs.scores, normalize_logits=True)\n",
    "input_length = inputs.input_ids.shape[1]\n",
    "generated_tokens = outputs.sequences[:, input_length:]\n",
    "for tok, score in zip(generated_tokens[0], transition_scores[0]):\n",
    "    print(f\"| {tok:5d} | {tokenizer.decode(tok):8s} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This concludes our overview of the `generate` function in 🤗 Transformers. Among the things that were not mentioned in this summary, we invite you to have a look at the blog post [Guiding Text Generation with Constrained Beam Search](https://huggingface.co/blog/constrained-beam-search) to learn how to impose constraints on the generation process, e.g. to specify that the German formal form `Wie alt sind Sie?` should be preferred over the informal form `Wie alt bist du?` when translating `How old are you?` from English to German."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ik-nlp-tutorials-fnrzgbw7-py3.9",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "926148ef506b27e4c5320622dec51791a789aa0e853a66d407eab8b330b15f1c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}