getting-started-with-igel-lora-finetuned.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/snipaid-nlg/igel-lora-finetune-news-snippets/blob/main/getting-started-with-igel-lora-finetuned.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Getting started with Snip-IGEL 📰🦔\n",
        "This notebook will guide you through the basics of using Snip-IGEL — an IGEL model with finetuned LoRA adapters for news snippet generation in German language.\n",
        "\n",
        "So, let's get started! First, let's check the GPU setup 🔍"
      ],
      "metadata": {
        "id": "7NNnBXGr3PS_"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y9cwNNVNkXvl"
      },
      "outputs": [],
      "source": [
        "!nvidia-smi"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Step 1: Install required packages 📦"
      ],
      "metadata": {
        "id": "CjmwiR_j4LH6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q transformers accelerate bitsandbytes peft"
      ],
      "metadata": {
        "id": "6yUqvf05ks3j"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Step 2: Load base model, LoRA adapter and tokenizer 🚀\n",
        "\n",
        "Loading the adapter model for inference requires a GPU with ~ 8 GB of VRAM."
      ],
      "metadata": {
        "id": "Yv3jIKXYMfw2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "from peft import PeftModel\n",
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "\n",
        "#@markdown Select one of the following LoRA adapters:\n",
        "\n",
        "adapter = 'snip-igel-500-v2' #@param [\"snip-igel-10\", \"snip-igel-100\", \"snip-igel-500\", \"snip-igel-500-v2\"]\n",
        "\n",
        "#@markdown - **snip-igel-500-v2:** The most advanced adapter capable of title, teaser, keyword, summary, serp and tweet generation from news text.\n",
        "#@markdown - **snip-igel-500:** An adapter with decent capability for title, teaser, keyword and tweet generation.  \n",
        "#@markdown - **snip-igel-100:** Same as snip-igel-500 but fine-tuned with less data.\n",
        "#@markdown - **snip-igel-10:** Same as snip-igel-10 but finetuned with the least amount of data.\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"philschmid/instruct-igel-001\")\n",
        "model = AutoModelForCausalLM.from_pretrained(\n",
        "    \"malteos/bloom-6b4-clp-german\",\n",
        "    torch_dtype=torch.float16,\n",
        "    load_in_8bit=True,\n",
        "    device_map=\"auto\",\n",
        ")\n",
        "model = PeftModel.from_pretrained(model, f\"snipaid/{adapter}/\")"
      ],
      "metadata": {
        "id": "BqV4dUBdk05M"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Step 3: Setup config and utility functions for generation with the Snip-IGEL adapter model ⚙️"
      ],
      "metadata": {
        "id": "G9bviZ308LAD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import GenerationConfig\n",
        "\n",
        "generation_config = GenerationConfig(\n",
        "    top_p=0.9,\n",
        "    top_k=0,\n",
        "    temperature=1,\n",
        "    do_sample=True,\n",
        "    early_stopping=True,\n",
        "    length_penalty=1,\n",
        "    eos_token_id=tokenizer.eos_token_id,\n",
        "    pad_token_id=tokenizer.eos_token_id,\n",
        ")\n",
        "\n",
        "def generate_prompt(instruction, input=None):\n",
        "  if input:\n",
        "    return f\"\"\"Nachfolgend ist eine Anweisung, die eine Aufgabe beschreibt, zusammen mit einer Eingabe, die weiteren Kontext liefert. Schreiben Sie eine Antwort, die die Aufgabe angemessen erfüllt.\n",
        "### Anweisung:\n",
        "{instruction}\n",
        "### Eingabe:\n",
        "{input}\n",
        "### Antwort:\"\"\"\n",
        "  else:\n",
        "    return f\"\"\"Nachfolgend ist eine Anweisung, die eine Aufgabe beschreibt, zusammen mit einer Eingabe, die weiteren Kontext liefert. Schreiben Sie eine Antwort, die die Aufgabe angemessen erfüllt.\n",
        "### Anweisung:\n",
        "{instruction}\n",
        "### Antwort:\"\"\"\n",
        "\n",
        "def evaluate(instruction, input=None):\n",
        "    prompt = generate_prompt(instruction, input)\n",
        "    inputs = tokenizer(prompt, return_tensors=\"pt\")\n",
        "    input_ids = inputs[\"input_ids\"].cuda()\n",
        "    generation_output = model.generate(\n",
        "        input_ids=input_ids,\n",
        "        generation_config=generation_config,\n",
        "        return_dict_in_generate=True,\n",
        "        output_scores=True,\n",
        "        max_new_tokens=256\n",
        "    )\n",
        "    for s in generation_output.sequences:\n",
        "        output = tokenizer.decode(s).split(\"### Antwort:\")[1].strip()\n",
        "        return output.replace(\"<|endoftext|>\", \"\")"
      ],
      "metadata": {
        "id": "TBkEnsVenavG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "7S9CQyCRneIY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Step 4: Get some news article text 📰"
      ],
      "metadata": {
        "id": "suWRf2CP8tHj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q newspaper3k"
      ],
      "metadata": {
        "id": "hZQqncqP8pVp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown Copy and paste an url to a news article to extract its text below.\n",
        "\n",
        "from newspaper import Article\n",
        "\n",
        "url = 'https://t3n.de/news/hippocamera-app-gegen-demenz-1535827/' #@param {type:\"string\"}\n",
        "article = Article(url, browser_user_agent=\"Googlebot-News\")\n",
        "article.download()\n",
        "article.parse()\n",
        "article.text"
      ],
      "metadata": {
        "id": "ZG9E-NmU8j8v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "gM6YzwZuAJQZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Step 5: Generate snippets 💬\n",
        "\n",
        "Below are some exaple promts:\n",
        "\n",
        "- Welche Überschrift passt am besten zum Inhalt des Artikels?\n",
        "- Generiere einen Teaser zu folgendem Artikel.\n",
        "- Fasse den folgenden Artikel in wenigen Sätzen zusammen für einen Newsletter.\n",
        "- Nenne die zehn wichtigsten Keywords aus dem Text.\n",
        "- Generiere eine SERP (Title-Tag und Meta-Description) für den folgenden Inhalt.\n",
        "- Schreibe einen Tweet über den Artikel.\n",
        "\n",
        "Feel free to experiment with some other prompts as well.  \n",
        "The model is capable to answer more than just those."
      ],
      "metadata": {
        "id": "Uw5I3AzT8zh6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "evaluate(input(\"Anweisung:\"), article.text[:1000])"
      ],
      "metadata": {
        "id": "upOB2AQJSW9-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "evaluate(input(\"Anweisung: \"))"
      ],
      "metadata": {
        "id": "Mw_ZylDirMAN"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}