Skip to content

Commit

Permalink
final cleanup
Browse files Browse the repository at this point in the history
  • Loading branch information
BenoitDherin committed Apr 17, 2024
1 parent a570cdf commit 0a5670b
Showing 1 changed file with 27 additions and 161 deletions.
188 changes: 27 additions & 161 deletions notebooks/vertex_genai/solutions/vertex_llm_evaluation.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {
"id": "PyQmSRbKA8r-",
"tags": []
Expand All @@ -59,20 +59,12 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": null,
"metadata": {
"id": "oM1iC_MfAts1",
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Your project ID is set to dherin-dev\n"
]
}
],
"outputs": [],
"source": [
"project_id_list = !gcloud config get-value project 2> /dev/null\n",
"PROJECT_ID = project_id_list[0]\n",
Expand Down Expand Up @@ -102,7 +94,7 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": null,
"metadata": {
"id": "NIq7R4HZCfIc",
"tags": []
Expand All @@ -125,7 +117,7 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"metadata": {
"id": "j4KEcQEWROby",
"tags": []
Expand Down Expand Up @@ -182,107 +174,12 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"metadata": {
"id": "R-_ettKRxfxT",
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>document</th>\n",
" <th>response_a</th>\n",
" <th>response_b</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>40159674</td>\n",
" <td>The 33-year-old, capped 81 times by the Republ...</td>\n",
" <td>Whelan joined Aston Villa on a free transfer f...</td>\n",
" <td>- Glenn Whelan, 33, has joined Aston Villa fro...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>36925620</td>\n",
" <td>Paul Frew, MLA for North Antrim, appeared in c...</td>\n",
" <td>- A 15-year-old girl is suing DUP MLA Paul Fre...</td>\n",
" <td>- Paul Frew, MLA for North Antrim, appeared in...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>39729595</td>\n",
" <td>Party leader Jeremy Corbyn said half of the ho...</td>\n",
" <td>Labour says it will build 100,000 new homes a ...</td>\n",
" <td>- Labour leader Jeremy Corbyn announced a hous...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>16788700</td>\n",
" <td>No clear winner emerged between incumbent Jean...</td>\n",
" <td>The African Union summit in Addis Ababa ended ...</td>\n",
" <td>The African Union (AU) summit in Addis Ababa, ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>19389625</td>\n",
" <td>The ban has been called by opposition coalitio...</td>\n",
" <td>Opposition coalition in Togo has called for a ...</td>\n",
" <td>Togo's opposition coalition, Let's Save Togo, ...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id document \\\n",
"0 40159674 The 33-year-old, capped 81 times by the Republ... \n",
"1 36925620 Paul Frew, MLA for North Antrim, appeared in c... \n",
"2 39729595 Party leader Jeremy Corbyn said half of the ho... \n",
"3 16788700 No clear winner emerged between incumbent Jean... \n",
"4 19389625 The ban has been called by opposition coalitio... \n",
"\n",
" response_a \\\n",
"0 Whelan joined Aston Villa on a free transfer f... \n",
"1 - A 15-year-old girl is suing DUP MLA Paul Fre... \n",
"2 Labour says it will build 100,000 new homes a ... \n",
"3 The African Union summit in Addis Ababa ended ... \n",
"4 Opposition coalition in Togo has called for a ... \n",
"\n",
" response_b \n",
"0 - Glenn Whelan, 33, has joined Aston Villa fro... \n",
"1 - Paul Frew, MLA for North Antrim, appeared in... \n",
"2 - Labour leader Jeremy Corbyn announced a hous... \n",
"3 The African Union (AU) summit in Addis Ababa, ... \n",
"4 Togo's opposition coalition, Let's Save Togo, ... "
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"evaluation_gemini_df = pd.read_json(EVALUATION_FILE_URI, lines=True)\n",
"evaluation_gemini_df.head()"
Expand All @@ -300,7 +197,7 @@
"\n",
"* `evaluation_dataset` to indicate where the evaluation dataset location. In this case, it is the JSONL Cloud bucket URI.\n",
"\n",
"* `id_colums` to distinguish evaluation examples that are unique. Here, as you can imagine, your have `id` and `document` fields.\n",
"* `id_colums` to distinguish evaluation examples that are unique. Here, as you can imagine, your have `id` and `document` fields. These fields will be added in the judgment table generated by AutoSxS.\n",
"\n",
"* `task` to indicate the task type you want to evaluate. It can be `summarization` or `question_answer`. In this case you have `summarization`.\n",
"\n",
Expand All @@ -318,7 +215,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": null,
"metadata": {
"id": "Cp7e-hOmNMhA",
"tags": []
Expand All @@ -331,7 +228,7 @@
"\n",
"parameters = {\n",
" \"evaluation_dataset\": EVALUATION_FILE_URI,\n",
" \"id_columns\": [\"id\"],\n",
" \"id_columns\": [\"id\", \"document\"],\n",
" \"task\": \"summarization\",\n",
" \"autorater_prompt_parameters\": {\n",
" \"inference_context\": {\"column\": \"document\"},\n",
Expand All @@ -353,37 +250,12 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": null,
"metadata": {
"id": "AjFHT5ze9m4L",
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Creating PipelineJob\n",
"PipelineJob created. Resource name: projects/115851500182/locations/us-central1/pipelineJobs/autosxs-1713383655691384\n",
"To use this PipelineJob in another session:\n",
"pipeline_job = aiplatform.PipelineJob.get('projects/115851500182/locations/us-central1/pipelineJobs/autosxs-1713383655691384')\n",
"View Pipeline Job:\n",
"https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/autosxs-1713383655691384?project=115851500182\n",
"PipelineJob projects/115851500182/locations/us-central1/pipelineJobs/autosxs-1713383655691384 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n",
"PipelineJob projects/115851500182/locations/us-central1/pipelineJobs/autosxs-1713383655691384 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n",
"PipelineJob projects/115851500182/locations/us-central1/pipelineJobs/autosxs-1713383655691384 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n",
"PipelineJob projects/115851500182/locations/us-central1/pipelineJobs/autosxs-1713383655691384 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n",
"PipelineJob projects/115851500182/locations/us-central1/pipelineJobs/autosxs-1713383655691384 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n",
"PipelineJob projects/115851500182/locations/us-central1/pipelineJobs/autosxs-1713383655691384 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n"
]
}
],
"outputs": [],
"source": [
"job = aiplatform.PipelineJob(\n",
" job_id=display_name,\n",
Expand Down Expand Up @@ -525,7 +397,8 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "w2RISjQSJk9R"
"id": "w2RISjQSJk9R",
"tags": []
},
"outputs": [],
"source": [
Expand All @@ -548,7 +421,9 @@
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def print_aggregated_metrics(scores):\n",
Expand Down Expand Up @@ -597,7 +472,8 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mbfsO2uw9-i5"
"id": "mbfsO2uw9-i5",
"tags": []
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -633,7 +509,7 @@
"\n",
"parameters = {\n",
" \"evaluation_dataset\": HUMAN_EVALUATION_FILE_URI,\n",
" \"id_columns\": [\"id\"],\n",
" \"id_columns\": [\"id\", \"document\"],\n",
" \"task\": \"summarization\",\n",
" \"autorater_prompt_parameters\": {\n",
" \"inference_context\": {\"column\": \"document\"},\n",
Expand All @@ -649,7 +525,8 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KbhIPY-_3SSB"
"id": "KbhIPY-_3SSB",
"tags": []
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -681,7 +558,8 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JLUOJFjA38ja"
"id": "JLUOJFjA38ja",
"tags": []
},
"outputs": [],
"source": [
Expand Down Expand Up @@ -710,7 +588,9 @@
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def print_human_preference_metrics(metrics):\n",
Expand All @@ -725,20 +605,6 @@
"pprint.pprint(human_aligned_metrics)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TpV-iwP9qw9c"
},
"source": [
"## Cleaning up\n",
"\n",
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial. Clicking the __End Lab__ button in your lab instructions will take care of this for you.\n",
"\n",
"Otherwise, you can delete the individual resources you created in this tutorial."
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down

0 comments on commit 0a5670b

Please sign in to comment.