DP02_02a_Introduction_to_TAP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to the Table Access Protocol (TAP) Service\n",
    "\n",
    "<img align=\"left\" src = https://project.lsst.org/sites/default/files/Rubin-O-Logo_0.png width=250 style=\"padding: 10px\"> \n",
    "<br>\n",
    "Contact authors: Leanne Guy, Melissa Graham <br>\n",
    "Last verified to run: 2024-12-17 <br>\n",
    "LSST Science Pipelines version: Weekly 2024_50 <br>\n",
    "Container Size: medium <br>\n",
    "Targeted learning level: beginner <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Description:** Explore the DP0.2 catalogs with the TAP service."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Skills:** Use the TAP service. Make simple ADQL queries. Visualize retrieved datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**LSST Data Products:** Catalog schema. Object table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Packages:** lsst.rsp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Credit:**\n",
    "Originally developed by Leanne Guy in the context of the Rubin DP0.1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Get Support:**\n",
    "Find DP0-related documentation and resources at <a href=\"https://dp0.lsst.io\">dp0.lsst.io</a>. Questions are welcome as new topics in the <a href=\"https://community.lsst.org/c/support/dp0\">Support - Data Preview 0 Category</a> of the Rubin Community Forum. Rubin staff will respond to all questions posted there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 1. Introduction\n",
    "\n",
    "This notebook provides a beginner-level demonstration of how to use\n",
    "Astronomy Data Query Language (ADQL) to access DP0.2 catalog data via\n",
    "Rubin's Table Access Protocol (TAP) service.\n",
    "\n",
    "The [documentation for Data Preview 0.2](https://dp0-2.lsst.io/) includes definitions\n",
    "of the data products, descriptions of catalog contents, and ADQL recipes.\n",
    "The RSP's Portal Aspect, with its graphical user interface, is also\n",
    "recommended for exploring the available catalogs and for testing queries.\n",
    "\n",
    "### 1.1. TAP basics\n",
    "\n",
    "TAP provides standardized access to catalog data for discovery, search, and retrieval.\n",
    "Full [documentation for TAP](http://www.ivoa.net/documents/TAP) is provided by the\n",
    "International Virtual Observatory Alliance (IVOA).\n",
    "\n",
    "TAP Glossary\n",
    "* schema - Database terminology for the abstract design that represents the storage of data in a database.\n",
    "* `tap_schema` - The set of tables that describe the data tables and their columns.\n",
    "* table collection - A collection of tables. E.g., `dp02_dc2_catalogs`.\n",
    "* table - A collection of related data held in a table format in a database. E.g., `dp02_dc2_catalogs.Object`.\n",
    "* query - A string formatted in ADQL that selects data from a table, with contraints if desired. \n",
    "* results - The output of the TAP service's search method when a query is passed.\n",
    "\n",
    "### 1.2. ADQL basics\n",
    "\n",
    "The [documentation for ADQL](http://www.ivoa.net/documents/latest/ADQL.html)\n",
    "includes more information about syntax, keywords, operators, functions, and so on.\n",
    "ADQL is similar to SQL (Structured Query Langage).\n",
    "\n",
    "A typical ADQL statement has at least three components:\n",
    "\n",
    "`SELECT <columns> FROM <catalog> WHERE <constraints>`,\n",
    "\n",
    "where\n",
    " * `<columns>` is a comma-separated list of the columns to return,\n",
    " * `<catalog>` is the name of the catalog to retreive data from, and\n",
    " * `<constraints>` imposes a restriction that only rows with column values that meet the constraints are returned.\n",
    "\n",
    "For example, say there is a catalog called \"mysurveydata\" with 5 columns, \"col1\", \"col2\", and so on.\n",
    "The ADQL statement:\n",
    "\n",
    "`SELECT col3, col4, col5 FROM mysurveydata WHERE col1 > 0.5 AND col5 < 10`\n",
    "\n",
    "would \n",
    "return a table that has three columns, and as many rows as meet both of the restrictions in the `WHERE` statement.\n",
    "\n",
    "As the DP0.2 catalogs are _very_ large, the ADQL statement to return an entire\n",
    "table **should not** be used (`SELECT * FROM mysurveydata`).\n",
    "\n",
    "\n",
    "### 1.3. What is not covered in this tutorial?\n",
    "\n",
    "The following intermediate-level TAP functionalities for retrieving catalog data are _not_ demonstrated in this tutorial,\n",
    "but are covered in other tutorials.\n",
    "\n",
    "See DP0.2 tutorial notebook 02b for:\n",
    " * table joins\n",
    " * renaming columns\n",
    " * applying mathematical functions\n",
    " * converting fluxes to magnitudes\n",
    " * querying and visualizing large data sets\n",
    "\n",
    "See DP0.2 tutorial notebook 02c for ObsTAP queries for images (metadata and pixel data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4. Recommendations for TAP queries\n",
    "\n",
    "The following recommendations are all demonstrated in Section 3.\n",
    "\n",
    "#### 1.4.1. Use asynchronous queries\n",
    "In Section 2, synchronous queries are used with `results = service.search(query)`. \n",
    "This is OK for querying schema or small amounts of data, but when more data is being\n",
    "retrieved it is recommended to use asynchronous jobs with `job = service.submit_job(query)`\n",
    "as demonstrated in Section 3.\n",
    "\n",
    "#### 1.4.2. Include coordinate constraints\n",
    "When possible include coordinate constraints (e.g., use a cone search).\n",
    "The TAP-accessible tables are sharded by coordinate (RA, Dec).\n",
    "ADQL query statements that include constraints by coordinate do not require a\n",
    "whole-catalog search, and are typically faster (and can be much faster) than\n",
    "ADQL query statements which only include constraints for other columns.\n",
    "\n",
    "#### 1.4.3. Constrain detect_isPrimary = True\n",
    "As a default, it is recommended to include a constraint of `detect_isPrimary = True` in queries for the \n",
    "`Object`, `Source`, and `ForcedSource` catalogs.\n",
    "This parameter is `True` if a source has no children, is in the inner region of a coadd patch, \n",
    "is in the inner region of a coadd tract, and is not detected in a pseudo-filter.\n",
    "Including this constraint will remove any duplicates (i.e., will not include both a parent and its deblended children).\n",
    "Only remove this constraint in situations where duplicates are desired.\n",
    "\n",
    "#### 1.4.4. If limiting rows, use TOP\n",
    "Use of `TOP` is preferred to `maxrec` because most\n",
    "TAP services map `maxrec` to `SELECT TOP` on the back end anyways.\n",
    "Using `TOP` ensures that queries are copy-pasteable between interfaces\n",
    "(e.g., into the Portal Aspect).\n",
    "\n",
    "#### 1.4.5. If sorting, take care with ORDER BY\n",
    "Combined use of TOP and ORDER BY in ADQL queries can be dangerous: it may take an unexpectedly long\n",
    "time because the database is trying to first sort, and *then* extract the top N elements.\n",
    "It is best to only combine TOP and ORDER BY if the query's WHERE statements significantly cut\n",
    "down the number of objects that would need to be sorted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5. Import packages\n",
    "\n",
    "Import general python packages\n",
    "([pandas](https://pandas.pydata.org/docs/user_guide/index.html),\n",
    "[numpy](https://numpy.org/doc/stable/index.html), and\n",
    "[matplotlib](https://matplotlib.org/)),\n",
    "two classes from [astropy](https://www.astropy.org/),\n",
    "and the Rubin Science Platform (RST) TAP service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import astropy.units as u\n",
    "from astropy.visualization.wcsaxes import SphericalCircle\n",
    "from lsst.rsp import get_tap_service, retrieve_query"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.6. Define functions and parameters\n",
    "\n",
    "Get an instance of the TAP service, and assert that it exists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service = get_tap_service(\"tap\")\n",
    "assert service is not None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the maximum number of rows to display from pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pandas.set_option('display.max_rows', 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Explore the TAP schema \n",
    "\n",
    "This section will use ADQL queries of increasing complexity to both explore \n",
    "the available catalog data and to demonstrate how to use the TAP service and ADQL.\n",
    "\n",
    "### 2.1. Synchronous queries\n",
    "\n",
    "This section uses \"synchronous\" TAP queries.\n",
    "This means that the query is run and the results are retrieved at the same time.\n",
    "No other cells can be run while this is happening.\n",
    "\n",
    "Since the queries in this section are very simple, and are only retrieving schemas (not data),\n",
    "they are very quick and synchronous queries work fine.\n",
    "However, when data is retrieved from catalogs, asynchronous queries should be used (Section 3)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. List all catalogs (schema)\n",
    "\n",
    "Create an ADQL query that selects all columns (*) from the `tap_schema`.\n",
    "The results of the query will be a list of available schemas (in other words, the available catalogs).\n",
    "\n",
    "Store it in a string named `query`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = 'SELECT * FROM tap_schema.schemas'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the query by passing `query` to the TAP service's search method,\n",
    "and store the output in `results`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = service.search(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the type of `results`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(type(results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`TAPResults` is a python class from the\n",
    "[PyVO Data Access](https://pyvo.readthedocs.io/en/latest/dal/index.html) package.\n",
    "The [documentation for TAPResults](https://pyvo.readthedocs.io/en/latest/api/pyvo.dal.TAPResults.html)\n",
    "has more information about the class and its methods.\n",
    "\n",
    "Typically, for convenience it is recommended to convert the results into an\n",
    "[Astropy table](https://docs.astropy.org/en/stable/table/).\n",
    "\n",
    "Rerun the query, but store the results as an astropy table with the `to_table()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "results_table = service.search(query).to_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the type of `results` and see that it is an `astropy` table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(results_table))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The table above shows that the DP0.2 data `schema_name` is `dp02_dc2_catalogs`.\n",
    "\n",
    "\"DC2\" stands for \"Data Challenge 2\" and it was the name of the effort by\n",
    "the Dark Energy Science Collaboration to create the simulated data\n",
    "([The DESC DC2 Simulated Sky Survey](https://arxiv.org/abs/2010.05926)).\n",
    "\n",
    "Although the description says \"Data Preview 0.2 contains the image and catalog products...\",\n",
    "the `dp02_dc2_catalogs` contains only catalog data and image data for DP0.2 is \n",
    "stored in the `ObsCore`, with `schema_name` = `ivoa`. \n",
    "Accessing image data via the TAP service is covered in another tutorial.\n",
    "\n",
    "The above table is quite short, and it is obvious from looking at it \n",
    "that the table collection for DP0.2 is `dp02_dc2_catalogs`.\n",
    "However, there may be times in the future when there are many more\n",
    "schemas available.\n",
    "The following demonstrates a way to search for catalogs by name.\n",
    "\n",
    "Use a `for` loop to look for the presence of `dp02` in the\n",
    "`schema_name` column for every row of the results table,\n",
    "and write the name when there is a match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for name in results_table['schema_name']:\n",
    "    if name.find('dp02') > -1:\n",
    "        print(name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean up by deleting the query and its results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del query, results, results_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3. List all tables in the DP0.2 catalog\n",
    "\n",
    "Create a query that selects all columns from `tap_schema.tables`\n",
    "for the `dp02_dc2_catalogs` schema.\n",
    "Add an ORDER BY statement to return the results sorted by `table_index`.\n",
    "\n",
    "This query will return the names of all tables that are available in the `dp02_dc2_catalogs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT * FROM tap_schema.tables \" \\\n",
    "        \"WHERE tap_schema.tables.schema_name = 'dp02_dc2_catalogs'\" \\\n",
    "        \"ORDER BY table_index ASC\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use a synchronous query and return the results as an astropy table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = service.search(query).to_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the results to see the tables in the DP0.2 schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del query, results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4. List all columns in the Object table\n",
    "\n",
    "For the example, use the first table: the `Object` table (see Section 3).\n",
    "\n",
    "The following query selects four colums from the `tap_schema.columns` table,\n",
    "column names, data types, descriptions, and units,\n",
    "only for the `Object` table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT column_name, datatype, description, unit \" \\\n",
    "        \"FROM tap_schema.columns \" \\\n",
    "        \"WHERE table_name = 'dp02_dc2_catalogs.Object'\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the synchronous query and store results in an astropy table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = service.search(query).to_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, notice that the table is very long (991 rows) and the display has been truncated (the '...' represents skipped rows).\n",
    "There are too many columns to read through.\n",
    "\n",
    "Use a `for` loop to look for the presence of `coord` in the\n",
    "`column_name` column for every row of the results table,\n",
    "and write the name when there is a match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_string = 'coord'\n",
    "for cname in results['column_name']:\n",
    "    if cname.find(search_string) > -1:\n",
    "        print(cname)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeat the above exercise, but for all columns that are related to the *g*-band PSF (point spread function) flux measurements.\n",
    "This flux measurement is appropriate to use for point-like objects, like stars.\n",
    "\n",
    "Notice that there is a flux, and error, and several flag values.\n",
    "Flags indicate various warnings, e.g., if the source was close to the edge of the detector and thus might be missing some flux."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_string = 'g_psfFlux'\n",
    "for cname in results['column_name']:\n",
    "    if cname.find(search_string) > -1:\n",
    "        print(cname)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `datatype` column has a limited set of values.\n",
    "Print the unique data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.unique(results['datatype']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `units` column also has a limited set of values.\n",
    "Print the unique set of units used in the `Object` table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.unique(results['unit']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del query, results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Query the DP0.2 Object table\n",
    "\n",
    "The `Object` table contains measurements for all objects detected\n",
    "in the deep coadd (stacked) images.\n",
    "It includes objects detected with a signal-to-noise ratio of at least five in at least one filter.\n",
    "Detections are deblended, e.g., overlapping galaxies (blended sources in the image), are split into\n",
    "individual objects (referred to as 'parent' and 'child' objects).\n",
    "The measurements of these objects, such as shapes, sizes, and fluxes, are made in all filters.\n",
    "\n",
    "The Object catalog (`dp02_dc2_catalogs.Object`) contains sources detected in the coadded images (also called stacked or combined images)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Asynchronous queries\n",
    "\n",
    "This section uses \"asynchronous\" TAP queries.\n",
    "This means that the query is submitted as a job to the TAP service,\n",
    "and it can run in the background until it completes.\n",
    "The results can then be retrieved right away, or at a later time.\n",
    "Other cells can be run while this is happening.\n",
    "\n",
    "Since the queries in this section are more complex, and retrieving rows of data (not just schema),\n",
    "asynchronous queries are used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Cone search\n",
    "\n",
    "A cone search means a catalog query that returns all objects within a given radius of a sky coordinate.\n",
    "Since a given angular diameter corresponds to a larger physical diameter at larger distances,\n",
    "the volume queried is a cone, not a cylinder.\n",
    "\n",
    "In ADQL, a cone search is executed with `WHERE CONTAINS(POINT(), CIRCLE()) = 1`.\n",
    " * `POINT()` passes the catalog's columns for sky coordinates.\n",
    " * `CIRCLE()` defines the center and radius of the search circle.\n",
    " * `CONTAINS() = 1` constrains the query to only return rows for which the statement \"this circle contains this point\" is \"True\" (=1).\n",
    "\n",
    "In Section 2.4, the columns for sky coordinate in the `Object` catalog\n",
    "were shown to be `coord_ra` and `coord_dec`, with units of degrees. \n",
    "The approximate center of the DP0.2 area is RA, Dec = 62.0, -37.0 (decimal degrees).\n",
    "A conservative radius that ensures a quick query with a limited number of rows returned in 0.01 degrees (36 arcsec).\n",
    "\n",
    "The following query would return the `coord_ra` and `coord_dec` columns for all rows of the `Object` table\n",
    "which are within 0.05 degrees of RA, Dec = 62.0, -37.0, and for which the `detect_isPrimary` column is \"True\" (as recommended above).\n",
    "The `'ICRS'` stands for \"International Celestial Reference System\", the reference frame for the coordinates.\n",
    "\n",
    "```\n",
    "SELECT coord_ra, coord_dec FROM dp02_dc2_catalogs.Object \n",
    "WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), CIRCLE('ICRS', 62, -37, 0.05\")) = 1 \n",
    "AND detect_isPrimary = 1\n",
    "```\n",
    "\n",
    "Define the center coordinates and the radius.\n",
    "TAP queries are created as strings, so define `str_center_coords` and `str_radius` as strings\n",
    "which can be inserted into the query string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "center_ra = 62\n",
    "center_dec = -37\n",
    "radius = 0.01\n",
    "\n",
    "str_center_coords = str(center_ra) + \", \" + str(center_dec)\n",
    "str_radius = str(radius)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the query string, and print it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT coord_ra, coord_dec \"\\\n",
    "        \"FROM dp02_dc2_catalogs.Object \"\\\n",
    "        \"WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), \"\\\n",
    "        \"CIRCLE('ICRS', \" + str_center_coords + \", \" + str_radius + \")) = 1 \"\\\n",
    "        \"AND detect_isPrimary = 1\"\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Submit the `query` as an asynchronous job to the TAP service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = service.submit_job(query)\n",
    "print('Job URL is', job.url)\n",
    "print('Job phase is', job.phase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for the job status to be either \"COMPLETED\" or \"ERROR\".\n",
    "\n",
    "This is not mandatory, but is recommended; if it is not run, there will be no other\n",
    "automatic notification that the job is no longer running.\n",
    "However, note that if this cell is executed then no other cells can be executed while the job runs in the background."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job.wait(phases=['COMPLETED', 'ERROR'])\n",
    "print('Job phase is', job.phase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional:** If the status returned was \"ERROR\", uncomment and execute this cell to print the error messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# job.raise_if_error()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the status returned was \"COMPLETED\", execute this cell to retrieve the query results as an `astropy` table\n",
    "and print the length of the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = job.fetch_result().to_table()\n",
    "print(len(results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-10T18:08:45.946400Z",
     "iopub.status.busy": "2024-07-10T18:08:45.945717Z",
     "iopub.status.idle": "2024-07-10T18:08:46.556444Z",
     "shell.execute_reply": "2024-07-10T18:08:46.555565Z",
     "shell.execute_reply.started": "2024-07-10T18:08:45.946377Z"
    }
   },
   "source": [
    "**Optional:** Uncomment and execute to display the table contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the coordinates, right ascension (RA) vs. declination (Dec).\n",
    "\n",
    "Use the `astropy` class `SphericalCircle` to shade the query region.\n",
    "This class takes into account the \"cos-dec\" factor\n",
    "(for points off the celestial equator, i.e., at non-zero declination, the real distance\n",
    "in the right ascension axis is smaller by a factor of the cosine of the declination)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(4, 4))\n",
    "\n",
    "region = SphericalCircle((center_ra * u.deg, center_dec * u.deg),\n",
    "                         radius * u.deg, alpha=0.2, color='blue')\n",
    "plt.gca().add_patch(region)\n",
    "\n",
    "plt.plot(results['coord_ra'], results['coord_dec'],\n",
    "         'o', alpha=0.5, color='black', mew=0)\n",
    "\n",
    "plt.xlabel('RA')\n",
    "plt.ylabel('Dec')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 1: The Right Ascension (RA) versus Declination (Dec) coordinates for Objects retrieved by the query (black dots) within the query region (blue circle).\n",
    "\n",
    "Clean up (and save memory) by deleting the job and its results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job.delete()\n",
    "del query, results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3. Limit rows returned with TOP\n",
    "\n",
    "For debugging and testing queries, it can be useful to only retrieve a subset of the rows\n",
    "which meet the query constraints, because this is faster.\n",
    "\n",
    "Alter the query used above to use `TOP` to return only `N` rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "N = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "query = \"SELECT TOP \" + str(N) + \" coord_ra, coord_dec \"\\\n",
    "        \"FROM dp02_dc2_catalogs.Object \"\\\n",
    "        \"WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), \"\\\n",
    "        \"CIRCLE('ICRS', \" + str_center_coords + \", \" + str_radius + \")) = 1 \"\\\n",
    "        \"AND detect_isPrimary = 1\"\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the job asynchronously and wait until it is finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = service.submit_job(query)\n",
    "job.run()\n",
    "job.wait(phases=['COMPLETED', 'ERROR'])\n",
    "print('Job phase is', job.phase)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = job.fetch_result().to_table()\n",
    "print(len(results))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(4, 4))\n",
    "region = SphericalCircle((center_ra * u.deg, center_dec * u.deg),\n",
    "                         radius * u.deg, alpha=0.2, color='blue')\n",
    "plt.gca().add_patch(region)\n",
    "plt.plot(results['coord_ra'], results['coord_dec'],\n",
    "         'o', alpha=0.5, color='black', mew=0)\n",
    "plt.xlabel('RA')\n",
    "plt.ylabel('Dec')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 2: The Right Ascension (RA) versus Declination (Dec) coordinates for only the top 20 Objects retrieved by the query (black dots) within the query region (blue circle). All of the black dots are in the lower half of the blue circle.\n",
    "\n",
    "\n",
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job.delete()\n",
    "del query, results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4. Use of maxrec instead of TOP\n",
    "\n",
    "TAP queries run in the Notebook Aspect of the RSP can also use the `maxrec` keyword for the TAP service.\n",
    "Use of `TOP` is preferred, as described in Section 1.4.4, but they work the same way from a Jupyter Notebook.\n",
    "\n",
    "Use the same query as above, but without `TOP`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT coord_ra, coord_dec \"\\\n",
    "        \"FROM dp02_dc2_catalogs.Object \"\\\n",
    "        \"WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), \"\\\n",
    "        \"CIRCLE('ICRS', \" + str_center_coords + \", \" + str_radius + \")) = 1 \"\\\n",
    "        \"AND detect_isPrimary = 1\"\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-10T18:32:33.275011Z",
     "iopub.status.busy": "2024-07-10T18:32:33.274479Z",
     "iopub.status.idle": "2024-07-10T18:32:33.382310Z",
     "shell.execute_reply": "2024-07-10T18:32:33.381776Z",
     "shell.execute_reply.started": "2024-07-10T18:32:33.274989Z"
    }
   },
   "source": [
    "Include `maxrec` as a keyword when submitting the query to the TAP service. Run the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = service.submit_job(query, maxrec=N)\n",
    "job.run()\n",
    "job.wait(phases=['COMPLETED', 'ERROR'])\n",
    "print('Job phase is', job.phase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Retrieve the job results.\n",
    "\n",
    "> **Warning:** Use of `maxrec` might return a `DALOverflowWarning` when the results are retrieved. This warns the user that partial results have been returned. In this case, it is ok to ignore this warning, because the query's intent was to return partial results by using `maxrec`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = job.fetch_result().to_table()\n",
    "print(len(results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(4, 4))\n",
    "region = SphericalCircle((center_ra * u.deg, center_dec * u.deg),\n",
    "                         radius * u.deg, alpha=0.2, color='blue')\n",
    "plt.gca().add_patch(region)\n",
    "plt.plot(results['coord_ra'], results['coord_dec'],\n",
    "         'o', alpha=0.5, color='black', mew=0)\n",
    "plt.xlabel('RA')\n",
    "plt.ylabel('Dec')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 3: Identical to Figure 2.\n",
    "\n",
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job.delete()\n",
    "del query, results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5. Sort results with ORDER BY\n",
    "\n",
    "As described in Section 1.4.5, **use `ORDER BY` and `TOP` together with caution**.\n",
    "\n",
    "The TAP service first applies `WHERE` constraints, then `ORDER BY`, and then `TOP`.\n",
    "If the query is not well constrained, i.e., if thousands or more objects meet the `WHERE`\n",
    "constraints, then they all must first be sorted before the top number are returned. \n",
    "This is a waste of time and compute resources.\n",
    "\n",
    "Create the query. Add an `ORDER BY` statement to the TAP query to return the `TOP` N objects\n",
    "sorted by `coord_ra`, in ascending (`ASC`) order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT TOP \" + str(N) + \" coord_ra, coord_dec \"\\\n",
    "        \"FROM dp02_dc2_catalogs.Object \"\\\n",
    "        \"WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), \"\\\n",
    "        \"CIRCLE('ICRS', \" + str_center_coords + \", \" + str_radius + \")) = 1 \"\\\n",
    "        \"AND detect_isPrimary = 1 \"\\\n",
    "        \"ORDER BY coord_ra ASC\"\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-10T19:20:01.849536Z",
     "iopub.status.busy": "2024-07-10T19:20:01.848539Z",
     "iopub.status.idle": "2024-07-10T19:20:02.571184Z",
     "shell.execute_reply": "2024-07-10T19:20:02.570347Z",
     "shell.execute_reply.started": "2024-07-10T19:20:01.849499Z"
    }
   },
   "source": [
    "Run the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = service.submit_job(query)\n",
    "job.run()\n",
    "job.wait(phases=['COMPLETED', 'ERROR'])\n",
    "print('Job phase is', job.phase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Retrieve the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = job.fetch_result().to_table()\n",
    "print(len(results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(4, 4))\n",
    "region = SphericalCircle((center_ra * u.deg, center_dec * u.deg),\n",
    "                         radius * u.deg, alpha=0.2, color='blue')\n",
    "plt.gca().add_patch(region)\n",
    "plt.plot(results['coord_ra'], results['coord_dec'],\n",
    "         'o', alpha=0.5, color='black', mew=0)\n",
    "plt.xlabel('RA')\n",
    "plt.ylabel('Dec')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 4: Similar to Figure 2, except the top 20 Objects returned (black dots) are all on the left-most edge of the blue circle, because they have the lowest RA values,\n",
    "and the query results were sorted by RA in ascending order before the top 20 were selected.\n",
    "\n",
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job.delete()\n",
    "del query, results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6. Sort results with pandas\n",
    "\n",
    "The `pandas` package provides high-performance, easy-to-use data structures and data analysis tools for the Python programming language.\n",
    "TAP query results can be converted to `pandas` dataframes, and then `pandas` functionality (such as sorting) can be applied to the dataframe.\n",
    "\n",
    "Use the same query as above for a small, unsorted set of Objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT coord_ra, coord_dec \"\\\n",
    "        \"FROM dp02_dc2_catalogs.Object \"\\\n",
    "        \"WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), \"\\\n",
    "        \"CIRCLE('ICRS', \" + str_center_coords + \", \" + str_radius + \")) = 1 \"\\\n",
    "        \"AND detect_isPrimary = 1\"\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = service.submit_job(query)\n",
    "job.run()\n",
    "job.wait(phases=['COMPLETED', 'ERROR'])\n",
    "print('Job phase is', job.phase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Retrieve the results, and use the `to_pandas()` method to convert `results` to a `pandas` dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = job.fetch_result().to_table().to_pandas()\n",
    "print(len(results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the type of `results` and see that it is a `pandas` dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, the order of results from TAP queries cannot be assumed to be the same every time,\n",
    "and they will not necessarily be sorted unless `ORDER BY` has been used in the query statement.\n",
    "\n",
    "Use the `pandas` method `sort_values` to sort the dataframe by `coord_ra`, save it as a new\n",
    "dataframe named `sorted_results`, and display the new sorted dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_results = results.sort_values('coord_ra')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the index values did not update, and are not in the order of the sorted dataframe.\n",
    "\n",
    "To reset the index, use the `pandas` method `set_index` to reset the index of the `sorted_results` dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_results.set_index(np.array(range(len(sorted_results))), inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job.delete()\n",
    "del query, results, sorted_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Retrieve query results with job URL\n",
    "\n",
    "Job results are generally available from previously run queries,\n",
    "and can be retrieved if the URL of the job is known _and_ if the job\n",
    "has not been deleted.\n",
    "\n",
    "Do not use `job.delete()` if the results will be retrieved later!\n",
    "\n",
    "First, execute the same query as used above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT coord_ra, coord_dec \"\\\n",
    "        \"FROM dp02_dc2_catalogs.Object \"\\\n",
    "        \"WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), \"\\\n",
    "        \"CIRCLE('ICRS', \" + str_center_coords + \", \" + str_radius + \")) = 1 \"\\\n",
    "        \"AND detect_isPrimary = 1\"\n",
    "print(query)\n",
    "\n",
    "job = service.submit_job(query)\n",
    "job.run()\n",
    "job.wait(phases=['COMPLETED', 'ERROR'])\n",
    "print('Job phase is', job.phase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of using `job.fetch_result()`, store the `job.url` as `my_job_url` and print it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_job_url = str(job.url)\n",
    "print(my_job_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This URL can be used to retrieve the query results.\n",
    "\n",
    "The URL could be shared with another user of the Rubin TAP service, and they could retrieve the same results.\n",
    "\n",
    "Retrieve the job by passing `my_job_url` to `retrieve_query`, then retrieve the results with `fetch_result()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "retrieved_job = retrieve_query(my_job_url)\n",
    "retrieved_results = retrieved_job.fetch_result().to_table().to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "retrieved_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job.delete()\n",
    "del query, retrieved_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1. Retrieve results from a Portal query\n",
    "\n",
    "It is also possible to retrieve the results of queries executed in the Portal Aspect of the Rubin Science Platform."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1.1. Run an ADQL query in the Portal\n",
    "\n",
    "In a new browser tab, go to <a href=\"https://data.lsst.cloud/\">data.lsst.cloud</a> and enter the Portal Aspect.\n",
    "\n",
    "Click on the tab \"DP0.2 Catalogs\" at the top of the screen.\n",
    "\n",
    "Click \"Edit ADQL\" at upper right.\n",
    "\n",
    "Copy-paste the following query into the ADQL box as shown in the screenshot below.\n",
    "\n",
    "```\n",
    "SELECT coord_ra, coord_dec FROM dp02_dc2_catalogs.Object\n",
    "WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), CIRCLE('ICRS', 62, -37, 0.01)) = 1\n",
    "AND detect_isPrimary = 1\n",
    "```\n",
    "\n",
    "<img src=\"data/DP02_02a_S4_fig1.png\" width=700>\n",
    "\n",
    "Figure 5: A screenshot of the Portal's ADQL interface, with the query statement copied in.\n",
    "\n",
    "Click \"Search\" in the lower left corner of the Portal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1.2. Get the job URL from the Portal\n",
    "\n",
    "The default results view will appear as in the screenshot below.\n",
    "\n",
    "<img src=\"data/DP02_02a_S4_fig2.png\" width=700>\n",
    "\n",
    "Figure 6: A screenshot of the Portal's results view for the ADQL query in Figure 5.\n",
    "\n",
    "The search results have been automatically saved and assigned a URL.\n",
    "\n",
    "Click on the \"Info\" button (the letter i in a circle),\n",
    "which is in the upper right-hand corner of the table (bottom of the screen).\n",
    "\n",
    "The pop-up window contains the URL (the Job Link).\n",
    "\n",
    "<img src=\"data/DP02_02a_S4_fig3.png\" width=500>\n",
    "\n",
    "Figure 7: A screenshot of the table information, including the Job URL."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1.3. Retrieve the results\n",
    "\n",
    "Follow the instructions above to create a new URL.\n",
    "\n",
    "> **Warning:** Using the job URL in the screenshot above will not work, it has expired.\n",
    "\n",
    "Copy the newly created URL into the empty quotes below to define `my_portal_url`.\n",
    "\n",
    "Uncomment and execute the following cells to obtain the results of \n",
    "the Portal query here in the Notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# my_portal_url = ''\n",
    "# retrieved_job = retrieve_query(my_portal_url)\n",
    "# retrieved_results = retrieved_job.fetch_result().to_table().to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# retrieved_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete the retrieved job and clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# retrieved_job.delete()\n",
    "# del retrieved_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LSST",
   "language": "python",
   "name": "lsst"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}