Bug tracker and Buganizer fixes (#147)

* Empty Dataset handled in tree view * condition changes * Big Query project hierarchy and cleanup * DPMS old code retained changes * Dataset API pageToken and CSS tree changes * changing dataplex API to bigquery APIs * Big Query region added and stored in local storage * FE changes for APIs from Dataplex to bigQuery * ID-213. Removed cluster option for BigQuery * Big query enhancement changes * Big Query schema added fields and tree condition changes * BQ region grouping added and stored value in settings * BQ dataset filter by BQ region changes * settings changes * BQ region onchange for empty handled * version updated to 0.1.76 * Handling table count logic change * adding page token * Table preview API page token added * Tree view based on total table counts * preview check * Tree view loader fix * Preview API error handling * code cleanup * BQ table, dataset info page loader and preview empty handled * BE code review changes and formating * Internal review comments changes FE * Review comments - bq region dropdown created * license header year change for new files created * variable name change to "dataset" * table style handled using css selector * useEffect review comments fixed * useEffect review comments fixes * usememo removed for schema tables * preview column type added * minor fix for Loader * Serverless notebook logo based on language * Bug tracker - 222 dag run always for selected date * Type added * setIsLoading missing added * Table and dataset code bigquery refactoring * Rename wrapper file * Bq preview index file changes * Wrapper files name change * Tree structure refactor logic changes * Plugin toast handled based on BQ feature enable * big query and dpms refactor * ID:221-Toast message timer fix * Multiple projects handling in dataset tree view 1. BQ project list api added. 2. All BQ api project added based on FE selection. 3. Handled tree structure FE for new view. * Empty schema info page handled * search functionality code commented out * Height css changes * schema empty error message added * schema page minor UI fixes scroll * Network configuration Error messages style change * No cluster available handled for create pages * search code enabled * ipynb file removed * Revert "search code enabled" This reverts commit 799f604. * Revert "ipynb file removed" This reverts commit 0844b26. * Delete ipynb file * History server cluster no data message added Toast removed for empty cluster * Big Query search changes FE and BE * code cleanup changes * commented code removed * Fetching all the records in backend * Incorporated review comments * code formatting * removed utitlities * By default display BigQuery tree in panel * BQ search FE search removed and handled in BE * big query tree loader icon size reduced * No dataset - Toast removed * search loader size reduced * Handled dataset empty condition * handled text overflow for big query tree data * ID:231, 233 bugs fixed * css fixes * Dataset explorer reload after project/region change * code cleanup * name change * license year change * toast errors changes * active toast fix * bug tracker fixes - 235, 238, 241 Initial fixes for above bugs * console log fix * Bug tracker ID 237 fix * code cleanuo * code cleanup * Bug tracker ID - 243 fix * Preview scroll fix * minor css fix * Preview page pagination server side changes * license year change * Preview data object type handled * Big Query dataset explorer new panel created * Loader changes and pagination changes * class name changes - BigQueryWidget * Dataset explorer refresh icon added * Refresh Icon moved to top level * icon changes * handling loader * removed commented * configure gateway name change * get cached credentials added in utils * Credentials cache added * token expiry fetch changes * Tooltip added for custom created panels * Linear progress bar added for calendar load * UI fixes and Loader added for tree all levels * remove cache * code formatting * loader padding changes * prettier fix * Tooltip added for tree * bug fix 1.Calendar progress bar, div fix 2.BQ, DPMS sequence fix 3.Left panel last letters fix * code cleanup * execution history height changes * Backend file, folder name changes * controller name fix * Big Query code separation - FE * Dpms service file separation changes * Height resize handled on window height change * version updated to 0.1.77 * ID 249- BQ preview pagination fixes * ID 341324879 - Buganizer P4 fix * ID 341318983 - P3 BQ region settings fixes * Bug Tracker Id - 251 and Buganiser id 341323620 fix * toast handling in dag list * Buganizer 341324188 and ID - 227 fixes * code fix * BQ preview page loader added for page switch * version updated to 0.1.78 * debounce added * page size increased and year change * Changed all the credentials API to async await * Moved a method from urls to gcpUrlService * List jobs API call stop after unmounting the page * Moving the gcp_service_url back to urls file * Review comments * Create batch - Network & SubNetwork fixes handled for create and clone loader fix * internal bug fix * internal bug fix ID-1 * Refactored backend code as per review comments * removed unwanted code * Bug fix for the prepare_dag * Internal bug fixes * handling filter undefined * Dag output download fix * Removing unused input file download and papermill name change * removing dag input file download code * black code formatting * internal bug fix ID: 5 * Internal bug ID-1 * Internal bug fix ID: 2 * Storing dag file in scheduled-jobs folder * var name change * creating subfolder for dag and papermill * Settings page FE changes * unit test cases * Submit job small fix "Spark" * multiple toast fix * toast id change * Settings name modified in test cases * import order and commented code removal * black code formatter * isort import ordering * black code formatting * removing string format in unsed places * removing unused imports --------- Co-authored-by: saranyaloganathan23 <[email protected]> Co-authored-by: Shubha-accenture <[email protected]> Co-authored-by: aditee-accenture <[email protected]> Co-authored-by: harsha-accenture <[email protected]> Co-authored-by: harsha-accenture <[email protected]>
GoogleCloudDataproc · May 30, 2024 · f00d3f4 · f00d3f4
1 parent 23c3e9d
commit f00d3f4
Show file tree

Hide file tree

Showing 63 changed files with 1,901 additions and 1,692 deletions.
diff --git a/dataproc_jupyter_plugin/__init__.py b/dataproc_jupyter_plugin/__init__.py
@@ -12,15 +12,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.from ._version import __version__
 import logging
-from google.cloud.jupyter_config.tokenrenewer import CommandTokenRenewer
 
-from kernels_mixer.kernelspecs import MixingKernelSpecManager
+from google.cloud.jupyter_config.tokenrenewer import CommandTokenRenewer
+from jupyter_server.services.sessions.sessionmanager import SessionManager
 from kernels_mixer.kernels import MixingMappingKernelManager
+from kernels_mixer.kernelspecs import MixingKernelSpecManager
 from kernels_mixer.websockets import DelegatingWebsocketConnection
 
-from jupyter_server.services.sessions.sessionmanager import SessionManager
-
-from .handlers import setup_handlers, configure_gateway_client_url, DataprocPluginConfig
+from .handlers import DataprocPluginConfig, configure_gateway_client_url, setup_handlers
 
 
 def _jupyter_labextension_paths():

diff --git a/dataproc_jupyter_plugin/commons/constants.py b/dataproc_jupyter_plugin/commons/constants.py
@@ -12,27 +12,20 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-
-from dataproc_jupyter_plugin.services.gcpUrlService import gcp_service_url
-
 ENVIRONMENT_API = "https://composer.googleapis.com/v1"
 TAGS = "dataproc_jupyter_plugin"
 GCS = "gs://"
 CONTENT_TYPE = "application/json"
-
-
-dataproc_url = gcp_service_url("dataproc")
-dataproc_url = gcp_service_url("dataproc")
-compute_url = gcp_service_url(
-    "compute", default_url="https://compute.googleapis.com/compute/v1"
-)
-metastore_url = gcp_service_url("metastore")
-cloudkms_url = gcp_service_url("cloudkms")
-cloudresourcemanager_url = gcp_service_url("cloudresourcemanager")
-datacatalog_url = gcp_service_url("datacatalog")
-storage_url = gcp_service_url(
-    "storage", default_url="https://storage.googleapis.com/storage/v1/"
-)
-dataplex_url = gcp_service_url(
-    "dataplex", default_url="https://dataplex.googleapis.com/v1"
-)
+PACKAGE_NAME = "dataproc_jupyter_plugin"
+COMPOSER_SERVICE_NAME = "composer"
+DATAPROC_SERVICE_NAME = "dataproc"
+BIGQUERY_SERVICE_NAME = "bigquery"
+DATACATALOG_SERVICE_NAME = "datacatalog"
+CLOUDRESOURCEMANAGER_SERVICE_NAME = "cloudresourcemanager"
+STORAGE_SERVICE_NAME = "storage"
+COMPUTE_SERVICE_NAME = "compute"
+METASTORE_SERVICE_NAME = "metastore"
+CLOUDKMS_SERVICE_NAME = "cloudkms"
+COMPUTE_SERVICE_DEFAULT_URL = "https://compute.googleapis.com/compute/v1"
+STORAGE_SERVICE_DEFAULT_URL = "https://storage.googleapis.com/storage/v1/"
+WRAPPER_PAPPERMILL_FILE = "wrapper_papermill.py"
diff --git a/dataproc_jupyter_plugin/commons/gcloudOperations.py b/dataproc_jupyter_plugin/commons/gcloudOperations.py
diff --git a/dataproc_jupyter_plugin/controllers/airflow.py b/dataproc_jupyter_plugin/controllers/airflow.py
@@ -0,0 +1,168 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+
+import tornado
+from jupyter_server.base.handlers import APIHandler
+
+from dataproc_jupyter_plugin import credentials
+from dataproc_jupyter_plugin.services import airflow
+
+
+class DagListController(APIHandler):
+    @tornado.web.authenticated
+    async def get(self):
+        try:
+            client = airflow.Client(await credentials.get_cached(), self.log)
+            composer_name = self.get_argument("composer")
+            dag_list = await client.list_jobs(composer_name)
+            self.finish(json.dumps(dag_list))
+        except Exception as e:
+            self.log.exception("Error fetching cluster list")
+            self.finish({"error": str(e)})
+
+
+class DagDeleteController(APIHandler):
+    @tornado.web.authenticated
+    async def get(self):
+        try:
+            client = airflow.Client(await credentials.get_cached(), self.log)
+            composer = self.get_argument("composer")
+            dag_id = self.get_argument("dag_id")
+            from_page = self.get_argument("from_page", default=None)
+            delete_response = await client.delete_job(composer, dag_id, from_page)
+            if delete_response == 0:
+                self.finish(json.dumps({"status": delete_response}))
+            else:
+                self.log.exception("Error deleting dag file")
+                self.finish(json.dumps({"status": delete_response}))
+        except Exception as e:
+            self.log.exception(f"Error deleting dag file: {str(e)}")
+            self.finish({"error": str(e)})
+
+
+class DagUpdateController(APIHandler):
+    @tornado.web.authenticated
+    async def get(self):
+        try:
+            client = airflow.Client(await credentials.get_cached(), self.log)
+            composer = self.get_argument("composer")
+            dag_id = self.get_argument("dag_id")
+            status = self.get_argument("status")
+            update_response = await client.update_job(composer, dag_id, status)
+            if update_response == 0:
+                self.finish({"status": 0})
+            else:
+                self.log.exception("Error updating status")
+                self.finish(json.dumps(update_response))
+        except Exception as e:
+            self.log.exception(f"Error updating status: {str(e)}")
+            self.finish({"error": str(e)})
+
+
+class DagRunController(APIHandler):
+    @tornado.web.authenticated
+    async def get(self):
+        try:
+            client = airflow.Client(await credentials.get_cached(), self.log)
+            composer_name = self.get_argument("composer")
+            dag_id = self.get_argument("dag_id")
+            start_date = self.get_argument("start_date")
+            offset = self.get_argument("offset")
+            end_date = self.get_argument("end_date")
+            dag_run_list = await client.list_dag_runs(
+                composer_name, dag_id, start_date, end_date, offset
+            )
+            self.finish(json.dumps(dag_run_list))
+        except Exception as e:
+            self.log.exception(f"Error fetching dag run list {str(e)}")
+            self.finish({"error": str(e)})
+
+
+class DagRunTaskController(APIHandler):
+    @tornado.web.authenticated
+    async def get(self):
+        try:
+            client = airflow.Client(await credentials.get_cached(), self.log)
+            composer_name = self.get_argument("composer")
+            dag_id = self.get_argument("dag_id")
+            dag_run_id = self.get_argument("dag_run_id")
+            dag_run_list = await client.list_dag_run_task(
+                composer_name, dag_id, dag_run_id
+            )
+            self.finish(json.dumps(dag_run_list))
+        except Exception as e:
+            self.log.exception(f"Error fetching dag run tasks: {str(e)}")
+            self.finish({"error": str(e)})
+
+
+class DagRunTaskLogsController(APIHandler):
+    @tornado.web.authenticated
+    async def get(self):
+        try:
+            client = airflow.Client(await credentials.get_cached(), self.log)
+            composer_name = self.get_argument("composer")
+            dag_id = self.get_argument("dag_id")
+            dag_run_id = self.get_argument("dag_run_id")
+            task_id = self.get_argument("task_id")
+            task_try_number = self.get_argument("task_try_number")
+            dag_run_list = await client.list_dag_run_task_logs(
+                composer_name, dag_id, dag_run_id, task_id, task_try_number
+            )
+            self.finish(json.dumps(dag_run_list))
+        except Exception as e:
+            self.log.exception(f"Error fetching dag run task logs: {str(e)}")
+            self.finish({"error": str(e)})
+
+
+class EditDagController(APIHandler):
+    @tornado.web.authenticated
+    async def get(self):
+        try:
+            client = airflow.Client(await credentials.get_cached(), self.log)
+            bucket_name = self.get_argument("bucket_name")
+            dag_id = self.get_argument("dag_id")
+            dag_details = await client.edit_jobs(dag_id, bucket_name)
+            self.finish(json.dumps(dag_details))
+        except Exception as e:
+            self.log.exception("Error getting dag details")
+            self.finish({"error": str(e)})
+
+
+class ImportErrorController(APIHandler):
+    @tornado.web.authenticated
+    async def get(self):
+        try:
+            client = airflow.Client(await credentials.get_cached(), self.log)
+            composer_name = self.get_argument("composer")
+            import_errors_list = await client.list_import_errors(composer_name)
+            self.finish(json.dumps(import_errors_list))
+        except Exception as e:
+            self.log.exception("Error fetching import error list")
+            self.finish({"error": str(e)})
+
+
+class TriggerDagController(APIHandler):
+    @tornado.web.authenticated
+    async def get(self):
+        try:
+            client = airflow.Client(await credentials.get_cached(), self.log)
+            dag_id = self.get_argument("dag_id")
+            composer = self.get_argument("composer")
+            trigger = await client.dag_trigger(dag_id, composer)
+            self.finish(json.dumps(trigger))
+        except Exception as e:
+            self.log.exception("Error triggering dag")
+            self.finish({"error": str(e)})
diff --git a/dataproc_jupyter_plugin/controllers/bigquery.py b/dataproc_jupyter_plugin/controllers/bigquery.py
@@ -15,15 +15,12 @@
 
 import json
 
-from jupyter_server.base.handlers import APIHandler
 import tornado
-
-from google.cloud.jupyter_config import gcp_project
+from jupyter_server.base.handlers import APIHandler
 
 from dataproc_jupyter_plugin import credentials
 from dataproc_jupyter_plugin.services import bigquery
 
-
 # GCP project holding BigQuery public datasets.
 BQ_PUBLIC_DATASET_PROJECT_ID = "bigquery-public-data"
 
@@ -38,7 +35,7 @@ async def get(self):
             dataset_list = await client.list_datasets(page_token, project_id)
             self.finish(json.dumps(dataset_list))
         except Exception as e:
-            self.log.exception(f"Error fetching datasets")
+            self.log.exception("Error fetching datasets")
             self.finish({"error": str(e)})
 
 
@@ -53,7 +50,7 @@ async def get(self):
             table_list = await client.list_table(dataset_id, page_token, project_id)
             self.finish(json.dumps(table_list))
         except Exception as e:
-            self.log.exception(f"Error fetching datasets")
+            self.log.exception("Error fetching datasets")
             self.finish({"error": str(e)})
 
 
@@ -67,7 +64,7 @@ async def get(self):
             dataset_info = await client.list_dataset_info(dataset_id, project_id)
             self.finish(json.dumps(dataset_info))
         except Exception as e:
-            self.log.exception(f"Error fetching dataset information")
+            self.log.exception("Error fetching dataset information")
             self.finish({"error": str(e)})
 
 
@@ -82,7 +79,7 @@ async def get(self):
             table_info = await client.list_table_info(dataset_id, table_id, project_id)
             self.finish(json.dumps(table_info))
         except Exception as e:
-            self.log.exception(f"Error fetching table information")
+            self.log.exception("Error fetching table information")
             self.finish({"error": str(e)})
 
 
@@ -105,7 +102,7 @@ async def get(self):
             )
             self.finish(json.dumps(preview_data))
         except Exception as e:
-            self.log.exception(f"Error fetching preview data")
+            self.log.exception("Error fetching preview data")
             self.finish({"error": str(e)})
 
 
@@ -124,7 +121,7 @@ async def get(self):
             project_list = await bq_projects_list()
             self.finish(json.dumps(project_list))
         except Exception as e:
-            self.log.exception(f"Error fetching projects")
+            self.log.exception("Error fetching projects")
             self.finish({"error": str(e)})
 
 
@@ -142,5 +139,5 @@ async def get(self):
             )
             self.finish(json.dumps(search_data))
         except Exception as e:
-            self.log.exception(f"Error fetching search data")
+            self.log.exception("Error fetching search data")
             self.finish({"error": str(e)})
diff --git a/dataproc_jupyter_plugin/controllers/clusterController.py b/dataproc_jupyter_plugin/controllers/clusterController.py