Merge pull request #39 from moj-analytical-services/dev

Dev

.github/workflows/lint.yml

@@ -0,0 +1,31 @@
+name: Lint
+on: [pull_request]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      max-parallel: 4
+      matrix:
+        python-version: [3.8]
+
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v1
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install flake8 yamllint
+      - name: Lint Python files with flake8
+        run: >
+          flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics &&
+          flake8 . --count --exit-zero --max-complexity=10 --max-line-length=88
+          --statistics
+      - name: Lint YAML files with yamllint
+        run: >
+          yamllint . -d "{extends: default, rules: {line-length:
+          {allow-non-breakable-words: true, allow-non-breakable-inline-mappings:
+          true}}}" --no-warnings

.github/workflows/test.yaml

@@ -0,0 +1,23 @@
+name: Run tests
+
+on: pull_request
+
+jobs:
+  python-version-run-tests:
+    runs-on: ubuntu-latest
+    strategy:
+      max-parallel: 4
+      matrix:
+        python-version: [3.7, 3.8]
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v1
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install poetry
+          poetry update
+          poetry run pytest -vv tests/test_sql_parse.py

CHANGELOG.md

@@ -5,6 +5,14 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## v3.0.0 - 2021-01-26
+
+### Changed
+- `pydbtools` now acts as a wrapper for the athena module in awswrangler
+- Previous functions `get_athena_query_response` and `read_sql` are now deprecated (but still currently work with this release. later releases may remove them).
+- Allows users to create temporary tables that are stored in a database aliased by the name `__temp__`. SQL queries will replace any reference to `__temp__` with the real database name before the call.
+
+
 ## v2.0.2 - 2020-11-26
 
 ### Fixed

README.md

@@ -1,5 +1,120 @@
 # pydbtools
 
+## Installation
+
+> Requires a pip release above 20.
+
+```bash
+## To install from pypi
+pip install pydbtools
+
+## Or install from git with a specific release
+pip install "pydbtools @ git+https://github.com/moj-analytical-services/[email protected]"
+```
+
+## Quickstart guide
+
+### Read an SQL Athena query into a pandas dataframe
+
+```python
+import pydbtools as pydb
+df = pydb.read_sql("SELECT * from a_database.table LIMIT 10")
+```
+
+### Run a query in Athena
+
+```python
+response = pydb.start_query_execution_and_wait("CREATE DATABASE IF NOT EXISTS my_test_database")
+```
+
+### Create a temporary table to do further separate SQL queries on later
+
+```python
+pydb.create_temp_table("SELECT a_col, count(*) as n FROM a_database.table GROUP BY a_col", table_name="temp_table_1")
+df = pydb.read_sql_query("SELECT * FROM __temp__.temp_table_1 WHERE n < 10")
+```
+
+## Introduction
+
+This package is a wrapper for [awswrangler](https://aws-data-wrangler.readthedocs.io/en/2.3.0/what.html) that which presets/defines some of the input parameters to the athena module functions to align with our platform setup. See the [awswrangler API reference documentation for Athena](https://aws-data-wrangler.readthedocs.io/en/2.3.0/api.html#amazon-athena) to see what functions you can call from pydbtools.
+
+The function parameters that are locked down / altered by `pydbtools` are:
+- **boto3_session:** This is auto generated by `pydbtools` (in order to grab the user credentials from the sts client - this is needed for the R version of this package which calls this package under the hood. In short forcing refreshed credentials are needed in R as boto3 credentials timeout and do not refresh when using reticulate (at least currently))
+- **s3_output:** The S3 path where database queries are written to. This is defined by `pydbtools` based on the IAM user/role calling the query (ensures that each role can only read/write to a S3 path only they can access).
+- **database:** Will either be set to `None` or `__temp__` depending on other user parameters (if `ctas_approach=True`). `__temp__` is an alias to an autogenerated temp database name which is generated from `pydbtools` again based on the IAM user/role. References to this temporary database can be referenced by the keyword `__temp__` in SQL queries see additional functionality to awswrangler section.
+- **sql:** We allows reference to the database name `__temp__` which is an alias to a user specific temporary database. When a function call has an SQL parameter the SQL is checked with an SQL parser and then any reference to `__temp__` as a database is replaced with the actual database name which is autogenerated. This replacement only occurs for `SELECT` queries.
+
+## Additional Functionality
+
+As well as acting as a wrapper function for awswrangler this package also allows you to do the following:
+
+### Run query and wait for a response
+
+This function essentially calls two functions from `awswrangler.athena`. First `start_query_execution` followed by `wait_query`.
+
+```python
+import pydbtools as pydb
+
+response = pydb.start_query_execution_and_wait("SELECT * from a_database.table LIMIT 10")
+```
+
+### Create Temporary Tables
+
+You can use the `create_temp_table` function to write SQL to create a store a temporary table that sits in your `__temp__` database.
+
+```python
+import pydbtools as pydb
+
+pydb.create_temp_table("SELECT * from a_database.table LIMIT 10", table_name="temp_table_1")
+df = pydb.read_sql_query("SELECT * from __temp__.temp_table_1")
+df.head()
+```
+
+## Usage / Examples
+
+### Simple 
+
+```python
+import pydbtools as pydb
+
+# Run a query using pydbtools
+response = pydb.start_query_execution_and_wait("CREATE DATABASE IF NOT EXISTS my_test_database")
+
+# Read data from an athena query directly into pandas
+pydb.read_sql("SELECT * from a_database.table LIMIT 10")
+
+# Create a temp table to do further seperate SQL queries later on
+pydb.create_temp_table("SELECT a_col, count(*) as n FROM a_database.table GROUP BY a_col", table_name="temp_table_1")
+df = pydb.read_sql_query("SELECT * FROM __temp__.temp_table_1 WHERE n < 10")
+```
+
+### More advanced usage
+
+Get the actual name for your temp database, create your temp db then delete it using awswrangler (note: `awswrangler` will raise an error if the database does not exist)
+
+```python
+import awswrangler as wr
+import pydbtools as pydb
+
+user_id, out_path = pydb.get_user_id_and_table_dir()
+temp_db_name = pydb.get_database_name_from_userid(user_id)
+print(temp_db_name)
+pydb.create_temp_table()
+print(wr.catalog.delete_database(name=temp_db_name))
+```
+
+# DEPRECATED
+
+## Functions
+
+The functions:
+- `pydbtools.get_athena_query_response`
+- `pydbtools.read_sql`
+
+Are now deprecated and calls to these functions will raise an warning. They have been replaced by `pydbtools.start_query_execution_and_wait` and `pydbtools.read_sql_query`.
+
+## Docs for versions below v3.0.0
+
 This is a simple package that let's you query databases using Amazon Athena and get the s3 path to the athena out (as a csv). This is significantly faster than using the the database drivers so might be a good option when pulling in large data. By default, data is converted into a pandas dataframe with equivalent column data types as the Athena table - see "Meta Data" section below.
 
 Note to use this package you need to be added to the StandardDatabaseAccess IAM Policy on the Analytical Platform. Please contact the team if you require access.