Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Danibishop/longitude rework #32

Merged
merged 52 commits into from
Feb 6, 2019
Merged
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
738b144
Scaffold as packagestarting in version 0.3.0
Jan 23, 2019
8845ea5
Scaffold structure including initial unit test structure + coverage c…
Jan 23, 2019
64073a7
Scaffold for carto data source including tests.
Jan 23, 2019
37f443a
Added per-query configuration (+ default). Exception wrapping to hide…
Jan 24, 2019
44bf413
Basic example added to query Carto
Jan 24, 2019
fb53a06
100% coverage for Carto data source. Common configuration for on prem…
Jan 24, 2019
7be594a
Added class to wrap query responses, regardless of the database speci…
Jan 24, 2019
2c74c55
Updated doc
Jan 24, 2019
8fb5849
Transparent cache logic inserted in base data source logic. README no…
Jan 25, 2019
ed0023b
Basic RAM cache implementation. Source code folder structure rework t…
Jan 28, 2019
63ba516
Basic Redis put/get feature. Cache config added. Updated README.
Jan 28, 2019
2458d02
Added flush to caches
Jan 28, 2019
000e826
Picke as default cache serializer.
Jan 28, 2019
3fff406
Cache now stores normalized payloads only
Jan 28, 2019
dcd2f10
Updated tests
Jan 28, 2019
08aa397
Added tests for Redis cache and RAM cache. Improved total coverage up…
Jan 28, 2019
fec1359
Cache configuration wrapped in class instead of dictionary (only for …
Jan 28, 2019
1d9c0fe
Initial scaffold for postgres data source
Jan 29, 2019
20ece33
Requied Python version updated to 3.7. Reference to shared google doc…
Jan 29, 2019
f3a7005
Merge branch 'danibishop/longitude-rework' into danibishop/postgres-i…
Jan 29, 2019
2ecf601
Basic Postgres sample running
Jan 29, 2019
fe2078c
psycopg2 package removed in favour of psycopg2-binary distribution. F…
Jan 29, 2019
a50754f
Write query comprobation removed. Closes #33. Use of cache is now con…
Jan 30, 2019
edff45b
Previews for query response removed.
Jan 30, 2019
07054fc
Configuration logic extracted to simple class. Now it is common for d…
Jan 30, 2019
2dd8546
Cache can be safely disabled temporarily using the DisableCache conte…
Jan 30, 2019
60d4680
Common response class extracted to its own module. Rows data is now s…
Jan 30, 2019
4211956
Imports reordered (isort) and setup.py updated with current structure.
Jan 31, 2019
e449a3c
Updated pytest dependency. Added some pytest plugins.
Jan 31, 2019
26e214f
Fix. Queries were being passed already formatted to the specific sub-…
Jan 31, 2019
5a2b610
Fix. Queries were being passed already formatted to the specific sub-…
Jan 31, 2019
e1d7773
Merge branch 'danibishop/longitude-rework' into danibishop/sqlalchemy…
Jan 31, 2019
86a159b
Added SQLAlchemy and dependencies updated.
Jan 31, 2019
d295ad1
get_config() now returns default and custom configurations merged if …
Jan 31, 2019
8aa0034
Merge branch 'danibishop/longitude-rework' of github.com:GeographicaG…
Feb 1, 2019
415772f
Merge branch 'danibishop/distribution-rework' into danibishop/sqlalch…
Feb 1, 2019
120336e
100% coverage for config and helpers. 86% total coverage.
Feb 1, 2019
6feda08
Sample use of SQL Alchemy using Expression Language
Feb 1, 2019
333e275
Bugfix. Cache object was putting parameters instead of payloads.
Feb 1, 2019
10039b0
SQLAlchemy data source full sample.
Feb 1, 2019
a17482e
Configuration object class created.
Feb 1, 2019
7af7f58
Example for configuration file using a Carto and Postgres data source…
Feb 2, 2019
4a36a9a
Merge branch 'danibishop/longitude-rework' of github.com:GeographicaG…
Feb 2, 2019
746788b
Expiration time added to cache interface
Feb 2, 2019
ba494a3
Expanded explanation about issues when rendering binded parameters to…
Feb 2, 2019
78bc949
Source folder renamed as longitude.
Feb 2, 2019
6566bb2
Build + development environment switched to Poetry
Feb 2, 2019
1c43064
Python requirement lowered to 3.6 or bigger instead of 3.7. Fixed cov…
Feb 4, 2019
74773c5
First try with Jenkins + Poetry
Feb 4, 2019
ca0d967
Updated data source base including some shortcut query methods and da…
Feb 5, 2019
11029de
Updated dependencies
Feb 5, 2019
ad0feb7
Config reworked as in ch31846.
Feb 5, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Config reworked as in ch31846.
EnvironmentConfiguration is now a domain class that exposes a single get(key) method.
It will parse environment variables in the form LONGITUDE__PARENT_OBJECT__CHILD_OBJECT__VALUE=42 as {'parent_object': {'child_object': {'value': 42 } }
It also allows to recover the values using nested keys ('.' is the joiner): Config.get('parent_object.child_object.value') returns 42 (as integer).
Also, if a value can be parsed as integer, it will be parsed.
  • Loading branch information
Dani Ramirez committed Feb 5, 2019
commit ad0feb7b7622f6650ac8005e5b965c7daca50d6e
86 changes: 64 additions & 22 deletions longitude/core/common/config.py
Original file line number Diff line number Diff line change
@@ -5,36 +5,78 @@


class EnvironmentConfiguration:
prefix = 'LONGITUDE'
separator = '__'
config = None

def __init__(self, d):
self._original_config = d
self._parsed_config = dict(d)
@classmethod
def _load_environment_variables(cls):
"""
It loads environment variables into the internal dictionary.

Load is done by grouping and nesting environment variables following this convention:
1. Only variables starting with the prefix are taken (i.e. LONGITUDE)
2. For each separator used, a new nested object is created inside its parent (i.e. SEPARATOR is '__')
3. The prefix indicates the root object (i.e. LONGITUDE__ is the default root dictionary)

:return: None
"""
cls.config = {}
for v in [k for k in os.environ.keys() if k.startswith(cls.prefix)]:
value_path = v.split(cls.separator)[1:]
cls._append_value(os.environ.get(v), value_path, cls.config)

self._parse_env_vars_references(self._parsed_config)
@classmethod
def get(cls, key=None):
"""
Returns a nested config value from the configuration. It allows getting values as a series of joined keys using
dot ('.') as separator. This will search for keys in nested dictionaries until a final value is found.

def __getitem__(self, key):
return self._parsed_config[key]
:param key: String in the form of 'parent.child.value...'. It must replicate the configuration nested structure.
:return: It returns an integer, a string or a nested dictionary. If none of these is found, it returns None.
"""

# We do a lazy load in the first access
if cls.config is None:
cls._load_environment_variables()

if key is not None:
return cls._get_nested_key(key, cls.config)
else:
return cls.config

@staticmethod
def _parse_env_vars_references(dictionary):
def _get_nested_key(key, d):
"""
Modifies a dictionary like this:
* Recursively
* If a value is a string starting with '=', it gets substituted by the corresponding environment variable
:param dictionary: Dictionary that will be modified.
:return: Nothing

:param key:
:param d:
:return:
"""
key_path = key.split('.')
root_key = key_path[0]

for k in dictionary.keys():
if isinstance(dictionary[k], dict):
EnvironmentConfiguration._parse_env_vars_references(dictionary[k])
elif isinstance(dictionary[k], str) and dictionary[k].startswith('='):
env_var = dictionary[k][1:] # We remove the '='
value = os.environ.get(env_var)
if value:
dictionary[k] = value
else:
dictionary[k] += ' [NOT FOUND]'
if root_key in d.keys():
if len(key_path) == 1:
return d[root_key] # If a single node is in the path, it is the final one
# If there are more than one nodes left, keep digging...
return EnvironmentConfiguration._get_nested_key('.'.join(key_path[1:]), d[root_key])
else:
return None # Nested key was not found in the config

@staticmethod
def _append_value(value, value_path, d):
root_path = value_path[0].lower()
if len(value_path) == 1:

try:
d[root_path] = int(value)
except ValueError:
d[root_path] = value
else:
if root_path not in d.keys():
d[root_path] = {}
EnvironmentConfiguration._append_value(value, value_path[1:], d[root_path])


class LongitudeConfigurable:
File renamed without changes.
52 changes: 29 additions & 23 deletions longitude/core/tests/test_environment_configuration_dictionary.py
Original file line number Diff line number Diff line change
@@ -1,32 +1,38 @@
from unittest import TestCase, mock
from longitude.core.common.config import EnvironmentConfiguration
from longitude.core.common.config import EnvironmentConfiguration as Config

fake_environment = {
'PATATUELA_RULES': 'my_root_value'
'LONGITUDE__PARENT__CHILD__VALUE_A': '42',
'LONGITUDE__PARENT__CHILD__VALUE_B': 'wut',
'LONGITUDE__VALUE_A': '8008'
}


@mock.patch.dict('longitude.core.common.config.os.environ', fake_environment)
class TestConfigurationDictionary(TestCase):

@mock.patch.dict('longitude.core.common.config.os.environ', fake_environment)
def test_base(self):
d = EnvironmentConfiguration({
'root_patatuela': '=PATATUELA_RULES',
'patata': 'patata value',
'potato': 'potato value',
'potatoes': [
'potato A', 'poteito B'
],
'potato_sack': {
'colour': 'meh',
'taste': 'buah',
'texture': {
'external': 'oh no',
'internal': 'omg',
'bumpiness': '=SOME_VALUE_FOR_BUMPINESS'
}
}
})
def test_existing_values_return_strings_or_integers(self):
self.assertEqual(42, Config.get('parent.child.value_a'))
self.assertEqual('wut', Config.get('parent.child.value_b'))
self.assertEqual(8008, Config.get('value_a'))

self.assertEqual('my_root_value', d['root_patatuela'])
self.assertEqual('=SOME_VALUE_FOR_BUMPINESS [NOT FOUND]', d['potato_sack']['texture']['bumpiness'])
def test_non_existing_values_return_none(self):
self.assertEqual(None, Config.get('wrong_value'))
self.assertEqual(None, Config.get('wrong_parent.child.value'))
self.assertEqual(None, Config.get('parent.wrong_child.value'))
self.assertEqual(None, Config.get('parent.child.wrong_value'))
self.assertEqual(None, Config.get('parent.wrong_child'))

def test_existing_nested_values_return_dictionaries(self):
fake_config = {
'parent':
{'child':
{
'value_a': 42,
'value_b': 'wut'
}
},
'value_a': 8008
}
self.assertEqual(fake_config, Config.get())
self.assertEqual(fake_config['parent']['child'], Config.get('parent.child'))
32 changes: 7 additions & 25 deletions longitude/samples/mixed_datasources.py
Original file line number Diff line number Diff line change
@@ -28,7 +28,7 @@
from longitude.core.caches.redis import RedisCache
from longitude.core.data_sources.postgres.default import DefaultPostgresDataSource
from longitude.core.data_sources.carto import CartoDataSource
from longitude.core.common.config import EnvironmentConfiguration
from longitude.core.common.config import EnvironmentConfiguration as Config


def import_table_values_from_carto(limit):
@@ -59,33 +59,15 @@ def import_table_values_from_carto(limit):
params=params,
needs_commit=True)

res = postgres.query('select * from county_population')
print(res.rows)


if __name__ == "__main__":

# This is the global config object
# We are going to retrieve some values from a table in Carto, create a local table and copy the values
# doing simple inserts (to show how to do queries)

config = EnvironmentConfiguration({
'carto_main': {
'api_key': "=CARTO_API_KEY",
'user': "=CARTO_USER",

'cache': {
'password': '=REDIS_PASSWORD'
}
},
'postgres_main': {
'host': "=POSTGRES_HOST",
'port': "=POSTGRES_PORT",
'db': "=POSTGRES_DB",
'user': "=POSTGRES_USER",
'password': "=POSTGRES_PASS"
}
})

carto = CartoDataSource(config['carto_main'], cache_class=RedisCache)
postgres = DefaultPostgresDataSource(config['postgres_main'])
print('REDIS password is %s' % Config.get('carto_main.cache.password'))
carto = CartoDataSource(Config.get('carto_main'), cache_class=RedisCache)
postgres = DefaultPostgresDataSource(Config.get('postgres_main'))
carto.setup()
postgres.setup()