Skip to content

Commit

Permalink
Merge pull request #32 from thejunglejane/milestone/v1
Browse files Browse the repository at this point in the history 
Milestone/v1
  • Loading branch information
@thejunglejane
thejunglejane committed Jan 31, 2016
2 parents f08f02b + 90c792e commit 71848a7
Show file tree
Hide file tree
Showing 35 changed files with 1,838 additions and 514 deletions.
109 changes: 70 additions & 39 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ Datums is a PostgreSQL pipeline for [Reporter](http://www.reporter-app.com/). Da
# Getting Started

Skip ahead to [Migrating to v1.0.0](#migrating-to-v100)

## Create the Database

To create the datums database, first ensure that you have postgres installed and that the server is running locally. To create the database
Expand Down Expand Up @@ -70,40 +72,66 @@ or, from Python
>>> base.database_teardown(base.engine)
```

#### Migrating to v1.0.0

##### `alembic`
v1.0.0 introduces some changes to the database schema and the datums data model. To upgrade your existing datums database to a v1.0.0-compatible schema, a series of alembic mirations have been provided. To access these migrations, you will need to have the datums repository cloned to your local machine. If you've installed datums via pip, feel free to delete the cloned repository after you migrate your database, but remember to `pip install --upgrade datums` before trying to add more reports.

To migrate your database, clone (or pull) this repository and run the setup script, then `cd` into the repository and run the migrations with

```bash
/path/to/datums/ $ alembic upgrade head
/path/to/datums/ $ datums --update "/path/to/reporter/folder/*.json"
/path/to/datums/ $ datums --add "/path/to/reporter/folder/*.json"
```

After migrating, it's important to `--update` all reports to add the `pressure_in` and `pressure_mb` attributes on weather reports as well as the `inland_water` attribute to placemark reports. You can safely ignore the `UserWarning` that no `uniqueIdentifier` can be found for altitude reports; those altitude reports will be added when you `--add` in the next step.

v1.0.0 adds support for altitude reports. After updating, you'll need to `--add` all your reports to capture altitude reports from before May, 2015. They must be added instead of updated because altitude reports have not always had `uniqueIdentifiers`. Adding will allow datums to create UUIDs for these earlier altitude reports. If no UUID is found for an altitude report, datums canot update or delete it. See [issue 29](https://github.com/thejunglejane/datums/issues/29) for more information.

##### Quick and Dirty
Alternatively, you could just teardown your existing datums database and setup a new one. Make sure you teardown your database before upgrading datums.
```bash
$ datums --teardown
$ pip install --upgrade datums
$ datums --setup
$ datums --add "/path/to/reporter/folder/*.json"
```

# Adding, Updating, and Deleting
The `pipeline` module allows you to add, update, and delete reports and questions.

### Definitions
We should define a few terms before getting into how to use the pipeline.

* A **reporter file** is a JSON file that contains all the **report**s and all the **question**s for a given day. These files should be located in your Dropbox/Apps/Reporter-App folder.
* A **report** comprises a **snapshot** and all the **response**s collected by Reporter when you make a report.
* A **snapshot** contains the information that the Reporter app automatically collects when you make a report, things like the weather, background noise, etc.
* A **reporter file** is a JSON file that contains all the **snapshot**s and all the **question**s for a given day. These files should be located in your Dropbox/Apps/Reporter-App folder.
* A **snapshot** comprises a **report** and all the **response**s collected by Reporter when you make a report.
* A **report** contains the information that the Reporter app automatically collects when you make a report, things like the weather, background noise, etc.
* A **response** is the answer you enter for a question.

Every **report** will have one **snapshot** and some number **response**s associated with it, and every **reporter file** will have some number **report**s and some number of **question**s associated with it, depending on how many times you make reports throughout the day.
Every **snapshot** will have one **report** and some number **response**s associated with it, and every **reporter file** will have some number **snapshot**s and some number of **question**s associated with it, depending on how many times you make reports throughout the day.

If you add or delete questions from the Reporter app, different **reporter file**s will have different **question**s from day to day. When you add a new **reporter file**, first add the **question**s from that day. If there are no new **question**s, nothing will happen; if there is a new **question**, datums will add it to the database.

## Adding questions and reports
## Adding questions, reports, and responses

When you first set datums up, you'll probably want to add all the questions and reports in your Dropbox Reporter folder.
When you first set datums up, you'll probably want to add all the questions, reports, and responses in your Dropbox Reporter folder.

#### Command Line
To add all the reporter files in your Dropbox Reporter folder from the command line, execute `datums` with the `--add` flag followed by the path to your Dropbox Reporter folder
To add all the Reporter files in your Dropbox Reporter folder from the command line, execute `datums` with the `--add` flag followed by the path to your Dropbox Reporter folder

```
$ datums --add "/path/to/reporter/folder/*.json"
```
Make sure you include the '*.json' at the end to exclude the extra files in that folder.

To add the questions and reports from a single reporter file, include the filepath after the `--add` flag instead of the directory's path
To add the questions and reports from a single Reporter file, include the filepath after the `--add` flag instead of the directory's path
```
$ datums --add "/path/to/file"
```

#### Python
You can add all the reporter files or a single reporter file from Python as well.
You can add all the Reporter files or a single Reporter file from Python as well.

```python
>>> from datums.pipeline import add
Expand All @@ -117,8 +145,9 @@ You can add all the reporter files or a single reporter file from Python as well
... # Add questions first because reports need them
... for question in day['questions']:
... add.add_question(question)
... for report in day['snapshots']:
... add.add_report(report)
... for snapshot in day['snapshots']:
... # Add report and responses
... add.add_snapshot(snapshot)
```
```python
>>> from datums.pipeline import add
Expand All @@ -128,11 +157,12 @@ You can add all the reporter files or a single reporter file from Python as well
>>> # Add questions first because reports need them
>>> for question in day['questions']:
... add.add_question(question)
>>> for report in day['snapshots']:
... add.add_report(report)
>>> for snapshot in day['snapshots']:
... # Add report and responses
... add.add_snapshot(snapshot)
```

You can also add a single report from a reporter file, if you need/want to
You can also add a single snapshot from a Reporter file, if you need/want to
```python
>>> from datums.pipeline import add
>>> import json
Expand All @@ -142,18 +172,18 @@ You can also add a single report from a reporter file, if you need/want to
>>> add.add_report(report)
```

## Updating reports
## Updating reports and responses

If you make a change to one of your Reporter files, or if Reporter makes a change to one of those files, you can also update your reports. If a new report has been added the file located at '/path/to/file', the update will create it in the database.
If you make a change to one of your Reporter files, or if Reporter makes a change to one of those files, you can also update your reports and responses. If a new snapshot has been added the file located at '/path/to/file', the update will create it in the database.

#### Command Line

To update all reports in all the files in your Dropbox Reporter folder
To update all snapshots in all the files in your Dropbox Reporter folder

```
$ datums --update "/path/to/reporter/folder/*.json"
```
and to update all the reports in a single reporter file
and to update all the snapshots in a single Reporter file
```
$ datums --update "/path/to/file"
```
Expand All @@ -169,42 +199,42 @@ From Python
>>> for file in all_reporter_files:
... with open(os.path.expanduser(file), 'r') as f:
... day = json.load(f)
... for report in day['snapshots']:
... update.update_report(report)
... for snapshot in day['snapshots']:
... update.update_snapshot(snapshot)
```
```python
>>> from datums.pipeline import update
>>> import json
>>> with open('/path/to/file', 'r') as f:
... day = json.load(f)
>>> for report in day['snapshots']:
... update.update_report(report)
>>> for snapshot in day['snapshots']:
... update.update_snapshot(snapshot)
```

To update an individual report within a reporter file with
To update an individual snapshot within a snapshoter file with
```python
>>> from datums.pipeline import update
>>> import json
>>> with open('/path/to/file', 'r') as f:
... day = json.load(f)
>>> report = day['snapshots'][n] # where n is the index of the report
>>> update.update_report(report)
>>> snapshot = day['snapshots'][n] # where n is the index of the snapshot
>>> update.update_snapshot(snapshot)
```
#### Changing a Report
> While it is possible to change your response to a question from Python, it's not recommended. Datums won't overwrite the contents of your files, and you will lose the changes that you make the next time you update the reports in that file. If you make changes to a file itself, you may run into conflicts if Reporter tries to update that file.
#### Changing a Snapshot
> While it is possible to change your response to a question from Python, it's not recommended. Datums won't overwrite the contents of your files, and you will lose the changes that you make the next time you update the snapshots in that file. If you make changes to a file itself, you may run into conflicts if Reporter tries to update that file.
> If you do need to change your response to a question, I recommend that you do so from the Reporter app. The list icon in the top left corner will display all of your reports, and you can select a report and make changes. If you have 'Save to Dropbox' enabled, the Dropbox file containing that report will be updated when you save your changes; if you don't have 'Save to Dropbox' enabled, the file containing the report will be updated the next time you export. Once the file is updated, you can follow the steps above to update the reports in that file in the database.
> If you do need to change your response to a question, I recommend that you do so from the Reporter app. The list icon in the top left corner will display all of your snapshots, and you can select a snapshot and make changes. If you have 'Save to Dropbox' enabled, the Dropbox file containing that snapshot will be updated when you save your changes; if you don't have 'Save to Dropbox' enabled, the file containing the snapshot will be updated the next time you export. Once the file is updated, you can follow the steps above to update the snapshots in that file in the database.
## Deleting reports
## Deleting reports and responses

Deleting reports from the database is much the same.
Deleting reports and responses from the database is much the same. Note that deleting a report will delete any responses included in the snapshot containing that report.

#### Command Line
You can delete all reports in your Dropbox Reporter folder with
You can delete all snapshots in your Dropbox Reporter folder with
```
$ datums --delete "/path/to/reporter/folder/*.json"
```
and the reports in a single file with
and the snapshots in a single file with
```
$ datums --delete "/path/to/file"
```
Expand All @@ -219,26 +249,26 @@ $ datums --delete "/path/to/file"
>>> for file in all_reporter_files:
... with open(os.path.expanduser(file), 'r') as f:
... day = json.load(f)
... for report in day['snapshots']:
... delete.delete_report(report)
... for snapshot in day['snapshots']:
... delete.delete_snapshot(snapshot)
```
```python
>>> from datums.pipeline import delete
>>> import json
>>> with open('/path/to/file', 'r') as f:
... day = json.load(f)
>>> for report in day['snapshots']:
... delete.delete_report(report)
>>> for snapshot in day['snapshots']:
... delete.delete_snapshot(snapshot)
```

To delete a single report within a reporter file
To delete a single snapshot within a Reporter file
```python
>>> from datums.pipeline import delete
>>> import json
>>> with open('/path/to/file', 'r') as f:
... day = json.load(f)
>>> report = day['snapshots'][n] # where n is the index of the report
>>> delete.delete_report(report)
>>> snapshot = day['snapshots'][n] # where n is the index of the snapshot
>>> delete.delete_snapshot(snapshot)
```

## Deleting questions
Expand All @@ -257,6 +287,7 @@ You can also delete questions from the database. Note that this will delete any
# Notes

1. This version of datums only supports JSON exports.
2. Photo sets are not supported.

# Licensing

Expand Down
68 changes: 68 additions & 0 deletions alembic.ini
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# A generic, single database configuration.

[alembic]
# path to migration scripts
script_location = datums/migrations

# template used to generate migration files
# file_template = %%(rev)s_%%(slug)s

# max length of characters to apply to the
# "slug" field
#truncate_slug_length = 40

# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false

# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false

# version location specification; this defaults
# to alembic/versions. When using multiple version
# directories, initial revisions must be specified with --version-path
# version_locations = %(here)s/bar %(here)s/bat alembic/versions

# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8

sqlalchemy.url = driver://user:pass@localhost/dbname


# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = WARN
handlers = console
qualname =

[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine

[logger_alembic]
level = INFO
handlers =
qualname = alembic

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
57 changes: 28 additions & 29 deletions bin/datums
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,14 @@
#!/usr/bin/env python
from __future__ import with_statement

import argparse
import glob
import json
import sys
import os

from datums import __version__

from datums import pipeline
from datums.pipeline import add
from datums.pipeline import update
from datums.pipeline import delete

from datums import models
from datums.models import base

Expand All @@ -21,18 +18,20 @@ from datums.models import base
def create_parser():
# Breaking out argument parsing for easier testing
parser = argparse.ArgumentParser(
prog='datums', description='PostgreSQL pipeline for Reporter.', usage='%(prog)s [options]')
parser.add_argument('-v', '--version', action='store_true')
parser.add_argument('--setup', action='store_true',
help='Setup the datums database')
parser.add_argument('--teardown', action='store_true',
help='Tear down the datums database')
parser.add_argument('-A', '--add',
help='Add the reports in file(s) specified')
parser.add_argument('-U', '--update',
help='Update the reports in the file(s) specified')
parser.add_argument('-D', '--delete',
help='Delete the reports in the file(s) specified from the database')
prog='datums', description='PostgreSQL pipeline for Reporter.',
usage='%(prog)s [options]')
parser.add_argument('-V', '--version', action='store_true')
parser.add_argument(
'--setup', action='store_true', help='Setup the datums database')
parser.add_argument(
'--teardown', action='store_true',
help='Tear down the datums database')
parser.add_argument(
'-A', '--add', help='Add the reports in file(s) specified')
parser.add_argument(
'-U', '--update', help='Update the reports in the file(s) specified')
parser.add_argument(
'-D', '--delete', help='Delete the reports in the file(s) specified')
return parser


Expand All @@ -51,26 +50,26 @@ def main():
files = glob.glob(os.path.expanduser(args.add))
for file in files:
with open(file, 'r') as f:
reports = json.load(f)
# Add questions first because reports need them
for question in reports['questions']:
add.add_question(question)
for snapshot in reports['snapshots']:
add.add_report(snapshot)
day = json.load(f)
# Add questions first because responses need them
for question in day['questions']:
pipeline.QuestionPipeline(question).add()
for snapshot in day['snapshots']:
pipeline.SnapshotPipeline(snapshot).add()
if args.update:
files = glob.glob(os.path.expanduser(args.update))
for file in files:
with open(file, 'r') as f:
reports = json.load(f)
for snapshot in reports['snapshots']:
update.update_report(snapshot)
day = json.load(f)
for snapshot in day['snapshots']:
pipeline.SnapshotPipeline(snapshot).update()
if args.delete:
files = glob.glob(os.path.expanduser(args.delete))
for file in files:
with open(file, 'r') as f:
reports = json.load(f)
for snapshot in reports['snapshots']:
delete.delete_report(snapshot)
day = json.load(f)
for snapshot in day['snapshots']:
pipeline.SnapshotPipeline(snapshot).delete()


if __name__ == '__main__':
Expand Down
Loading

0 comments on commit 71848a7

Please sign in to comment.