171/s3 local

[abandoned-prototype]

Status

Ready for review

Description of Changes

Fixes #171 .

This my approach to finish the great work @fritzdavenport was doing to enable us develop stuff locally that depends on S3. The original PR is here: #781

The issue I am addressing here is that we store the url of an image and want to use that url both on the front-end in the browser as well as the backend (inside the docker web-container). See the original PR for more discussion on that issue.
My approach to solve this is run ncat to forward connections to localhost:9000 to minio:9000 which accepts the local mocked-S3 connection. Then I needed to make sure this command is run both in entry_point.sh as well as when running the tests from the Makefile. Definitely interested in learning if there is a better approach to this

Tests and linting

• I have rebased my changes on current develop

• pytests pass in the development environment on my local machine

• flake8 checks pass

[fritzdavenport]

That's so smart!!

I had run in circles trying to get flask to do the nc thing, super smart!

👍

[rbavery]

Hi! I'm trying to spin this version up on my local machine. I get an error when populating the database with test data during the make dev process. Could this be related to running make dev with a previous version of OO before testing this out? This error is still present even when running make clean prior to make dev.

The error:

[*] Populating database with test data...
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "ix_departments_name"
DETAIL:  Key (name)=(Springfield Police Department) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "../test_data.py", line 25, in <module>
    add_mockdata(db.session)
  File "/usr/src/app/OpenOversight/tests/conftest.py", line 252, in add_mockdata
    session.commit()
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/scoping.py", line 162, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 1026, in commit
    self.transaction.commit()
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 493, in commit
    self._prepare_impl()
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 472, in _prepare_impl
    self.session.flush()
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 2451, in flush
    self._flush(objects)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 2589, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 129, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 2549, in _flush
    flush_context.execute()
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
    rec.execute(self)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
    uow,
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
    insert,
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/orm/persistence.py", line 1120, in _emit_insert_statements
    statement, params
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "ix_departments_name"
DETAIL:  Key (name)=(Springfield Police Department) already exists.

[SQL: INSERT INTO departments (name, short_name, unique_internal_identifier_label) VALUES (%(name)s, %(short_name)s, %(unique_internal_identifier_label)s) RETURNING departments.id]
[parameters: {'short_name': 'SPD', 'unique_internal_identifier_label': 'homer_number', 'name': 'Springfield Police Department'}]
(Background on this error at: http://sqlalche.me/e/gkpj)
Makefile:32: recipe for target 'populate' failed
make: *** [populate] Error 1

[rbavery]

I can add a photo to a submission window, so I think this works? I didn't actually check that a new record appeared in the database

Is there a good way to view the database in a browser window or check it with python? Not sure what link to use to query it.