Options for easier access / installation of "user data" (from examples/flask)

[emiliom]

@ocefpaf: We've sorted out the installation steps for a live WOFpy setup on an Ubuntu server. See WOFpyODM2LBRproduction_installation_notes.md.

One of the installation steps we'd like to simplify or clarify is the "example" Data Access Object (DAO) code and configuration files for different data source types. This is currently under the examples directory (or really, examples/flask; there's nothing else under examples). These can be viewed as template code and configurations that a user can take as-is and customize, ideally with only small changes to configuration files, not to the code (but a more enterprising user is free to alter the code as well, or write an entirely new DAO). Initially we'll focus on the two odm2 DAO examples, since they are our priority.

The conda package doesn't install these. The instructions we (@lsetiawan and I, mostly @lsetiawan, with a lot of input from @valentinedwv ) wrote ask users to clone the repo and then copy/grab the appropriate examples folder. How can we make this process simpler? You and I had had brief discussions about this previously.

Also, do you have any suggestions for simplifying the steps laid out in those instructions? Though suggestions involving other installation steps probably belong in separate issues.

@valentinedwv, any thoughts on this?

Thanks!

[ocefpaf]

The conda package doesn't install these.

Actually it does, but it is in an awkward directory. You can download https://anaconda.org/odm2/wofpy/2.0.9b0/download/linux-64/wofpy-2.0.9b0-py27_0.tar.bz2 , unpack it, and navigate to the example directory to find it. In a installed version that directory will depended on the Anaconda/miniconda root installation and environment name.

These can be viewed as template code and configurations that a user can take as-is and customize,

That is the problem! If they are configuration they need to be installed in a user space directory.

BTW do you need only runserver_multiple.py or the whole directory structure?

measurement
timeseries
__init__.py
runserver_multiple.py

Also, do you have any suggestions for simplifying the steps laid out in those instructions?

This part in particular I suggest a "deploy" script that will creates the script (or directory structure) on a CLI call. Something like:

odm2deploy my_test

And that will create a directory named my_test with those files where the users has write access.

If you like that idea I can implement it.

Though suggestions involving other installation steps probably belong in separate issues.

The only other suggestion is a docker image pre-configure for that.

[valentinedwv]

Need to access the files in an packaged release. Seems possible
Will need testing.

http://stackoverflow.com/questions/6028000/python-how-to-read-a-static-file-from-inside-a-package
http://stackoverflow.com/questions/3071327/problem-accessing-config-files-within-a-python-egg

[ocefpaf]

Need to access the files in an packaged release.

Accessing files in a packaged release is easy to do but the users needs to write in those files that @emiliom mentioned or did I miss something?

[emiliom]

Thanks, @ocefpaf and @valentinedwv !

Good to know that the conda package already brings in those user/example files, even if they're in an awkward directory!

do you need only runserver_multiple.py or the whole directory structure?

(Note: this is referring to examples/flask/odm2) I think there may different use cases where the whole directory structure is needed, vs only one of the sub-directories (eg, timeseries) is needed. The deploy script should provide options; OR we could copy the whole odm2 directory structure, then simply provide good instructions for users to decide what they will use for their needs.

This part in particular I suggest a "deploy" script that will creates the script (or directory structure) on a CLI call. Something like:
odm2deploy my_test
And that will create a directory named my_test with those files where the users has write access.
If you like that idea I can implement it.

This is where we want to go for sure, but let's discuss more details before moving forward.

Accessing files in a packaged release is easy to do but the users needs to write in those files that @emiliom mentioned or did I miss something?

Yes, the user needs to edit some of those files, even in the simplest of cases. My understanding is that the customized files also need to be copied into the web-server file space.

@lsetiawan, any comments?

[lsetiawan]

This part in particular I suggest a "deploy" script that will creates the script (or directory structure) on a CLI call. Something like:

odm2deploy my_test

And that will create a directory named my_test with those files where the users has write access.

I really like this suggestion by @ocefpaf. I think a cli script would be the simplest way to get users setup with a folder that they can customize to their needs, making changes to the DAO script etc. Then specify the server to use those configuration. This was somewhat similar to the direction we are taking for ODM2Admin...

@ocefpaf , regarding this:

The only other suggestion is a docker image pre-configure for that.

I have started a docker image for the WOFPy installation, this one is test and still have lots of tweaking to do: https://hub.docker.com/r/lsetiawan/wofpy/

[ocefpaf]

@lsetiawan my initial plan is to copy the files that @emiliom mentioned:

(Note: this is referring to examples/flask/odm2) I think there may different use cases where the whole directory structure is needed, vs only one of the sub-directories (eg, timeseries) is needed. The deploy script should provide options; OR we could copy the whole odm2 directory structure, then simply provide good instructions for users to decide what they will use for their needs.

I would go for the whole directory structure and we can add instructions to that directory.

However, in the future we can simplify that further with a single YAML config file (Not sure if that is possible though because I don't really understand all the moving parts of running WOFpy).

@emiliom should I start coding the CLI tool to copy that directory in the user space?

[emiliom]

@emiliom should I start coding the CLI tool to copy that directory in the user space?

Very late reply, but yes, please go ahead.

I'll review your PR #76 (and related PR's) soon.

[emiliom]

@lsetiawan, to help support discussions in #81, please add here a brief listing of:

• what files copied from wof/examples need to be edited by a user (use odm2/timeseries as your example)
• what other files need to be edited or created (wgsi.py, etc?), if they're not from wof/examples
• which ones need to be edited/created for a live vs a test deployment

Please don't bother explaining anything. Your installation docs already do that. I'm looking for something very brief and easy to read. A markdown table would probably do the trick.

Thanks!

[emiliom]

@lsetiawan, just a reminder, in case this got lost among the many other WOFpy github communication. Thanks.

[lsetiawan]

What files copied from wof/examples need to be edited by a user (use odm2/timeseries as your example)

The files that need to be edited by user for odm2/timeseries would be:

• odm2_config_timeseries.cfg for application customization.. I left this file alone because I'm still not sure about all the parts of the customization, but Ideally, users probably want to edit this
• runserver_odm2_timeseries.py for customizing database connections, and typically follow the convention to the production documentation

What other files need to be edited or created (wgsi.py, etc?), if they're not from wof/examples

• wsgi.py This will serve as the entrypoint for uWSGI
• timeseries.ini This is a uWSGI configuration file
• timeseries.conf or timeseries.service depending on init system
• timeseries This is NGINX new server block configuration file

For testing all you need to edit are odm2_config_timeseries.cfg and runserver_odm2_timeseries.py, then just execute runserver_odm2_timeseries.py.

For production, all files listed above should be edited/created. Thanks.

[emiliom]

Great, thanks! Regarding this:

runserver_odm2_timeseries.py for customizing database connections

Is the database connection the only thing that a user needs to customize on that file, typically? If it is, it seems like one setup simplification would be to move the database connection info to the odm2_config_timeseries.cfg. What do you think? On the other hand, based on looking at the test-deployment documentation (which I pasted here), it looks like a production deployment in theory could also be passed the db connection info as a command-line argument?

[lsetiawan]

one setup simplification would be to move the database connection info to the odm2_config_timeseries.cfg. What do you think?

Yea. This could possibly work, I'll take a closer look on configparser library to see if I can implement this.

based on looking at the test-deployment documentation (which I pasted here), it looks like a production deployment in theory could also be passed the db connection info as a command-line argument?

Because of how uwsgi do things, I think it's best not to use any command-line argument on production. The test-deployment documentation shows a command line argument because this is a quick test of the script, it doesn't use wsgi.py at all.

[emiliom]

Because of how uwsgi do things, I think it's best not to use any command-line argument on production.

Ah. Got it.

[emiliom]

A summary of what we (@lsetiawan, @ocefpaf and me) decided on Friday 4/21:

• wofpy_config.py will take two additional arguments (it already takes the copy-to file path): the "example" to be copied, and the deployment type (test vs production)
• split off into a new directory ("wof/examples/flask_notworking") the DAOs (directories) that have not been tested recently and will not be tested soon to ensure they work. For now, we're preserving and focusing only odm2 and odm_1_1) 5/8/2017 note: See rationale here and here.
• for now, the only tests we will get working and include in Travis-CI are test_wof.py and soon test_odm2_dao_sqlite.py. We may also enable test_wofpy_soap.py once I've double checked what it is (test_wof.py is already passing on Travis-CI)
• simplify odm2/timeseries/runserver_odm2_timeseries.py to remove things Don was already removing (he knows what they are!)
• split off the database connection information from "runserver.py" (odm2/timeseries/runserver_odm2_timeseries.py), so it's specified in an external file and most users will never have to edit this file. We didn't decide whether the database connection info will go in the DAO .cfg file or a new external JSON file (reason for having a second separate file is security; though it'd be more convenient to reuse the .cfg file)
• create a new directory under wof/examples to hold templates/samples of the "low level" config files (wsgi, linux init, nginx). These files will be copied by wofpy_config.py when a production deployment is specified (note for Don: I don't know if these files and file names are generic enough that they can be used with any DAO; we should discuss whether any of their content or file names need to be DAO-specific)

See issue #81 for recent, additional related discussions and explanations.

5/1 update: Changed from bullets to a check list, so we can track progress more easily.

[emiliom]

Just to make recent discussion of issuing a new release more prominent ...

@ocefpaf asked on the 26th, "I believe we are OK to issue a new release. What do you think?"

@lsetiawan replied: "I think we need to update the README first before another release to include things about the wofpy_config. Also, I'm working on fixing the DAO's to be able to work with all 3 databases."

And I replied: "Let's wait until I'm back, next week, before issuing a new release. I'd like the chance to review where we are and what was done this week."

I think my answer for now is that a new release should include the items I outlined above, after our Skype call over a week ago. We still have some items left to do.

Thoughts?

[ocefpaf]

• README update is complete in README improvements and wofpy_config example. #109
• timeseries DAO in waiting to be merged Many Changes with the timeseries dao. Now it is using odm2api. #111

I think we should go ahead with #111, pin it for the working version of odm2api, and release without support for Python 3 (still incomplete).

[emiliom]

I think we should go ahead with #111, pin it for the working version of odm2api

If it were working, yeah. But it's got some important problems (#112), so let's hold off.

[lsetiawan]

for now, the only tests we will get working and include in Travis-CI are test_wof.py and soon test_odm2_dao_sqlite.py. We may also enable test_wofpy_soap.py once I've double checked what it is (test_wof.py is already passing on Travis-CI)

As of 5/8/2017 test_odm2_dao_sqlite.py, test_wof.py and test_wofpy_soap.py are now passing.

test_wofpy_soap.py is currently testing against the swis database not odm2, does this need to change at all?

[emiliom]

test_wofpy_soap.py is currently testing against the swis database not odm2, does this need to change at all?

No need to change it at this time.

Wasn't test_odm2_dao_sqlite.py also passing? I'm pretty sure I saw it over the weekend on the Travis-CI list of successful ("passed") tests. Can you double check?

And FYI, there are also a couple of other tests that are passing, according to Travis-CI; though we don't care about them much, it's nice to see.

[emiliom]

@ocefpaf, it looks like the main remaining wofpy task for you, from the most recent list of target tasks, is this:

wofpy_config.py will take two additional arguments (it already takes the copy-to file path): the "example" to be copied, and the deployment type (test vs production)

This is partly dependent on decisions about directory names being implemented in PR #115. Those decisions are not settled, mainly due to me being too busy to review the PR :(. But would you be able to make some progress this week, before your travel starts?

It'll be awesome if we can issue a wofpy release (and conda package) before you head out. Not sure we can do it, but let's try.

[ocefpaf]

But would you be able to make some progress this week, before your travel starts?

Can we have a live meeting to discuss that? If you are too busy it can be @lsetiawan and myself, I have a few questions that may be easier and faster to ask over a call.

[lsetiawan]

@ocefpaf I am available whenever except Thursday. Thanks.

[ocefpaf]

Friday? (I am clocking out now 😩)

[lsetiawan]

Friday's good. :) Enjoy the rest of your day!

[emiliom]

Friday is good. We'll pick a time later.

[emiliom]

For reference, see these recent, important improvements and enhancements:

• PR Implement database string connection in configfile #113, "Implement database string connection in configfile"
• PR Fixed path to templates and DB in .cfg, and clean up runserver #117, "Fixed path to templates and DB in .cfg, and clean up runserver"
• PR Added low-level configs examples #115, "Added low-level configs examples" (setups for nginx, uwsgi, etc)
• PR Modified runserver to be able to use in production #120, "Modified runserver to be able to use in production"

[emiliom]

Note for follow-up: in the current production/server configuration (and also in the previous configuration), the database user & password are found in a file that's nominally exposed to the web. Can we change that in a way that doesn't make the setup much more complicated, and that works equally well for production and test deployments?

[lsetiawan]

the database user & password are found in a file that's nominally exposed to the web.

@emiliom Do you mean the .cfg file? Can't the user just move that to a different directory within the server and just put the absolute path within the runserver script?

[emiliom]

Do you mean the .cfg file?

Yes; previously it was hard-wired in the runserver script, which is obviously also exposed.

Can't the user just move that to a different directory within the server and just put the absolute path within the runserver script?

You tell me! Is that doable? I hope so. If it is, we should probably recommend it in the documentation as a "best practice".

[lsetiawan]

You tell me! Is that doable? I hope so. If it is, we should probably recommend it in the documentation as a "best practice".

Yea. That's doable. Definitely a great recommendation.

[emiliom]

Great.