2020 Model Validation (Contingency Plan)

.gitignore

@@ -16,24 +16,7 @@ venv
 ual_baus_install.sh
 configs/hlcm_renter_no_unplaced.yaml
 configs/hlcm_owner_no_unplaced.yaml
-data/2015_06_01_zoning_parcels.csv
-data/2015_08_13_parcels_geography.csv
-data/2015_08_13_zoning_parcels.csv
-data/2015_08_19_parcels_geography.csv
-data/2015_08_29_costar.csv
-data/2015_09_25_parcels_geography.csv
-data/2015_09_25_zoning_parcels.csv
-data/2015_09_30_2_parcels_geography.csv
-data/2015_09_30_parcels_geography.csv
-data/2015_10_05_zoning_parcels.csv
-data/2015_10_06_zoning_parcels.back.csv
-data/2015_10_06_zoning_parcels.csv
-data/2015_10_07_2_parcels_geography.csv
-data/2015_10_07_parcels_geography.csv
-data/2015_12_11_parcels_geography.csv
-data/2015_12_11_zoning_parcels.csv
-data/2015_12_16_2_zoning_parcels.csv
-data/2015_12_21_zoning_parcels.csv
-data/simple_parcels.*
-data/02_01_2016_parcels_geography.csv
-data/2015_12_21_zoning_parcels.csv
+configs/hlcm_renter_lowincome_no_unplaced.yaml
+configs/hlcm_owner_lowincome_no_unplaced.yaml
+.DS_Store
+site/*

.travis.yml

@@ -1,30 +1,25 @@
 language: python
-sudo: false
+
 python:
-- '2.7'
+  - '2.7'
+  - '3.6'
+
+# The Statsmodels line is to resolve a hopefully temporary problem --
+# Statsmodels 0.11 dropped support for Python 2.7, but as of 2020-02-23 is
+# being installed anyway by PyPI in Python 2.7 environments. If this isn't
+# fixed upstream, the best place to address it is in UrbanSim itself, where we
+# can make the Statsmodels version conditional on the Python version.
+
 install:
-- if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
-    wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;
-  else
-    wget http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
-  fi
-- bash miniconda.sh -b -p $HOME/miniconda
-- export PATH="$HOME/miniconda/bin:$PATH"
-- hash -r
-- conda config --set always_yes yes --set changeps1 no
-- conda update -q conda
-# Useful for debugging any issues with conda
-- conda info -a
-# don't think we need all these packages, but copying from urbansim
-- >
-  conda create -q -c synthicity -n test-environment
-  python=$TRAVIS_PYTHON_VERSION
-  cytoolz ipython-notebook jinja2 matplotlib numpy pandas patsy pip scipy
-  statsmodels pytables pytest pyyaml pandana
-- source activate test-environment
-- pip install pep8
-- pip install -r requirements.txt
+  - pip install statsmodels==0.10.*  # see comment above
+  - pip install .  # baus and requirements
+  - pip install coverage pycodestyle pytest  # testing tools
+  - pip list
+
 script:
-- pep8 baus
-- pep8 scripts
-- py.test baus
+  - pycodestyle baus
+  - pycodestyle scripts
+  - coverage run --source baus --module pytest --verbose
+
+after_success:
+  - coverage report

README.md

@@ -1,75 +1,32 @@
-DRAFT Bay Area UrbanSim (BAUS) Implementation
+Bay Area UrbanSim (BAUS) Implementation
 =======
 
-[![Build Status](https://travis-ci.org/UDST/bayarea_urbansim.svg?branch=master)](https://travis-ci.org/UDST/bayarea_urbansim)
+This is the UrbanSim implementation for the Bay Area. Documentation for the UrbanSim framework is available [here](https://udst.github.io/urbansim/). All documentation for Bay Area Urbansim is at: http://bayareametro.github.io/bayarea_urbansim/main/
 
-This is the DRAFT UrbanSim implementation for the Bay Area. Policy documentation for the Bay Area model is available [here](http://data.mtc.ca.gov/bayarea_urbansim/) and documentation for the UrbanSim framework is available [here](https://udst.github.io/urbansim/).
+## Installation
+Bay Area UrbanSim is written in Python and runs in a command line environment. It's compatible with Mac, Windows, and Linux, and with Python 2.7 and 3.5+. Python 3 is recommended. 
 
-* Install Python for your OS ([Anaconda](https://www.continuum.io/downloads) highly suggested)
-* Clone this repository
-* Install dependencies using `pip install -r requirements.txt` (may need to uncomment out pandana)
-* Get data using `python baus.py -c --mode fetch_data` (you will need an appropriately configured AWS credentials file which you must get from your MTC contact)
-* Preprocess data using `python baus.py -c --mode preprocessing`
-* Run a simulation using `python baus.py -c` (default mode is simulation)
+1. Install the Anaconda Python distribution (not strictly required, but makes things easier and more reliable)
+2. Clone this repository 
+3. Create a Python environment with the current dependencies: `conda env create -f baus-env-2023.yml`
+4. Activate the environment: `conda activate baus-env-2023`
+6. Store `run_setup.yaml` next to the repository (ask an MTC contact for access) and use it to specify the `inputs` and `outputs` folder locations and `run_name`
+7. Pull inputs into the model `inputs` folder (ask an MTC contact for access)
+8. Establish an outputs folder in the `outputs` folder using the `run_name` as the folder name
+9. Run `python baus.py` from the main model directory (more info about the command line arguments: `python baus.py --help`)
 
-### An overview of baus.py
-
-baus.py is a command line interface (cli) used to run Bay Area UrbanSim in various modes.  These modes currently include:
 
-* estimation, which runs a series of models to save parameter estimates for all statistical models
-* simulation, which runs all models to create a simulated regional growth forecast
-* fetch_data, which downloads large data files from Amazon S3 as inputs for BAUS
-* preprocessing, which performas long-running data cleaning steps and writes newly cleaned data back to the binary h5 file for use in the other steps
-* baseyearsim which runs a "base year simulation" which summarizes the data before the simulation runs (during simulation, summaries are written after each year, so the first year's summaries are *after* the base year is finished - a base year simulation writes the summaries before any models have run)
+## Optional Slack Messenger 
+* Install the Slack SDK using `pip install slack_sdk`
+* Set environment variable `SLACK_TOKEN = token` (you will need an appropriate slack token from your MTC contact)
+* Set environment variable `URBANSIM_SLACK = TRUE`
 
-### Urban Analytics Lab (UAL) Improvements
 
-#### Data schemas
+## Optional Model Run Visualizer
+* Configure the location that BAUS will write the visualizer files to in `run_setup.yaml` (stored on MTC's Box account for internal visualization)
+* Open the visualizer from the BAUS repository to explore the model run, and/or
+* Open the visualizer from the BAUS repository and publish it to the web (hosted on using MTC's Tableau). At this time runs can be removed from `model_run_inventory.csv` to select the runs to be shown on the web tool
 
-* Builds out the representation of individual housing units to include a semi-persistent tenure status, which is assigned based on characteristics of initial unit occupants
-* Joins additional race/ethnicity PUMS variables to synthetic households [NB: currently missing from the reconciled model, but will be re-added]
-* Adds a representation of market rents alongside market sale prices
 
-#### Model steps
-
-* Residential hedonics predict market rents and sale prices separately, with rents estimated from Craigslist listings
-* Household move-out choice is conditional on tenure status
-* Household location choice is modeled separately for renters and owners, and includes race/ethnicity measures as explanatory variables
-* Developer models are updated to produce both rental and ownership housing stock
-
-Notebooks, work history, code samples, etc are kept in a separate [bayarea_urbansim_work](https://github.com/ual/bayarea_urbansim_work) repository. 
-
-#### Current status (August 2016)
-
-* All of the UAL alterations have been refactored as modular orca steps
-* This code is contained in `baus/ual.py`, `configs/ual_settings.yaml` and individual `yaml` files as needed for regression models that have been re-estimated
-* There are *no* changes to `urbansim`, `urbansim_defaults`, or MTC's orca initialization and model steps
-* MTC and UAL model steps can be mixed and matched by passing different lists to orca; see `run.py` for examples
-* The UAL model steps document and test for required data characteristics, using the [orca_test](https://github.com/udst/orca_test) library
-
-### Outputs from Simulation (written to the runs directory)
-
-ALL OUTPUT IN THIS DIRECTORY IS NOT OFFICIAL OUTPUT. PLEASE CONTACT MTC FOR OFFICIAL OUTPUTS OF THE LAST PLAN BAY AREA.
-
-`[num]` = a positive integer used to identify each successive run.  This number usually starts at 1 and increments each time baus.py is called.
-
-Many files are output to the `runs/` directory. They are described below.
-
-filename |description
-----------------------------|-----------
-run[num]\_topsheet\_[year].csv | An overall summary of various housing and employment outcomes summarized by very coarse geographies.
-run[num]_parcel_output.csv 		| A csv of all new built space in the region.  This has a few thousand rows and dozens of columns which contain various inputs and outputs, as well as debugging information which helps explain why each development was picked by UrbanSim.
-run[num]\_parcel_data\_[year].csv 			|A CSV with parcel level output for *all* parcels with lat, lng and includes change in total_residential_units and change in total_job_spaces, as well as zoned capacity measures.
-run[num]\_building_data\_[year].csv 			|The same as above but for buildings.
-run[num]\_taz\_summarie\s_[year].csv 			|A CSV for [input to the MTC travel model](http://analytics.mtc.ca.gov/foswiki/UrbanSimTwo/OutputToTravelModel)
-run[num]\_pda_summaries\_[year].csv, run[num]\_juris_summaries\_[year].csv, run[num]\_superdistrict_summaries\_[year].csv | Similar outputs to the taz summaries but for each of these geographies.  Used for understanding the UrbanSim forecast at an aggregate level.
-run[runnum]_dropped_buildings.csv     | A summary of buildings which were redeveloped during the simulated forecast.
-run[runnum]_simulation_output.json | Used by the web output viewer.
-
-
-### Directory structure
-
-* baus/ contains all the Python code which runs the BAUS model.
-* data/ contains BAUS inputs which are small enough to store and render in GitHub (large files are stored on Amazon S3) - this also contains lots of scenario inputs in the form of csv files.  See the readme in the data directory for detailed docs on each file.
-* configs/ contains the model configuration files used by UrbanSim.  This also contains settings.yaml which provides simulation inputs and settings in a non-tabular form. 
-* scripts/ these are one-off scripts which are used to perform various input munging and output analysis tasks.  See the docs in that directory for more information.
+## Documentation
+* See the repository's `gh-pages` branch for instructions on installing the BAUS documentation packages and submittings documentation

baus-env-2023.yml

@@ -0,0 +1,22 @@
+# This environment pins dependencies at specific versions, to ensure consistent model
+# behavior. It also pins some of the more important sub-dependencies.
+
+# Install this environment by running `conda env create -f baus-env-2023.yml`
+
+name: baus-env-2023
+
+channels:
+  - conda-forge
+
+dependencies:
+  - python=3.6
+  - numpy=1.17
+  - orca=1.5
+  - orca_test=0.1
+  # - pandana=0.4
+  - pandas<1.0
+  - scikit-learn=0.22
+  - statsmodels=0.11
+  - urbansim=3.2
+  - urbansim_defaults=0.2
+  - pyyaml<6.0