NEOTREE METABASE EXPORTS

OVERVIEW

This is a python code base used to export dashboard questions as single images and also produce a json file containing all the image paths including bundled dashboards containing more than 1 image.This code base was tested against the Ubuntu and Windows 10 operating systems. The code works by first of all clearing all existing images then reexport the images again. The reason for clearing the images is to effect any changes that would have happened to the images since the last time the exporting was done

PREREQUISITES

==Python Version 3.XX or Higher == Working Link to an existing metabase site == Working Connection including credentials to an existing postgres database for the metabase site == The Metabase Secret Key

RULES AND GUIDELINES

In order to get the best out of the source code:

• Don't remove any lines from the .gitignore file we provide
• Don't commit any credentials or your local configuration to your repository. Keep all your credentials and local configuration in common_files/database.ini

INSTALLING DEPENDENCIES

At this stage the assumption is that we have cloned the code base to our local machine

1. cd into cwd ie (Current Working Directory or Project Directory)
2. Create a virtual environment using virtualenv,pipenv or any other virtual environment creation dependency name it env
3. Activate the env virtual environment
4. Run pip install -r requirements.txt
5. If you face any error, it is recommended to install the erronous dependencies independently using pip

• All dependencies are declared in requirements.txt

CONFIGURATIONS

LOCAL CONFIGURATION:

The common_files folder should be used for configuration that is either user-specific (e.g. IDE configuration) or protected (e.g. security keys).

• Note: Please do not check in any local configuration to version control. In the case of this repository, this folder contains the important file known as database.ini The database.ini file contains all sensitive and user specific configurations some which are optional and some which are required

Configuration Items:

• Note: You can define multiple Environments in the same database.ini file

1. Section: Determines The Environment To Be Used, Valid section Names Are ([postgresql_dev],][postgresql_stage] and [postgresql_prod])

The rule to name a section implies that it should have postgresql followed by an underscore _ then the environment name (dev,stage or prod) All enclosed in square brackets [] e.g [postgresql_dev]

2. host : The host for the postgres database for the metabase site e.g localhost -- REQUIRED FIELD

3. database: The database name for the metabase site e.g neotree_data -- REQUIRED FIELD

4. user : The username used to connect to the database e.g neotree_postgres_user -- REQUIRED FIELD

5. password: Connection password for the database user e.g connection_pw1@5Fi -- REQUIRED FIELD stage*

6. export_dir: The directory to which we want to export our images to. This should be specified following the operating system format of naming directories, it should exist and should have enough permissions. If the export_dir is not specified, the assumption is that we are running an ubuntu instance and the user is ubuntu and the export_dir will default to /home/ubuntu/metabase_exports/, hence make sure the directory exists before running the code.

7. cron_interval: An OPTIONAL number value used to determine the number of hours to be used before the next running of the automated metabase images exportation. If not specified it will default to 12 hours.

8. metabase_secret: The secret key generated by metabase (You can only get this from your metabase site administrator)

9. metabase_url: The url to the metabase site

EXAMPLE OF FULL database.ini FILE:

[postgresql_dev]
host= localhost
database=  neotree_metabase
user= user
password= password
#DIRECTORY TO EXPORT IMAGES TO#
export_dir = C:\/Users\/morris\/Documents\/Images ### WINDOWS EXAMPLE, FOR OTHER OSs USE THE OSs PATH PATTERNS       
cron_interval = 5 
metabase_secret: secret45ec56f6uyu787uioii
metabase_url: https://my_metabase.co.za

RUNNING THE PROJECT

At this point the assumption is that:

1. You have installed all dependencies
2. You have a working metabase site, the right dashboard id or ids, the metabase secret key
3. You have a working postgres database, and a valid username and password to connect to the database,
4. You have setup your configurations in database.ini
5. You have activated the virtual environment that you created earlier and (MAKE SURE THIS IS THE VIRTUAL ENVIRONMENT IN WHICH YOU HAVE INSTALLED YOUR DEPENDENCIES)

If all the above mentioned requirements are met then:

1. cd into the Project directory
2. Run the command python metabase_export.py specified_env where specified_env is the environment as specified in the SECTION part of the database.ini file. Using the example of the database.ini specified in the section EXAMPLE OF FULL database.ini FILE my command will be as follows: python metabase_export.py dev

After running the above command, logs should start appearing on your screen, detailing the steps that are being run.

After a successful running of the metabase exports, navigate to the exports_dir and check if your images exists, as well as the json file metabase_exports.json, which contains images paths and is used for bundling the images.

AUTOMATION

Currently this has been tested against Ubuntu Within the repository there is an automation script, which writes cron jobs to the cron service

Before Running The Automation Script:

1. Make sure that the cron service is running on your machine
2. Make sure that the currently logged in user is added to the cron users(Doing This Requires Root access)
3. Make sure the name of your virtual environment is env as there is a reference of the virtual environment name in the automation script

If the above are satisfied :

1. cd into the project directory
2. activate your virtual environment
3. Run the command python metabase_export_cron.py specified_env where specified_env is the environment as specified in the SECTION part of the database.ini file. e.g python metabase_export_cron.py dev or python metabase_export_cron.py stage Note: You can run the automation script on the same machine for all the 3 available environments i.e (dev,stage,prod) if the configurations are available.All you have to do is to run the command in point 3 changing the variable specified_env to suit the required environment.
4. To confirm that your entries have been written to the crontab file, run crontab -e then check if your entries are available

It is important to specify the time zone in the crontab file before starting to run the automation script so that you won't have challenges with differences in server time against the time zone that you want the automation script to run. To set the time zone append the following line at the top of your crontab file: TZ="SPECIFY_TIMEZONE e.g TZ= "Africa/Harare"

ALTERNATIVELY

If you have knowledge with the linux operating system, you can write the automation command directly to the cron service by following the the steps below:

1. Make sure that the cron service is running on your machine
2. Make sure that the currently logged in user is added to the cron users(Doing This Requires Root access)
3. Run crontab -e and a file to write your automation command will be opened. *Please Note:**When you first open the file, it will prompt you on the text editor that you want to use for opening the file, and that will be used as the default editor during subsequent opening of the file
4. Set Time zone as specified in the section above.
5. Write your automantion command, save your changes and exit.

THE AUTOMATION SCRIPT COMMAND

The automation command follows the general cron format containing the time,as well as the command to be run The time is specified in the format mm hh dd MM yy where : mm= Minutes and takes values 0 to 59 or * for every minute or */2 to 59 for any minute which is divisible by the number specified in the denominator hh= Hours and takes values between 0 to 23 or * for every hour or */2 to 23 for any hour which is divisible by the number specified in the denominator MM = Months and takes any value from 1 to 12 or * for every month or */2 to 12 for any month which is divisible by the number specified in the denominator YY = For the year

EXAMPLES

1. 0 */4 * * * implies that the script runs at on the zeroeth minute after every 4 hours, every day , every month, every year
2. 30 * * * * implies that the script runs every 30 minutes, every hour, every day, every month, every year

. The complete automation script command, contains :

1. The Time
2. getting into the project directory through the cd command
3. The python command to run the metabase_export.py, specifying the environment specified in the database.ini file

EXAMPLE OF THE COMPLETE AUTOMATION SCRIPT COMMAND

0 */6 * * * cd /home/ubuntu/neotree-metabase-exports env/bin/python metabase_export.py specified_env where specified_env is an environment specified in the database.ini file. NOTE: Your python environment should be named env else change the command env/bin/python to suit the name of your environment NOTE: The automation scripts should be set or run after all the setup process including the installation of dependencies has been completed

LOGS

By default on Ubuntu the log file is found in /var/log directory and it is called metabase_export.log.

REMOTE ACCESS TO EXPORTED IMAGES

1. Please ensure that the directory containing the exported content is password protected For Nginx password Authentication can be set up on the directory containing the exported diagrams. The following guide can be used: https://www.digitalocean.com/community/tutorials/how-to-set-up-password-authentication-with-nginx-on-ubuntu-14-04
2. Once completed, browse to the location where the images are exported You should have a pop-up window where you can enter your username and password. The content should only show, if the correct credentials were provided