Skip to content

Latest commit

 

History

History
101 lines (57 loc) · 3.9 KB

setup_ibis.md

File metadata and controls

101 lines (57 loc) · 3.9 KB
 ▄█  ▀█████████▄   ▄█     ▄████████
███    ███    ███ ███    ███    ███
███▌   ███    ███ ███▌   ███    █▀
███▌  ▄███▄▄▄██▀  ███▌   ███
███▌ ▀▀███▀▀▀██▄  ███▌ ▀███████████
███    ███    ██▄ ███           ███
███    ███    ███ ███     ▄█    ███
█▀   ▄█████████▀  █▀    ▄████████▀

Setup your IBIS

Follow the below 3 steps:-

  1. Update IBIS Property file

    For IBIS to work and communicate with Hadoop (Sqoop, Hive and Oozie), the following respective env property, corresponding job property and configuration files should be updated:

    A. Environment property file

    dev property file

    Some other property files specifically used for unit test

    Refer here How to Update property file?

    B. Environment's job property file

    dev job properties

    Refer here How to Update job's property file?

    C. Configuration properties

    dev configuration property file

    Refer here How to Update configuration file?

  2. Run IBIS one time setup and install all system dependencies

    Setup.sh will execute below to commands to install system dependencies. VENV WILL FAIL IF BUILT WITHOUT ZLIB SUPPORT AND CERTAIN DEPENDENCIES.

    But it is recomemded to run the below two commands manually, if not done setup.sh will be installing all the dependencies. Below command is for RHEL, if you are not on RHEL you will need to install all the corresponding dependencies manually.

    sudo yum groupinstall -y "development tools"

    sudo yum install krb5-devel gcc zlib-devel gcc-c++ python-devel cyrus-sasl-devel openssl openssl-devel libffi-devel bzip2-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel snappy-devel expat-devel patch -y --skip-broken &&

    To prepare the IBIS Env (create the required directories, DB and tables used by IBIS), run the following Env setup shell:

    sh setup.sh

  3. Build IBIS

    Build the IBIS project to create the Python egg, coverage report and the packaged ibis.tar.gz file containing the egg with the required IBIS supporting files

    To build the code, navigate to the ibis_build folder and use the following command:

    sh build.sh <Argument1: ibis_home> <Argument2 : Execution option>

    Argument 1 : Needs to be IBIS home directory path

    Argument 2 : Below are the acceptable argument values

         No arguments or leaving Blank will execute Test cases, Code check and Build process.
         skip-all-test - Skip all validation and create Egg
         skip-code-check - Skip code style check
         skip-build - Run all validations and skip build process

Refer Test and Build for more details on building IBIS and running unit tests independently

IBIS Dependencies and prerequisites for IBIS setup

1. Installation requirements

  • Unix like environment (Mac, Linux)

  • Python 2.7.4+

  • requirements.pip (Is executed in IBIS one time setup)

  • graphviz dot (a yum install)

  • Hadoop ecosystem (Oozie, Hive/Impala, Sqoop)

  • Git repos

    • ibis (this one!)

    • ibis-workflows - Storing the workflows generated by IBIS

2. Back-end tables (Hive/Impala)

/resources/ibis.hql

Alt text