-
Notifications
You must be signed in to change notification settings - Fork 252
Integrate with Teletraan
- Add Deploy Scripts Teletraan can deploy services written in any programming language, as long as any of the following deploy scripts packaged into your service build artifacts:
-
PRE_DOWNLOAD
Before deploy agent download the build. If build download might affect your service in anyway, you might want to use this opportunity to shutdown your service first. In general this script is seldom used. -
POST_DOWNLOAD
After deploy agent download the build. This script is also seldom used. -
PRE_RESTART
Before deploy agent restart the service. Usually this is the time to unregister with load-balancer or service discovery, drain traffic, basically anything needs to be done before shutdown your service. -
RESTARTING
Deploy agent is restarting the service. Could be just start, if service has already been stopped previously -
POST_RESTART
After deploy agent restart the service. Usually this is the time to warm up cache, run smoke tests or any sanity check, register back load-balancer or service discovery, and start to receive traffic etc.
Although being referred as Deploy Scripts, they can be written in any programming language, as long as they are executable. Also, even though we provided some recommendations above regard what each script should do, but you are free to use them to do just about anything you need. Sky is the limit when it comes to these Deploy Scripts.
It is important that these scripts use the exact same names as above, and they are put under a specific directory teletraan directly. The teletraan directory will have to be top-level directory in the artifact tar ball. For example:
>tar tvf servicex-adb87dc.tar.gz
lib/servicex.jar
teletraan/RESTARTING
teletraan/POST_RESTART
Sometimes the same artifact could be used to run different services with different configurations. In this case, create different directories under the top-level teletraan with the deploy environment names, and put the corresponding deploy scripts under the proper environment directories separately. For example:
>tar tvf service-adb87dc.tar.gz
lib/service.jar
teletraan/serverx/RESTARTING
teletraan/serverx/POST_RESTART
teletraan/servery/RESTARTING
- Host level deploys take the following steps:
UNKNOWN->PRE_DOWNLOAD->[DOWNLOADING]->POST_DOWNLOAD->[STAGING]->PRE_RESTART->RESTARTING->POST_RESTART->SERVING_BUILD
As you might have already guessed, deploy agent executes the corresponding deploy scripts (exact match) in each of the steps except DOWNLOADING and STAGING, which are reserved by Teletraan to download the build artifacts, untar, create symbolic links and change permissions etc. It is generally not recommended to mess with these build artifacts, directories, links manually on host.
- Pass Environment Variables to Deploy Scripts
Deploy agent passes the following environment variables to Deploy Scripts when invoking them:
- Environment Related
- ENV_NAME
- STAGE_NAME
- Deploy Related
- DEPLOY_ID
- DEPLOY_STEP
- OPCODE
- DEPLOY_TYPE
- Build Related
- BUILD_COMMIT
- BUILD_NAME
- BUILD_REPO
- BUILD_BRANCH
- BUILD_ID
- BUILD_URL
- Deploy Agent Config Related
- TARGET
- BUILDS_DIR
You can also define a set of Environment Variables through Script Config in the deployboard UI. They will be passed to all the Deploy Scripts when executing them. It gives Teletraan server extra power to control deploy agents and the services. For example, one can define LOG_LEVEL in the Script Config section on deployboard. Without creating a new build and deploy, one can simply change the LOG_LEVEL on deployboard, restart the current deploy, and the service will start to generate different level logging.
Any scripts or config files with .tmpl suffix under teletraan directory will be examined by Deploy Agent in STAGING step. Any variables in these files in the form of {$TELETRAAN_VALUE1:default} will either be replaced by Script Config in the deployboard UI, or if missing in the UI it will use the default value. This feature is important when these .tmpl files will be used outside Deploy Agent scripts scope. Once transformed, the .tmpl suffix will be dropped.
Deploy Agent can be run either in a long-running daemon mode or a command mode, which will exit for each run. It is up to you to decide which way to use to best fit your infrastructure. At Pinterest we use cronjob to run it every minute:
* * * * * /mnt/deployd_venv/bin/deploy-agent -f /etc/deployagent.conf >/var/log/deployd/deploy-start.log 2>&1
Deploy Agent uses config file to customize. A complete list of config directives could be found at here. In particular the following directives are worth to be reiterated here:
- deploy_agent_dir
This is the directory Deploy Agent put its important deploy status files. In particular:- env_status
- host_info
- builds_dir
This is where Deploy Agent stores every build artifacts and staging area. It will keep up to num_builds_to_retain builds. More importantly, it will maintain a symbolic link /mnt/<ENV_NAME> points to the latest build. As you might have guessed, rollback to the version still on the host will simply switch the symbolic link to point to the older build bypassing the DOWNLOADING and STAGING steps. - log_directory
This is where Deploy Agent keeps all its logs. Particularly, deploy-agent.log is the Deploy Agent's own log, and <ENV_NAME>.log is the log of executing service Deploy Scripts. - target_default_dir
This is where Deploy Agent stores every installed builds. - teletraan_service_url
The URL to connect to backend deploy-service - min_running_time
This is the frequency when Deploy Agent talks to server every X seconds. Default is 60 seconds. - max_sleep_interval
The maximum number of seconds to sleep before proceeding to the next Deploy step. Default is 60 seconds. - user_role
This is the user name to execute the Deploy Agent to give itself correct system permission. Default is the user who executes the Deploy Agent. - max_retry
The maximum number of times to retry deploy steps. Default is 3 times. - num_builds_to_retain
The number of packages to retain under the <builds_dir> directory. Default is 2 builds. - max_tail_bytes
The maximum number of bytes to store the script error message. Default is 10240 bytes.
One can certainly associate individual hosts with a particular Deploy Environment, but the real power of Teletraan comes when associating a cluster of, or a group of, hosts with a particular environment. Whenever new hosts are provisioned to be part of certain group, Teletraan will make sure these new hosts running on the same version of code as the rest of the existing members in that group. See Capacity Config for more details on how to associate hosts and groups with certain deploy environment.
To make Teletraan less opinionated on how Cluster or Group being defined, and to make Teletraan easily integrates with different Configuration Management systems and Cloud technologies, we decided to have the host provisioning process to provide a host_info File on each host in the very beginning. for example:
>cat /mnt/deployd/host_info
groups=prod,servicex
id=i-dcbcee23
host=service-x-001
We believe this file should be easily produced by most of the provisioning process. Notice that there could be multiple groups for a host, which basically makes a group hierarchy. Teletraan supports deploying multiple services onto the same host. Combining with the group hierarchy, Teletraan provides a powerful way to deploy services to different scope of hosts. In the above example, a more basic service could be deployed to all prod hosts, including all the hosts belong to servicex.
The location of this host_info File can be customized through Deploy Agent config file.
We noticed that have group membership directly supported by Teletraan has it usage. We would support this feature, through Teletraan server side tagging, or through other means, in the future.