1. Demo
2. Overview
3. Installation
4. Adding deployment to a project
5. Sample Deployment
6. Deployment Steps
7. Automated Testing
8. Licence
9. Customizing build steps
10. Configuration Variables
Example of the project in action[1]
[1] Example of the project in action
Screenshot of a deployment[2]
[2] Screenshot of a deployment
deploy-scripts is a collection of shell scripts to automate the packaging and deployment of projects.
It works similarly to capistrano, but has fewer dependencies.
Apart from the build dependencies of the project you are tying to deploy, it should only need the following dependencies:
- git # Currently only works with git repos - /bin/sh # A Bourne-like shell found on most unix-like systems - openssh-client # An SSH client to connect with remote machines where the software will be deployed - docker # (Optional) The docker command line tools in case the project uses docker - docker-compose # (Optional) The docker-compose tool to build images in case the project uses docker - kubectl # (Optional) The kubectl command line tool in case the project works with kubernetes - aws-cli # (Optional) The AWS command line tool (version <2) in case the project deploys to AWS ECS
The project is still under heavy development and is being updated to include support for other tools.
As much as possible, any new code added to this project should be able to run in a POSIX shell. Exceptions can be made depending on the project you are deploying (eg. bash is ok to use with rails specific parts because a lot of rails related tools need bash anyway).
The following stacks are currently supported
- Java lib projects and Spring Boot (with Maven Wrapper used for building the jar file)
- Ruby on Rails
- Django
- React JS
- Node JS
- Next JS
- PHP
- actix-web (rust web framework)
- Static HTML
Support has also been added to containerize and deploy projects built on the above stacks with:
- Docker
- Kubernetes
- Amazon ECS
deploy-scripts can be installed in any one of the following 2 ways
Just make a git checkout of the stable branch to your home directory with the following command.
git clone --single-branch --branch 0.6.0 https://github.com/masalachai/deploy-scripts.git $HOME/.deploy-scripts/0.6.0
Or download a bootstrap script that will use a dockerized version of deploy-scripts to manage your deployments.
You can save the script to one of the directories in your PATH environment variable for easy execution
# To just download as a shell script curl https://raw.githubusercontent.com/masalachai/deploy-scripts/0.6.0/deploy-scripts.sh -o $HOME/deploy.sh # To download the script and update your PATH variable for easy execution mkdir -p $HOME/.deploy-scripts \ && curl https://raw.githubusercontent.com/masalachai/deploy-scripts/0.6.0/deploy-scripts.sh -o $HOME/.deploy-scripts/deploy \ && chmod +x $HOME/.deploy-scripts/deploy \ && echo 'PATH="$HOME/.deploy-scripts:$PATH"' >> ~/.profile \ && . ~/.profile
Deployment support can be added to a project with the following commands
If you have installed deploy-scripts with the git checkout method mentioned above, you can install the required files to your project directory with the following commands.
cd $HOME/.deploy-scripts/0.6.0/installer # The installer script usage is # sh install.sh [project type] [your project directory] # For example: # For a Java Maven (mvnw) Project sh install.sh java /path/to/java/project # or for a Rails project sh install.sh rails /path/to/rails/project # or for a Django project sh install.sh python /path/to/django/project # or for an actix-web project sh install.sh rust /path/to/actix/project # or for a reactjs project sh install.sh reactjs /path/to/reactjs/project # or for a nextjs project sh install.sh nextjs /path/to/nextjs/project # or for a node project sh install.sh node /path/to/node/project # or for a simple PHP or static HTML site sh install.sh html /path/to/project
If you have installed deploy-scripts as a docker image as mentioned above, you can install the required files to your project directory with the following commands.
# The installer command usage is # sh deploy-scripts.sh --install [project type] [your project directory] # For example: # For a Java Maven (mvnw) Project deploy --install java /path/to/java/project # or for a Rails project deploy --install rails /path/to/rails/project # or for a Django project deploy --install python /path/to/django/project # or for an actix-web project deploy --install rust /path/to/django/project # or for a reactjs project deploy --install reactjs /path/to/reactjs/project # or for a nextjs project deploy --install nextjs /path/to/nextjs/project # or for a node project deploy --install node /path/to/node/project # or for a simple PHP or static HTML site deploy --install html /path/to/project
Follow the instructions given by the installer, if any.
Once the configuration files have been added, you can further configure your deployment by modifying the configuration variables in app-config.sh or the environment-specific config.sh.
To start deploying according to your configured settings, go to your project root and run the following command:
# For deploy-scripts installed via git checkout # Usage: sh deploy/deploy.sh [environment name] # For example for the environment called 'development' sh deploy/deploy.sh development # For deploy-scripts installed via docker # Usage: sh deploy-scripts.sh [your project directory] [environment name] # For example, for deploying the environment named 'development' deploy /my/project development
For a step-by-step understanding of how a deployment happens, and how to add deployment support to a project, please check the Sample Django Deployment[3] wiki page.
- Once added to a project, deploy-scripts copies over the following files:
![[2] Screenshot of a deployment](https://infra.finology.com.my/deploy-scripts-sample.png){kind=link}
- / # project root
- /deploy # the directory where deploy-scripts config files are added (/config/deploy-scripts in rails)
- app-config.sh # Project-wide varibles for deployment configuration
- /environments # multiple environments for the project
- /default # an example environment called default, this can be replaced with your own project environments
- config.sh # environment specific config, values in which can override the ones in app-config.sh
- /assets # files placed here are copied to the server side deployment, dir structure is maintained
# - /docker # (optional) environment specific docker files, can override project-wide files
# - Dockerfile # (optional) if present, will override the one in deploy/docker
# - docker-compose.yml # (optional) if present, will override the one in deploy/docker
# - /kubernetes # (optional) kubernetes configs
# - service.yaml # (optional) config for kubernetes service and deployment for the project
# - /scripts # (optional) Hook scripts that can be used for pre and post build hooks of each deployment step
# - repo.sh # (optional) example of a pre and post repo checkout hook file
# - /docker # (optional) project-wide docker files
# - Dockerfile # (optional) if present, will be used to build the docker image
# - docker-compose.yml # (optional) if present, will be used to build/start the container
It can also use the commented out optional files that are listed, but will work without them for non-containerized deployments.
All projects must have the following vars defined (in app-config.sh or config.sh) so that the relevant steps are executed.
# The project type. Currently supported options are: java, rails, python, node, reactjs and nextjs TYPE=rails # The name that will be used to label the app that is being deployed, commonly the hostname where the service is made available is used SERVICE_NAME=service.com
A typical deployment then proceeds in a number of steps, as follows. Some steps are optional. The expectation is that each of these steps should support several options which can be combined for different types of deployments. The number and order of steps can be varied and removal or addition of more steps to the build process can be done through Customizing build steps
1. Repo Checkout
2. Build
3. Format
4. Package
5. Push
6. Post-push
The desired branch is cloned or checked out into a working directory (deploy/.build/repo by default) and the deployment is done from that checkout.
Currently only git repos are supported.
Variables:
# The repository to clone REPO=git@github.com:namespace/repo.git # The branch to deploy. Typically set per environment in the corresponding config.sh GIT_BRANCH=default
Optional. In this step, the code may be compiled if needed, to produce the files that will be deployed.
Variables:
# The tool that will build the project. It varies by the project type (set by the TYPE variable) BUILD=mvnw
Optional. The files to be deployed are assembled in the required structure to be sent to the target server. By default it's done in the deploy/.build/package directory.
Variables:
# The tool that will assemble the files for deployment. It varies by the project type (set by the TYPE variable) FORMAT=rails
The assembled files in deploy/.build/package are then put into a packaged form that can be delivered to the server. The most common form we use is a new git repo, which is pushed to the target server (or another git repo), but it can take options to package it as a container as well
Variables:
# How to package the files to be deployed to be sent to the server PACKAGE=git
Optional. The packaged files can then be sent to the deployment server in a number of ways. The default one is a bare git repo on the server that will receive the files and run a **post-receive** hook to deliver it to the deployment directory.
# The method used to push the packaged files PUSH=git-bare # The target server (can also be a git repo URL) DEPLOYMENT_SERVER=service.com # Credentials used when pushing with git DEPLOYMENT_SERVER_USER=user # The SSH port being used in case of a git push DEPLOYMENT_SERVER_PORT=22
Optional. Sometimes (currently only in the case of kubernetes, ecs, or deploying a docker image from a registry), you need to perform some actions to update the deployment with the new version from the machine you are deploying from. Those can be performed in this step
# The steps that will be executed in the post push stage PUST_PUSH=kubernetes
The project contains automated tests to verify that deployments don't break when new changes are made to deploy-scripts.
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
Currently, deployments for the following project types can be tested
# Test Java (Spring Boot) deployment sh tests/java.sh # Test Ruby on Rails deployment sh tests/rails.sh # Test React JS deployment sh tests/reactjs.sh # Test Next JS deployment sh tests/nextjs.sh # Test Django deployment sh tests/django.sh # Test Static HTML and PHP website deployment sh tests/php.sh # Test actix-web deployment sh tests/actix.sh # Test Node JS deployment sh tests/node.sh # Test deployment with dockerization on remote host (dockerizes sample Django project) sh tests/docker.sh # Test deployment with push to docker registry and pull from registry to a remote host sh tests/docker-pull.sh
The code is distributed under the MIT License, a copy of which is included in the project repository.
The steps that are run during a deployment are set by the STEPS variable. Its is a shell variable with a space separated list of step names, whose default value is STEPS="repo build format package push post_push". A normal deployment proceeds along these steps, provided the proper configuration variables are set correctly.
Additional step(s) can be added to a deployment by defining shell variables with the BEFORE_ or AFTER_ prefixes in front of existing build step names to insert a step into the list of existing steps.
For example, to add a custom step called test before build, you can add a variable BEFORE_build="test" to config.sh or app-config.sh.
BEFORE_build="test"
And define the logic to be executed during this step inside a ds_exec_step() function inside either the deploy/scripts/steps/test.sh file (for project wide step) or inside deploy/environments/[environment name]/scripts/steps/test.sh (for environment specific step). The environment specific logic, if present, will override the project-wide one.
All the variables defined inside app-config.sh and config.sh are available in this function.
# deploy/scripts/steps/test.sh
ds_exec_step() {
title 'test step'
printf "Running tests for $SERVICE_NAME ... "
}
Complex builds and deployments can be configured containing dependencies from other projects, conditional logic while preparing the deployment, etc. with custom steps.
Simple deployment configuration is controlled almost entirely through shell variables, which can be defined in the project-wide app-config.sh or the environment-specific config.sh. Environment-specific variables override project wide variables.
Here is a list of variables that are configurable
Project type.
Currently allowed values:
- java
- rails
- python
- node
- reactjs
- nextjs
- rust - currently only tested with actix-web
- html - for static HTML sites or simple PHP sites
Build type depending on the project type variable.
Currently allowed values:
- mvnw - for java projects
- cargo - for rust projects
- npm - for reactjs and nextjs projects
The format that files should be arranged in for deployment.
Currently allowed values:
- spring-boot - for java projects
- rails - for rails projects
- node - for node projects
- django - for python projects
- reactjs - for reactjs projects
- nextjs - for nextjs projects
- actix-web - for actix-web projects
- static - for static HTML sites or simple PHP sites
An optional space-separated list of files and directories relative to the project root to be copied over to the deployment files being readied for packaging
The git repo from which to clone the project.
The branch which should be used for the deployment. If left unset, it will use the local git branch for deployment
The format in which to package the deployment files.
Currently allowed values:
- git - default value. The default method is to create a bare git repo on the deployment server and push the deployment files to it and use a post-receive git hook to set up and start the deployed project. The files are packaged as a git repo which is then pushed to the bare repo
- docker - to package the files into a docker image. Requires a Dockerfile and a docker-compose.yml file to be supplied
- zip - uses the zip program to archive the files to be packaged into a .zip archive
The method to deliver the deployment to the destination.
Currently allowed values:
- git-bare default value. By default the deployment files are pushed to a bare git repo on the deployment server and a post-recieve hook is used to set up and start the project
- docker - to push the docker image built when PACKAGE=docker to a docker registry
- s3 - to push to AWS S3. Currently only works with PACKAGE=zip. See S3_BUCKET_PATH variable on how to set the target S3 bucket.
The step to execute after the deployment files are pushed to the destination.
- docker-pull - Uses the docker-compose.yml file supplied to pull the image and start the container on a remote host
- kubernetes - creates (or updates) a kubernetes service and deployment with the built docker image
- ecs - restarts an existing Amazon ECS task
The name to identify the service being deployed. For example my-project.
Deduced from the name of the directory under environments/. No need to explicitly set.
Space-separated list of persistent directories to symlink the deployment to. Usually used when sharing persistent directories between deployments.
Space-separated list of persistent files to symlink the deployment to. Usually used when sharing persistent files, like configuration or login/access credential files between deployments.
The hostname or IP address of the server to deploy to.
The SSH user to use to connect to the DEPLOYMENT_SERVER for transferring the deployment files. Default value is deploy
The SSH port on the remote server to deploy to. Default value is 22.
The directory on the remote server to which the project must be deployed. The default value is $HOME/sites/$SERVICE_NAME/$PROJECT_ENVIRONMENT.
The directory under the project directory where deploy-scripts files are maintained. The default value is deploy/. For rails projects, the default value is config/deploy-scripts.
Update deploy-scripts before deploying project. Default value is true.
The type of repo to fetch the code from. Default and only currently supported value is git.
The shell command to execute to start the deployed service. Default value is sh deploy/run.sh restart, which uses the bundled run.sh script which can start/stop spring-boot, rails, and django projects.
The no of previous releases to keep on the remote server.
When set to true, it will use the supplied Dockerfile and docker-compose.yml on the remote server to build an image from the deployed files and start a container with it.
When PUSH=docker, the target docker registry it should push to is specified by the DOCKER_REGISTRY variable.
When PUSH=docker, the directory containing the config.json from which the docker registry login credentials are read.
Default value is false. When set to true, the built image will be deleted from the local system after it is pushed to the docker registry. **To use cached docker layers for faster image builds, keep this variable set to false**.
When POST_PUSH=kubernetes, the kubernetes secret that contains the credentials to pull docker images from a private registry. If your DOCKER_REGISTRY is private and needs a username and password to access, you must create a kubernetes secret with the credentials and set this variable.
When POST_PUSH=kubernetes, the directory containing the kubernetes cluster configuration yaml files needed to connect to and manage the cluster you are deploying to. Default value is $HOME/.kube.
When POST_PUSH=kubernetes, the identifier for the kubernetes cluster you are deploying to. deploy-scripts will search for a yaml file named with this identifier to use to connect to the cluster. For example, if is variable is set to my-cluster, it will look for a my-cluster.yaml in the KUBERNETES_HOME directory for the cluster configuration.
When POST_PUSH=kubernetes, you can explicitly specify the path of the cluster yaml file. If unspecified it will try to locate the yaml file at $KUBERNETES_HOME/$KUBERNETES_CLUSTER.yaml.
The kubernetes namespace under which the service should be deployed. If unspecified, the service is deployed under the default kubernetes namespace.
When POST_PUSH=kubernetes, the nginx ingress service being used as a load balancer for your kubernetes services. Currently only HTTP(s) load balancing is set up.
When POST_PUSH=kubernetes, this can be used to explicitly specify the hostname to use for the service. If unspecified, it will try to use SERVICE_NAME as the hostname for the service.
The service port to use for the deployed kubernetes service. Default value is 80.
When POST_PUSH=kubernetes, the certificate manager that has been configured for certificate issue and renewal on your kubernetes cluster, if you want to enable HTTPS for your deployment.
When POST_PUSH=kubernetes, whether to enable HTTPS for your deployment. Default value is false. If set to true, you must also set the KUBERNETES_CERT_MANAGER variable.
When POST_PUSH=kubernetes, the number of replicas to enable for your kubernetes service. Default value is 1.
When POST_PUSH=ecs, the name of your Amazon ECS cluster where your task is running.
When POST_PUSH=ecs, the name of your Amazon ECS task that you need to restart after the deployment.
When POST_PUSH=ecs, whether to kill the currently running task before starting a new one. Default value is false
The value to use for the --profile argument for aws cli. The default value is default
When PACKAGE=zip, the resulting zip file can be pushed to an S3 bucket, if the proper set of credentials for aws cli are configured in your home directory. A sample path would look like S3_BUCKET_PATH="my-bucket/prefix1/prefix2/etc". The path shouldn't contain the s3:// protocol string.