Dockerize Your Project, Part 1. Nginx, Registry and Jenkins
In this series of articles I am going to show how to dockerize your applications and services, and how to setup continious integration and dockerization with Jenkins.
The first step is to setup a docker registry as a storage of our images and Jenkins CI for the future builds.
Since these two services are independent from the apps I prefer to keep them on separate server. I will use Ubuntu 14.04.4
in the following articles.
Requirements:
docker
docker-compose
Project skeleton
Create a folder for the docker-compose
project in the home directory(~
or whenever you want), I will call it myproject-services
, then cd
to it:
$ mkdir myproject-services
$ cd myproject-services
Create three folders, their content will be used as docker volumes(shared between the host and containers):
$ mkdir data_nginx
$ mkdir data_registry
$ mkdir data_jenkins
Create a docker-compose.yml
file with the following instructions:
$ cat docker-compose.yml
nginx:
image: "nginx:1.9"
ports:
- 80:80
- 443:443
links:
- registry:registry
- jenkins:jenkins
volumes:
- ./data_nginx/:/etc/nginx/conf.d:ro
registry:
image: registry:2
environment:
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /registry_data
volumes:
- ./data_registry:/registry_data
jenkins:
image: "jenkins:1.642.2"
ports:
- 50000:50000
volumes:
- ./data_jenkins:/var/jenkins_home
Here we set nginx
container to expose ports 80
(HTTP) and 443
(HTTPS), and linking it with registry
and jenkins
(it allows nginx
container to access them, more about docker links).
As a security measures, we are going to setup HTTP Basic Auth and SSL for both services(it's required for the registry).
Prepare HTTP Basic Auth
Let's install apache2-utils
package, which contains htpasswd
utility:
$ sudo apt-get -y install apache2-utils
Then cd
into data_nginx
and use it to generate passwords for both services
$ cd ~/myproject-services/data_nginx/
$ htpasswd -c registry.password USERNAME
New password:
Re-type new password:
$ htpasswd -c jenkins.password USERNAME
New password:
Re-type new password:
Prepare SSL certificates
I am going to use letsencrypt.org to get free certificates for my domains. Let's generate them:
$ cd ~
$ git clone https://github.com/letsencrypt/letsencrypt
$ cd letsencrypt
$ ./letsencrypt-auto certonly --standalone -d registry.myproject.com
$ ./letsencrypt-auto certonly --standalone -d ci.myproject.com
It will generate 4 files:
- cert.pem
- chain.pem
- fullchain.pem
- privkey.pem
for each domain in the following folders:
/etc/letsencrypt/live/registry.myproject.com/
/etc/letsencrypt/live/ci.myproject.com/
We need fullchain.pem
and privkey.pem
for nginx
config, so let's create corresponding folders in data_nginx
and copy the certs
$ cd ~/myproject-services/data_nginx/
$ mkdir jenkins_ssl_keys
$ mkdir registry_ssl_keys
$ sudo cp /etc/letsencrypt/live/registry.myproject.com/fullchain.pem /etc/letsencrypt/live/registry.myproject.com/privkey.pem ~/myproject-services/data_nginx/registry_ssl_keys/
$ sudo cp /etc/letsencrypt/live/ci.myproject.com/fullchain.pem /etc/letsencrypt/live/ci.myproject.com/privkey.pem ~/myproject-services/data_nginx/jenkins_ssl_keys/
These files belong to root
, so change the owner to the HOME folder owner:
$ sudo chown $USER:$USER ~/myproject-services/data_nginx/registry_ssl_keys/*
$ sudo chown $USER:$USER ~/myproject-services/data_nginx/jenkins_ssl_keys/*
Nginx configs
Finally we need to prepare nginx
configs for both services. Create registry.conf
and jenkins.conf
files inside data_nginx
folder with the following content:
$ cat ~/myproject-services/data_nginx/registry.conf
upstream docker-registry {
server registry:5000;
}
server {
listen 443 ssl;
server_name registry.myproject.com;
ssl_certificate /etc/nginx/conf.d/registry_ssl_keys/fullchain.pem;
ssl_certificate_key /etc/nginx/conf.d/registry_ssl_keys/privkey.pem;
ssl_trusted_certificate /etc/nginx/conf.d/registry_ssl_keys/fullchain.pem;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
# What Mozilla calls "Intermediate configuration"
# Copied from https://mozilla.github.io/server-side-tls/ssl-config-generator/
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_prefer_server_ciphers on;
# HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
add_header Strict-Transport-Security max-age=15768000;
# OCSP Stapling
# fetch OCSP records from URL in ssl_certificate and cache them
ssl_stapling on;
ssl_stapling_verify on;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
# required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
chunked_transfer_encoding on;
location /v2/ {
# Do not allow connections from docker 1.5 and earlier
# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
return 404;
}
# To add basic authentication to v2 use auth_basic setting plus add_header
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/conf.d/registry.password;
add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;
proxy_pass http://docker-registry;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
}
}
$ cat ~/myproject-services/data_nginx/jenkins.conf
upstream docker-jenkins {
server jenkins:8080;
}
server {
listen 443 ssl;
server_name ci.myproject.com;
ssl_certificate /etc/nginx/conf.d/jenkins_ssl_keys/fullchain.pem;
ssl_certificate_key /etc/nginx/conf.d/jenkins_ssl_keys/privkey.pem;
ssl_trusted_certificate /etc/nginx/conf.d/jenkins_ssl_keys/fullchain.pem;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
# What Mozilla calls "Intermediate configuration"
# Copied from https://mozilla.github.io/server-side-tls/ssl-config-generator/
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_prefer_server_ciphers on;
# HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
add_header Strict-Transport-Security max-age=15768000;
# OCSP Stapling
# fetch OCSP records from URL in ssl_certificate and cache them
ssl_stapling on;
ssl_stapling_verify on;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
# required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
chunked_transfer_encoding on;
location / {
# Do not allow connections from docker 1.5 and earlier
# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
return 404;
}
# To add basic authentication use auth_basic setting plus add_header
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/conf.d/jenkins.password;
proxy_pass http://docker-jenkins;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
}
}
The config is mostly taken from letsencrypt.org nginx sample. In the upstream
stanza we specify linked containers and ports on which these services are running.
Run services
By this time your ~/myproject-services/
directory should have the following structure:
Finally we can start all services with:
$ cd ~/myproject-services/
$ docker-compose up -d
$ docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------------------------------
yourprojectservices_jenkins_1 /usr/local/bin/jenkins.sh Up 0.0.0.0:50000->50000/tcp, 8080/tcp
yourprojectservices_nginx_1 nginx -g daemon off; Up 0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp
yourprojectservices_registry_1 /bin/registry /etc/docker/ ... Up 5000/tcp
$ docker-compose logs
And of course both services should be live and available under
- https://registry.myproject.com
- https://ci.myproject.com
To stop them, type:
$ docker-compose down
Final notes
What you now probably want to do and what I left behind the scene:
- add this project to
git
, ignoring artifacts(certs), but keeping empty folders(data_jenkins
,data_registry
) - add cron job for automated ssl certs renewal(letsencrypt generates certs valid for 3 months)
- add monitoring tools
- add backup system
- add any kind of automation for the host initial setup(
docker
,docker-compose
, cron, monitoring, etc) withchef
,puppet
or anything else
Read next
Dockerize Your Project, Part 2. Rails and PostgreSQL
References
Docker compose reference
How to set up private docker registry by Digital Ocean
Easy cert generation and renewal