Rustam Gasanov / Tech

Dockerize Your Project, Part 1. Nginx, Registry and Jenkins

In this series of articles I am going to show how to dockerize your applications and services, and how to setup continious integration and dockerization with Jenkins.

The first step is to setup a docker registry as a storage of our images and Jenkins CI for the future builds.

Since these two services are independent from the apps I prefer to keep them on separate server. I will use Ubuntu 14.04.4 in the following articles.

Requirements:

  • docker
  • docker-compose

Project skeleton

Create a folder for the docker-compose project in the home directory(~ or whenever you want), I will call it myproject-services, then cd to it:

$ mkdir myproject-services
$ cd myproject-services

Create three folders, their content will be used as docker volumes(shared between the host and containers):

$ mkdir data_nginx
$ mkdir data_registry
$ mkdir data_jenkins

Create a docker-compose.yml file with the following instructions:

$ cat docker-compose.yml
nginx:
  image: "nginx:1.9"
  ports:
    - 80:80
    - 443:443
  links:
    - registry:registry
    - jenkins:jenkins
  volumes:
    - ./data_nginx/:/etc/nginx/conf.d:ro
registry:
  image: registry:2
  environment:
    REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /registry_data
  volumes:
    - ./data_registry:/registry_data
jenkins:
  image: "jenkins:1.642.2"
  ports:
    - 50000:50000
  volumes:
    - ./data_jenkins:/var/jenkins_home

Here we set nginx container to expose ports 80(HTTP) and 443(HTTPS), and linking it with registry and jenkins(it allows nginx container to access them, more about docker links).

As a security measures, we are going to setup HTTP Basic Auth and SSL for both services(it's required for the registry).

Prepare HTTP Basic Auth

Let's install apache2-utils package, which contains htpasswd utility:

$ sudo apt-get -y install apache2-utils

Then cd into data_nginx and use it to generate passwords for both services

$ cd ~/myproject-services/data_nginx/
$ htpasswd -c registry.password USERNAME
New password:
Re-type new password:
$ htpasswd -c jenkins.password USERNAME
New password:
Re-type new password:

Prepare SSL certificates

I am going to use letsencrypt.org to get free certificates for my domains. Let's generate them:

$ cd ~
$ git clone https://github.com/letsencrypt/letsencrypt
$ cd letsencrypt
$ ./letsencrypt-auto certonly --standalone -d registry.myproject.com
$ ./letsencrypt-auto certonly --standalone -d ci.myproject.com

It will generate 4 files:

  • cert.pem
  • chain.pem
  • fullchain.pem
  • privkey.pem

for each domain in the following folders:

  • /etc/letsencrypt/live/registry.myproject.com/
  • /etc/letsencrypt/live/ci.myproject.com/

We need fullchain.pem and privkey.pem for nginx config, so let's create corresponding folders in data_nginx and copy the certs

$ cd ~/myproject-services/data_nginx/
$ mkdir jenkins_ssl_keys
$ mkdir registry_ssl_keys
$ sudo cp /etc/letsencrypt/live/registry.myproject.com/fullchain.pem /etc/letsencrypt/live/registry.myproject.com/privkey.pem ~/myproject-services/data_nginx/registry_ssl_keys/
$ sudo cp /etc/letsencrypt/live/ci.myproject.com/fullchain.pem /etc/letsencrypt/live/ci.myproject.com/privkey.pem ~/myproject-services/data_nginx/jenkins_ssl_keys/

These files belong to root, so change the owner to the HOME folder owner:

$ sudo chown $USER:$USER ~/myproject-services/data_nginx/registry_ssl_keys/*
$ sudo chown $USER:$USER ~/myproject-services/data_nginx/jenkins_ssl_keys/*

Nginx configs

Finally we need to prepare nginx configs for both services. Create registry.conf and jenkins.conf files inside data_nginx folder with the following content:

$ cat ~/myproject-services/data_nginx/registry.conf
upstream docker-registry {
  server registry:5000;
}

server {
  listen 443 ssl;
  server_name registry.myproject.com;

  ssl_certificate         /etc/nginx/conf.d/registry_ssl_keys/fullchain.pem;
  ssl_certificate_key     /etc/nginx/conf.d/registry_ssl_keys/privkey.pem;
  ssl_trusted_certificate /etc/nginx/conf.d/registry_ssl_keys/fullchain.pem;
  ssl_session_timeout 1d;
  ssl_session_cache shared:SSL:50m;

  # What Mozilla calls "Intermediate configuration"
  # Copied from https://mozilla.github.io/server-side-tls/ssl-config-generator/
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
  ssl_prefer_server_ciphers on;

  # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
  add_header Strict-Transport-Security max-age=15768000;

  # OCSP Stapling
  # fetch OCSP records from URL in ssl_certificate and cache them
  ssl_stapling on;
  ssl_stapling_verify on;

  # disable any limits to avoid HTTP 413 for large image uploads
  client_max_body_size 0;

  # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
  chunked_transfer_encoding on;

  location /v2/ {
    # Do not allow connections from docker 1.5 and earlier
    # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
    if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
      return 404;
    }

    # To add basic authentication to v2 use auth_basic setting plus add_header
    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/conf.d/registry.password;
    add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

    proxy_pass                          http://docker-registry;
    proxy_set_header  Host              $http_host;   # required for docker client's sake
    proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
    proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_read_timeout                  900;
  }
}

$ cat ~/myproject-services/data_nginx/jenkins.conf
upstream docker-jenkins {
  server jenkins:8080;
}

server {
  listen 443 ssl;
  server_name ci.myproject.com;

  ssl_certificate         /etc/nginx/conf.d/jenkins_ssl_keys/fullchain.pem;
  ssl_certificate_key     /etc/nginx/conf.d/jenkins_ssl_keys/privkey.pem;
  ssl_trusted_certificate /etc/nginx/conf.d/jenkins_ssl_keys/fullchain.pem;
  ssl_session_timeout 1d;
  ssl_session_cache shared:SSL:50m;

  # What Mozilla calls "Intermediate configuration"
  # Copied from https://mozilla.github.io/server-side-tls/ssl-config-generator/
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
  ssl_prefer_server_ciphers on;

  # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
  add_header Strict-Transport-Security max-age=15768000;

  # OCSP Stapling
  # fetch OCSP records from URL in ssl_certificate and cache them
  ssl_stapling on;
  ssl_stapling_verify on;

  # disable any limits to avoid HTTP 413 for large image uploads
  client_max_body_size 0;

  # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
  chunked_transfer_encoding on;

  location / {
    # Do not allow connections from docker 1.5 and earlier
    # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
    if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
      return 404;
    }

    # To add basic authentication use auth_basic setting plus add_header
    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/conf.d/jenkins.password;

    proxy_pass                          http://docker-jenkins;
    proxy_set_header  Host              $http_host;   # required for docker client's sake
    proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
    proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_read_timeout                  900;
  }
}

The config is mostly taken from letsencrypt.org nginx sample. In the upstream stanza we specify linked containers and ports on which these services are running.

Run services

By this time your ~/myproject-services/ directory should have the following structure:

Services structure

Finally we can start all services with:

$ cd ~/myproject-services/
$ docker-compose up -d
$ docker-compose ps
           Name                         Command               State                    Ports
--------------------------------------------------------------------------------------------------------------
yourprojectservices_jenkins_1    /usr/local/bin/jenkins.sh        Up      0.0.0.0:50000->50000/tcp, 8080/tcp
yourprojectservices_nginx_1      nginx -g daemon off;             Up      0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp
yourprojectservices_registry_1   /bin/registry /etc/docker/ ...   Up      5000/tcp
$ docker-compose logs

And of course both services should be live and available under

  • https://registry.myproject.com
  • https://ci.myproject.com

To stop them, type:

$ docker-compose down

Final notes

What you now probably want to do and what I left behind the scene:

  • add this project to git, ignoring artifacts(certs), but keeping empty folders(data_jenkins, data_registry)
  • add cron job for automated ssl certs renewal(letsencrypt generates certs valid for 3 months)
  • add monitoring tools
  • add backup system
  • add any kind of automation for the host initial setup(docker, docker-compose, cron, monitoring, etc) with chef, puppet or anything else

Dockerize Your Project, Part 2. Rails and PostgreSQL

References

Docker compose reference
How to set up private docker registry by Digital Ocean
Easy cert generation and renewal