Intro

  • Add user to docker group for root access on docker host: sudo usermod -aG docker USER_NAME

  • Install Docker Machine and Docker Compose separately on Linux.

  • docker container run --publish 8080:80 --detach --name webhost nginx:1.11 nginx -T

    • 8080 is the host listening port.
    • --detach runs the container in the backgroud.
    • 1.11 is the image version.
    • nginx -T is the CMD run on start.
    • Use -e to pass environment variables.
    • Use --rm to automatically remove the container after it exists.

What happens in “docker container run”

  1. Looks for that image locally in image cache, if it doesn’t find anything:
  2. Then looks in remote image repository (defaults to Docker Hub)
  3. Downloads the latest version of that image (or specific version if specified)
  4. Creates new container based on that image and prepares to start
  5. Gives it a virtual IP on a private network inside docker engine
  • docker container logs: show logs of a container.
  • docker container top: process list.
  • docker container inspect: config detail.
  • docker container stats: performance stats for all containers.

Run bash inside centos-7 image: docker container run --rm -it centos:7 bash.

  • docker container run -it: start new container interactively.
  • docker container run exec -it: run additional command in existing container.

Networks

  • docker container port CONTAINER_ID lists ports on each container.
  • Docker containers use a different IP address from host.
  • docker container inspect --format '{{ .NetworkSettings.IPAddress }}' CONTAINER_ID shows the IP address of specified container. (the --format option acts like a grep command, but more consistent for docker config inspection.)
  • Docker containers can talk to each other on the virtual network without explicitly opening up their ports to host.

Virtual Network

Docker comes with the “bridge” virtual network, and by default when a Docker container is created it is added to the “bridge” network. Containers on the same virtual network can communicate with each other without restriction, container names are also served as the DNS name.

Docker Network Structure

By default no port on the container is exposed to the host machine, to change that, use docker run -p [IP_EXPOSING_TO:]HOST_PORT:CONTAINER_PORT to expose port to host through the virtual network this container is on.

DNS

Instead of using the container name as DNS name, we can give our container a specific DNS name by using --network-alias DNS_NAME when starting a container. This way we can assign multiple containers to the same DNS name, and by default all network requests sent to the same DNS name would be routed to different containers using (random) Round Robin method.

# Create a network for elastic search containers to attach to.
docker network create searchers

# Create two elasticsearch 2 containers, and them to "searchers" network, and give them the same DNS name "search".
docker container run --net searchers --network-alias search -d --name els1 elasticsearch:2
docker container run --net searchers --network-alias search -d --name els2 elasticsearch:2

# Use alpine to lookup the DNS of "search".
docker container run --rm --net searchers alpine nslookup search 

# Run following command multiple times to recieve different results.
docker container run --rm --net searchers centos curl -s search:9200

Container Images

A version of image can have multiple tags, latest is a special tag. Since multiple tags can point to the same image (same content hash), running containers with different tags that points to the same version uses the same image instead of downloading it multiple times.

Image Layers

A Docker image is made up of layers of file system change history and metadata. We can use docker history IMAGE to inspect different layers of the image. Not all layer causes a file size change. Each layer has its own unique hash, so if two images uses a same layer, instead of downloading it again, Docker is able to reuse the same layer for both images.

docker history nginx:latest

# Output
# IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
# 5ad3bd0e67a9        10 days ago         /bin/sh -c #(nop)  CMD ["nginx" "-g" "daemon…   0B
# <missing>           10 days ago         /bin/sh -c #(nop)  STOPSIGNAL SIGTERM           0B
# <missing>           10 days ago         /bin/sh -c #(nop)  EXPOSE 80                    0B
# <missing>           10 days ago         /bin/sh -c ln -sf /dev/stdout /var/log/nginx…   22B
# <missing>           10 days ago         /bin/sh -c set -x     && addgroup --system -…   57.5MB
# <missing>           10 days ago         /bin/sh -c #(nop)  ENV PKG_RELEASE=1~buster     0B
# <missing>           10 days ago         /bin/sh -c #(nop)  ENV NJS_VERSION=0.3.8        0B
# <missing>           10 days ago         /bin/sh -c #(nop)  ENV NGINX_VERSION=1.17.8     0B
# <missing>           5 weeks ago         /bin/sh -c #(nop)  LABEL maintainer=NGINX Do…   0B
# <missing>           5 weeks ago         /bin/sh -c #(nop)  CMD ["bash"]                 0B
# <missing>           5 weeks ago         /bin/sh -c #(nop) ADD file:04caaf303199c81ff…   69.2MB

Relationship between Container and Image

The file system inside a container seems to be just a normal file system, but it is actually layers of file system changes. When we run multiple containers on top of the same image, each container only record the differences between the container itself and the base image (which is read-only). However, if there is a container that changes the file in the base image, a copy-on-write action is triggered and this container would have a copy of the base image file, along with the changes it made.

Image Tag

Images do not have a name, a specific image can be identified with repository, tag.

Use docker image tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG] to give an existing image a new tag, and use docker image push [OPTIONS] NAME[:TAG] to publish the image.

Dockerfile

Dockerfile is the recipe for images. Dockerfile is the default name for a Docker file, however, we can use -f option to specify any Docker file (e.g., docker build -f DOCKER_FILE_PATH -t NAME:TAG).

Commands

Each Dockerfile command is a layer of change, so the order does matter. Since each command is a layer of change, it’s best practice to keep things that change the least at top of the Docker file, so Docker can use the cached result and only rerun commands at later layers.

Command Description
FROM Base image layer.
ENV Set environment variables.
RUN Execute bash command inside the container itself.
WORKDIR Change working directory, preferred to using RUN cd /some/path.
COPY Copy file from local machine to current container.
EXPOSE Expose specified ports on Docker virtual network.
CMD Required command that is ran every time a container is launched.

When a Docker file inherit from another one, it also inherits the CMD command, which will be ran after all file changes, including the changes in the new image.

Logging

The best practice for logging inside Docker is not to log into log files, instead log everything to stdout and stderr.

# e.g., for the nginx container, log everything to stdout and stderr.
RUN ln -sf /dev/stdout /var/log/nginx/access.log && ln -sf /dev/stderr /var/log/nginx/error.log

Cleaning Up

Command Description
docker image prune [-a] Clean up “dangling” images, -a remove all not being used.
docker system prune Clean up everything.

Container Lifetime & Persistent Data

Containers are usually immutable and ephemeral. Container’s UFS file system does not go away if the container is stopped, it goes away when the container is removed. This problem is known as “persistent data”.

There are two ways to persistent data in Docker:

  • Data Volumes: make special location outside of container UFS, can be attached to any container later.
  • Bind Mounts: link container path to host path.

Volumes

Use VOLUME PATH-TO-VOLUME in Docker file to create volume inside of container. Volume has to be manually removed, it does not go away after the container is removed. Use docker container run -v VOLUME_NAME:VOLUME_PATH IMAGE_NAME to specify volume at time of starting container.

Bind Mount

Use docker container run -v LOCAL_PATH:BIND_MOUNT_PATH IMAGE_NAME to specify bind mount.

Docker Swarm

Swarm Mode

A single Swarm service can have multiple tasks, and each task will launch a container.

docker service in a Swarm replaces docker run.

  • docker service create alpine ping 8.8.8.8 creates an alpine service that pings Google’s DNS server.
  • docker service ls shows a list of services.
  • docker service ps SERVICE_NAME shows the containers running that service.
  • docker service update SERVICE_NAME --replicas NUM_REPLICAS scales the services up into multiple containers.
  • When a container goes down in a service, Swarm automatically launches a new one.

Creating Service on a Swarm

Create local Docker machines with docker-machine create NAME to create a machine, and then use docker-machine ssh NAME to connect to the machine.

Use docker swarm init --advertise-addr IP_ADDRESS to initialize the Swarm. (Running docker swarm init only will give an error with suggested IP addresses.)

After initializing the first Swarm node (as leader), use docker swarm join --token TOKEN IP_ADDRESS on other nodes to join the Swarm.

Workers in a Swarm do not have privilege to run Docker Swarm commands, use docker node update --role manager NODE_NAME in a leader node to promote other nodes to leaders.

Below is an example of creating a service on a Swarm.

# Creating service on node1
$ docker service create --replicas 3 alpine ping 8.8.8.8
# pfq0lfntgwtox2dv5hakjs4jn
# overall progress: 3 out of 3 tasks
# 1/3: running   [==================================================>]
# 2/3: running   [==================================================>]
# 3/3: running   [==================================================>]
# verify: Service converged

$ docker node ps node1
# ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
# hyhx9nvq9wt1        loving_yonath.1     alpine:latest       node1               Running             Running 13 seconds ago
$ docker node ps node2
# ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
# x8cooqtfe7fi        loving_yonath.2     alpine:latest       node2               Running             Running 23 seconds ago
$ docker node ps node3
# ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
# zz1o6v63vg20        loving_yonath.3     alpine:latest       node3               Running             Running 44 seconds ago

$ docker service ps loving_yonath
# ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
# hyhx9nvq9wt1        loving_yonath.1     alpine:latest       node1               Running             Running 59 seconds ago
# x8cooqtfe7fi        loving_yonath.2     alpine:latest       node2               Running             Running 58 seconds ago
# zz1o6v63vg20        loving_yonath.3     alpine:latest       node3               Running             Running 59 seconds ago

Overlay Network

Use docker network create --driver overlay NETWORK_NAME to create an overlay network. All services added to this network can talk to each other just like they are running on the same subnet, even if they are running on different nodes.

Routing Mesh

Accessing a service through the IP address of any node will result in the correct behavior, that’s due to routing mesh. Routing mesh routes ingress (incoming) packets for a Service to proper Task, it spans all nodes in Swarm and load balances Swarm Services across their Tasks. (It achieves this through IPVS from Linux Kernel.)

Secrets

A secret is:

  • Usernames and passwords
  • TLS certificates and keys
  • SSH keys
  • Any sensitive data.

Secrets are only stored on disk on Manager nodes, they are first stored in Swarm, then assigned to Services. Only containers in assigned Services can see them. They look like files in container but are actually in-memory fs.

Local docker-compose can use file-based secrets, but not secure.

Use docker secret create SECRET_NAME PATH_TO_SECRET_FILE or echo SECRET_CONTENT | docker secret create SECRETE_NAME to create a secret in Docker Swarm (but they are not safe methods for production!) After creating the Secret, use docker secret inspect SECRET_NAME to inspect the secret. Swarm stores all secrets in /run/secrets/ (in-memory) directory. When using a secret, -e ENV_NAME_FILE=/run/secrets/SECRET_NAME will actually store the content of the file into ENV_NAME environment variable.

To create and use a Secret in a service:

# Create Secrets
$ docker secret create psql_pass /path/to/psql/pass/file
$ echo "mypsql_user" | docker secret create psql_user

# Inspect Secrets
$ docker secret inspect psql_pass

# Use Secret in a Service
$ docker service create \
    --name psql \
    --secret psql_user \
    --secret psql_pass \
    -e POSTGRES_PASSWORD_FILE=/run/secrets/psql_pass \
    -e POSTGRES_USER_FILE=/run/secrets/psql_user \
    postgres

Secrets and Stack

version: 3.7

services:
  service_name:
    image: image_name
    secrets:
      - db_user_name
      - db_password
    environment:
      DB_USER_NAME_FILE: /run/secrets/db_user_name
      DB_PASSWORD_FILE: /run/secrets/db_password

secrets:
  db_user_name:
    file: ./db_user.txt
  db_password:
    file: ./db_pass.txt

$ docker stack deploy --compose-file docker-compose.yml

Swarm App Lifecycle

docker-compose automatically picks up docker-compose.yml and docker-compose.override.yml, use these two files for local development.

A good practice is to have docker-compose.test.yml file, and use docker-compose -f docker-compose.yml -f docker-compose.test.yml up -d

# Local development:
$ docker-compose up  # Look for docker-compose.yml and docker-compose.override.yml

# CI testing:
$ docker-compose -f docker-compose.yml -f docker-compose.test.yml up -d

# Generate Swarm production config file:
$ docker-compose  -f docker-compose.yml -f docker-compose.prod.yml config > prod-compose.yml

Service Updates

Swarm service updates provides rolling replacement of tasks/constainers in a service, it limits downtime.

# Update the image to use a new version
$ docker service update --image IMAGE_NAME SERVICE_NAME

# Adding an environment variable and remove a port
$ docker service update --env-add NODE_ENV=production --publish-rm 8080

# Change number of replicas of two services
$ docker service scale web=8 api=6

Docker Health Checks

Docker engine will exec's the command in the container (e.g. curl localhost), it expects exit 0 (OK) or exit 1 (Error).

$ docker run \
    --health-cmd="curl -f localhost:9200/_cluster/health || false" \
    --health-interval=5s \
    --health-retries=3 \
    --health-timeout=2s \
    --health-start-period=15s \
    elasticsearch:2