This the multi-page printable view of this section. Click here to print.
Docker
- 1: Docker for Beginners
- 1.1: Agenda
- 1.2: Pre-requisite
- 1.3: Getting Started with Docker Image
- 1.4: Accessing & Managing Docker Image
- 1.5: Getting Started with Dockerfile
- 1.6: Creating Private Docker Registry
- 1.7: Docker Volumes
- 1.8: Docker Networks
- 2: Docker for intermediate
- 2.1: Advanced Docker networking
- 2.2: Docker compose
- 2.3: First Look at Docker Application Packages?
- 2.4: Introduction to Docker Swarm
- 2.5: Multistage build
- 2.6: What happens when Containers are Launched?
- 3: Using Jenkins
- 4: Using Jenkins 2
- 5: DockerTools
1 - Docker for Beginners
1.1 - Agenda
| Description | Timing |
|---|---|
| Welcome | 8:45 AM to 9:00 AM |
| Creating a DockerHub Account | 9:00 AM to 9:15 AM |
| Getting Started with Docker Image | 9:15 AM to 10:15 AM |
| Accessing & Managing Docker Container | 10:15 AM to 11:15 AM |
| Coffee/Tea Break | 11:15 AM to 11:30 AM |
| Getting Started with Dockerfile - Part 1 | 11:30 AM to 1:00 PM |
| Lunch | 1:00 PM to 2:00 PM |
| Getting Started with Dockerfile - Part 2 | 2:00 PM to 3:30 PM |
| Creating Private Docker Registry | 3:30 PM to 4:00 PM |
| Docker Volumes | 4:00 PM to 4:30 PM |
| Coffee/Tea Break | 4:00 PM to 4:30 PM |
| Docker Networking | 4:45 PM to 5:45 PM |
| Quiz/Prize/Certificate Distribution | 5:45 PM to 6:00 PM |
1.2 - Pre-requisite
- Creating Your DockerHub Account - 15 min
Creating a DockerHub Account
Open https://hub.docker.com and click on “Sign Up” for DockerHub
Enter your username as DockerID and provide your email address( I would suggest you to provide your Gmail ID)
Example:
I have added ajeetraina as my userID as shown below. Please note that we will require this userID at the later point of time during the workshop. Hence, do keep it handy.
That’s it. Head over to your Email account to validate this account.
1.3 - Getting Started with Docker Image
- Running Hello World Example
- Working with Docker Image
- Saving Images and Containers as Tar Files for Sharing
- Building Your First Alpine Docker Image and Push it to DockerHub
- Test Your Knowledge
Getting Started with Docker Image
Demonstrating Hello World Example
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Running Hello World Example
$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
Digest: sha256:31b9c7d48790f0d8c50ab433d9c3b7e17666d6993084c002c2ff1ca09b96391d
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
Explanation
This image is a prime example of using the scratch image effectively. See hello.c in https://github.com/docker-library/hello-world for the source code of the hello binary included in this image.
So what’s happened here? We’ve called the docker run command, which is responsible for launching containers.
The argument hello-world is the name of the image someone created on dockerhub for us. It will first search for “hello-world” image locally and then search in Dockerhub.
Once the image has been downloaded, Docker turns the image into a running container and executes it.
Did you Know?
- The Hello World Docker Image is only 1.84 KB size.
[node1] (local) root@192.168.0.18 ~
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest 4ab4c602aa5e 6 weeks ago 1.84kB
- While running
docker pscommand, it doesn’t display any running container. Reason - It gets executed once and exit immediately.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS
PORTS NAMES
- You can use
docker inspect <imagename>command to inspect about this particular Docker Image.
$ docker inspect 4ab
[
{
"Id": "sha256:4ab4c602aa5eed5528a6620ff18a1dc4faef0e1ab3a5eddeddb410714478c67f",
"RepoTags": [
"hello-world:latest"
],
"RepoDigests": [
"hello-world@sha256:0add3ace90ecb4adbf7777e9aacf18357296e799f81cabc9fde470971e499788"
],
"Parent": "",
"Comment": "",
"Created": "2018-09-07T19:25:39.809797627Z",
"Container": "15c5544a385127276a51553acb81ed24a9429f9f61d6844db1fa34f46348e420",
"ContainerConfig": {
"Hostname": "15c5544a3851",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"/hello\"]"
],
"ArgsEscaped": true,
"Image": "sha256:9a5813f1116c2426ead0a44bbec252bfc5c3d445402cc1442ce9194fc1397027",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"DockerVersion": "17.06.2-ce",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/hello"
],
"ArgsEscaped": true,
"Image": "sha256:9a5813f1116c2426ead0a44bbec252bfc5c3d445402cc1442ce9194fc1397027",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"Architecture": "amd64",
"Os": "linux",
"Size": 1840,
"VirtualSize": 1840,
"GraphDriver": {
"Data": {
"MergedDir": "/var/lib/docker/overlay2/e494ae30abc49ad403ef5c2a32bcb894629ea4da6d4d226fbca70d27ed9a74d8/merged",
"UpperDir": "/var/lib/docker/overlay2/e494ae30abc49ad403ef5c2a32bcb894629ea4da6d4d226fbca70d27ed9a74d8/diff",
"WorkDir": "/var/lib/docker/overlay2/e494ae30abc49ad403ef5c2a32bcb894629ea4da6d4d226fbca70d27ed9a74d8/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:428c97da766c4c13b19088a471de6b622b038f3ae8efa10ec5a37d6d31a2df0b"
]
},
"Metadata": {
"LastTagTime": "0001-01-01T00:00:00Z"
}
}
]
Working with Docker Image
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Listing the Docker Images
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest 4ab4c602aa5e 6 weeks ago 1.84kB
Show all images (default hides intermediate images)
docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest 4ab4c602aa5e 6 weeks ago 1.84kB
List images by name and tag
The docker images command takes an optional [REPOSITORY[:TAG]] argument that restricts the list to images that match the argument. If you specify REPOSITORY but no TAG, the docker images command lists all images in the given repository.
To demo this, let us pull all various versions of alpine OS
docker pull alpine:3.6
docker pull alpine:3.7
docker pull alpine:3.8
docker pull alpine:3.9
[node4] (local) root@192.168.0.20 ~
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
alpine 3.6 43773d1dba76 7 days ago 4.03MB
alpine 3.7 6d1ef012b567 7 days ago 4.21MB
alpine 3.8 dac705114996 7 days ago 4.41MB
alpine 3.9 5cb3aa00f899 7 days ago 5.53MB
[node4] (local) root@192.168.0.20 ~
$ docker images alpine:3.7
REPOSITORY TAG IMAGE ID CREATED SIZE
alpine 3.7 6d1ef012b567 7 days ago 4.21MB
List the full length image IDs
$ docker images --no-trunc
REPOSITORY TAG IMAGE ID CREATED
SIZE
alpine 3.6 sha256:43773d1dba76c4d537b494a8454558a41729b92aa2ad0feb23521c3e58cd0440 7 days ago
4.03MB
alpine 3.7 sha256:6d1ef012b5674ad8a127ecfa9b5e6f5178d171b90ee462846974177fd9bdd39f 7 days ago
4.21MB
alpine 3.8 sha256:dac7051149965716b0acdcab16380b5f4ab6f2a1565c86ed5f651e954d1e615c 7 days ago
4.41MB
alpine 3.9 sha256:5cb3aa00f89934411ffba5c063a9bc98ace875d8f92e77d0029543d9f2ef4ad0 7 days ago
5.53MB
Listing out images with filter
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ubuntu latest 94e814e2efa8 3 days ago 88.9MB
alpine 3.6 43773d1dba76 7 days ago 4.03MB
alpine 3.7 6d1ef012b567 7 days ago 4.21MB
alpine 3.8 dac705114996 7 days ago 4.41MB
alpine 3.9 5cb3aa00f899 7 days ago 5.53MB
If you want to filter out just alpine
$ docker images --filter=reference='alpine'
REPOSITORY TAG IMAGE ID CREATED SIZE
alpine 3.6 43773d1dba76 7 days ago 4.03MB
alpine 3.7 6d1ef012b567 7 days ago 4.21MB
alpine 3.8 dac705114996 7 days ago 4.41MB
alpine 3.9 5cb3aa00f899 7 days ago 5.53MB
Saving Images and Containers as Tar Files for Sharing
Imagine a scenario where you have built Docker images and containers that you would be interested to keep and share it with your other collaborators or colleagues. The below methods shall help you achieve it.
Four basic Docker CLI comes into action:
- The
docker export- Export a container’s filesystem as a tar archive - The
docker import- Import the contents from a tarball to create a filesystem image - The
docker save- Save one or more images to a tar archive (streamed to STDOUT by default) - The
docker load- Load an image from a tar archive or STDIN
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Create Nginx Container
$ docker run -d -p 80:80 nginx
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
a5a6f2f73cd8: Pull complete
1ba02017c4b2: Pull complete
33b176c904de: Pull complete
Digest: sha256:5d32f60db294b5deb55d078cd4feb410ad88e6fe77500c87d3970eca97f54dba
Status: Downloaded newer image for nginx:latest
df2caf9283e84a15bb2321a17aabe84e3e0762ec82fc180e2a4c15fcf0f96588
[node1] (local) root@192.168.0.33 ~
Displaying Running Container
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
df2caf9283e8 nginx "nginx -g 'daemon of…" 35 seconds ago Up 34 seconds 0.0.0.0:80->80/tcp vigorous_jang
$ docker export df2 > nginx.tar
You could commit this container as a new image locally, but you could also use the Docker import command:
$ docker import - mynginx < nginx.tar
sha256:aaaed50d250a671042e8dc383c6e05012e245f5eaf555d10c40be63f6028ee7b
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
mynginx latest aaaed50d250a 25 seconds ago 107MB
nginx latest 568c4670fa80 2 weeks ago 109MB
If you wanted to share this image with one of your collaborators, you could upload the tar file on a web server and let your collaborator download it and use the import command on his Docker host.
If you would rather deal with images that you have already committed, you can use the load and save commands:
$ docker save -o mynginx1.tar nginx
$ ls -l
total 218756
-rw------- 1 root root 112844800 Dec 18 02:53 mynginx1.tar
-rw-r--r-- 1 root root 111158784 Dec 18 02:50 nginx.tar
$ docker rmi mynginx
Untagged: mynginx:latest
Deleted: sha256:aaaed50d250a671042e8dc383c6e05012e245f5eaf555d10c40be63f6028ee7b
Deleted: sha256:41135ad184eaac0f5c4f46e4768555738303d30ab161a7431d28a5ccf1778a0f
Now delete all images and containers running and try to run the below command to load Docker image into your system:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
$ docker load < mynginx1.tar
Loaded image: nginx:latest
[node1] (local) root@192.168.0.33 ~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 568c4670fa80 2 weeks ago 109MB
[node1] (local) root@192.168.0.33 ~
$
Building Your First Alpine Docker Image and Push it to DockerHub
How to build Your First Alpine Docker Image and Push it to DockerHub
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Under this tutorial we will see how to build our own first alpine based Docker Image.
$ docker run -dit alpine sh
620e1bcb5ab6e84b75a7a5c35790a77691112e59830ea1d5d85244bc108578c9
[node4] (local) root@192.168.0.20 ~
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
620e1bcb5ab6 alpine "sh" 3 seconds ago Up 2 seconds keen_alba
ttani
[node4] (local) root@192.168.0.20 ~
$ docker attach 62
/ #
/ #
/ # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.9.2
PRETTY_NAME="Alpine Linux v3.9"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
/ #
Updating APK Packages
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
v3.9.2-21-g3dda2a36ce [http://dl-cdn.alpinelinux.org/alpine/v3.9/main]
v3.9.2-19-gfdf726d41a [http://dl-cdn.alpinelinux.org/alpine/v3.9/community]
OK: 9756 distinct packages available
/ # ^
/ # apk add git
(1/7) Installing ca-certificates (20190108-r0)
(2/7) Installing nghttp2-libs (1.35.1-r0)
(3/7) Installing libssh2 (1.8.0-r4)
(4/7) Installing libcurl (7.64.0-r1)
(5/7) Installing expat (2.2.6-r0)
(6/7) Installing pcre2 (10.32-r1)
(7/7) Installing git (2.20.1-r0)
Executing busybox-1.29.3-r10.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 20 MiB in 21 packages
/ #
Now lets come out of it by Ctrl+P+Q and commit the changes
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
620e1bcb5ab6 alpine "sh" 4 minutes ago Up 4 minutes keen_alba
ttani
[node4] (local) root@192.168.0.20 ~
$ docker commit -m "Added GIT" 620 ajeetraina/alpine-git
sha256:9a8cd6c3bd8761013b2b932c58af2870f5637bfdf4227d7414073b0458ed0c54
[node4] (local) root@192.168.0.20 ~
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ajeetraina/alpine-git latest 9a8cd6c3bd87 11 seconds ago 31.2MB
ubuntu latest 94e814e2efa8 3 days ago 88.9MB
alpine 3.6 43773d1dba76 7 days ago 4.03MB
alpine 3.7 6d1ef012b567 7 days ago 4.21MB
alpine 3.8 dac705114996 7 days ago 4.41MB
alpine 3.9 5cb3aa00f899 7 days ago 5.53MB
alpine latest 5cb3aa00f899 7 days ago 5.53MB
There you see a new image just created.
Time to tag the image
$ docker tag --help
Usage: docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]
Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
[node4] (local) root@192.168.0.20 ~
$ docker tag ajeetraina/alpine-git:latest ajeetraina/alpine-git:1.0
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ajeetraina/alpine-git 1.0 9a8cd6c3bd87 2 minutes ago 31.2MB
ajeetraina/alpine-git latest 9a8cd6c3bd87 2 minutes ago 31.2MB
ubuntu latest 94e814e2efa8 3 days ago 88.9MB
alpine 3.6 43773d1dba76 7 days ago 4.03MB
alpine 3.7 6d1ef012b567 7 days ago 4.21MB
alpine 3.8 dac705114996 7 days ago 4.41MB
alpine 3.9 5cb3aa00f899 7 days ago 5.53MB
alpine latest 5cb3aa00f899 7 days ago 5.53MB
Pushing it to DockerHub
$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker
.com to create one.
Username: ajeetraina
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[node4] (local) root@192.168.0.20 ~
$ docker push ajeetraina/alpine-git:1.0
The push refers to repository [docker.io/ajeetraina/alpine-git]
3846235f8c17: Pushed
bcf2f368fe23: Mounted from library/alpine
1.0: digest: sha256:85d50f702e930db9e5b958387e667b7e26923f4de340534085cea184adb8411e size: 740
[node4] (local) root@192.168.0.20 ~
Test Your Knowledge
| S. No. | Question. | Response |
|---|---|---|
| 1 | What is difference between Docker Image and Docker Container? | |
| 2 | Where are all Docker images stored? | |
| 3 | Is DockerHub a public or private Docker registry? | |
| 4 | What is the main role of Docker Engine? | |
| 5 | Can you run Alpine container without even pulling it? | |
| 6 | What is the minimal size of Docker image you have built? | |
| 7 | I have mix of Ubuntu and CentOS-based Docker images. How shall I filter it out? |
1.4 - Accessing & Managing Docker Image
- Accessing the Container Shell
- Running a Command inside running Container
- Managing Docker Containers
- Test Your Knowledge
Accessing the Container Shell
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Create Ubuntu Container
docker run -dit ubuntu
Accessing the container shell
docker exec -t <container-id> bash
Accesssing the container shell
docker attach <container-id>
Running a command inside running Container
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Create Ubuntu Container
docker run -dit ubuntu
Opening up the bash shell
docker exec -t <container-id> bash
Managing Docker containers
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Preparations
- Clean your docker host using the commands (in bash):
$ docker rm -f $(docker ps -a -q)
Instructions
- Run the following containers from the dockerhub:
$ docker run -d -p 5000:5000 --name app1 selaworkshops/python-app:1.0
$ docker run -d -p 5001:5001 -e "port=5001" --name app2 selaworkshops/python-app:2.0
- Ensure the containers are running:
$ docker ps
- Stop the first container:
$ docker stop app1
- Kill the second container:
$ docker kill app2
- Display running containers:
$ docker ps
- Show all the containers (includind non running containers):
$ docker ps -a
- Let’s start both containers again:
$ docker start app1 app2
- Restart the second container:
$ docker restart app2
- Display the docker host information with:
$ docker info
- Show the running processes in the first container using:
$ docker top app1
- Retrieve the history of the second container:
$ docker history selaworkshops/python-app:2.0
- Inspect the second container image:
$ docker inspect selaworkshops/python-app:2.0
- Inspect the first container and look for the internal ip:
$ docker inspect app1
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "822cb66790c6358d9decab874916120f3bdeff7193a4375c94ca54d50832303d",
"EndpointID": "9aa96dc29be08eddc6d8f429ebecd2285c064fda288681a3611812413cbdfc1f",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:03",
"DriverOpts": null
}
}
- Show the logs of the second container using the flag –follow:
$ docker logs --follow app2
- Browse to the application and see the containers logs from the terminal:
http://localhost:5001
- Stop to tracking logs:
$ CTRL + C
Test Your Knowledge - Quiz2
| S. No. | Question. | Response |
|---|---|---|
| 1 | What are different ways to access container shell? | |
| 2 | How to run a command inside a Docker container | |
| 3 | Is it possible to stop overall Docker containers in a single shot? | |
| 4 | How do you remove all dangling images in Docker? | |
| 5 | How do you access services ports under Docker? |
1.5 - Getting Started with Dockerfile
- What is a Dockerfile?
- Understanding Layering Concept with Dockerfile
- Creating Docker Image
- ENTRYPOINT vs RUN
- Writing Dockerfile with Hello Python Script Added
- Test Your Knowledge
What is a Dockerfile?
- A Dockerfile is a text file which contains a series of commands or instructions.
- These instructions are executed in the order in which they are written.
- Execution of these instructions takes place on a base image.
- On building the Dockerfile, the successive actions form a new image from the base parent image.
Understanding Image Layering Concept with Dockerfile
Docker container is a runnable instance of an image, which is actually made by writing a readable/writable layer on top of some read-only layers.
The parent image used to create another image from a Dockerfile is read-only. When we execute instructions on this parent image, new layers keep adding up. These layers are created when we run docker build command.
The instructions RUN, COPY, ADD mostly contribute to the addition of layers in a Docker build.
Each layer is read-only except the last one - this is added to the image for generating a runnable container. This last layer is called “container layer”. All changes made to the container, like making new files, installing applications, etc. are done in this thin layer.
Let’s understand this layering using an example:
Consider the Dockerfile given below:
FROM ubuntu:latest
RUN mkdir -p /hello/hello
COPY hello.txt /hello/hello
RUN chmod 600 /hello/hello/hello.txt
Layer ID
Each instruction the Dockerfile generates a layer. Each of this layer has a randomly generated unique ID. This ID can be seen at the time of build. See the image below:
To view all these layers once an image is built from a Dockerfile, we can use docker history command.
To see more information about the Docker image and the layers use ‘docker inspect’ command as such:
# docker inspect testimage:latest
[
{
"Id": "sha256:c5701e02ed095ae7cabaef9fcef009d1f272206ff707deca13a680e024db7f02",
"RepoTags": [
"testimage:latest"
],
"RepoDigests": [],
"Parent": "sha256:694569c6db07ecef432cee1a9a4a6d45f2fd1f6be16814bf59e101bed966e612",
"Comment": "",
"Created": "2019-06-03T23:47:01.026463541Z",
"Container": "ac8873a003cb9ed972b4675f8d27181b99112e7530a5803ff89780e3ecc18b1c",
"ContainerConfig": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh",
"-c",
"chmod 600 /home/hello/hello.txt"
],
"ArgsEscaped": true,
"Image": "sha256:694569c6db07ecef432cee1a9a4a6d45f2fd1f6be16814bf59e101bed966e612",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"DockerVersion": "18.03.1-ce",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/bash"
],
"ArgsEscaped": true,
"Image": "sha256:694569c6db07ecef432cee1a9a4a6d45f2fd1f6be16814bf59e101bed966e612",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"Architecture": "amd64",
"Os": "linux",
"Size": 222876395,
"VirtualSize": 222876395,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/86a76eac21ae67f6d78e59076107a121e6dfb9cc922e68e1be975fc97e711eb1/diff:/var/lib/docker/overlay2/0604b502d31eff670769257ba3411fca09fbe2eab03343660ba557024915a1e6/diff:/var/lib/docker/overlay2/16af32e079fbc252ea5de044628285d5c3a34fc8441602a762729482666b2431/diff:/var/lib/docker/overlay2/732c4ab0164f92664ce831b4a830251132bf17cbcb7d093334a7a367b1a665e5/diff:/var/lib/docker/overlay2/c8a69709e5093c6eefa317f015cbf1422a446b2fe5d3f3d52a7e0d8af8dc6a28/diff:/var/lib/docker/overlay2/c93b36ec3a753592518727a2ea4547ab4e53d58489b9fae0838b2806e9c18346/diff:/var/lib/docker/overlay2/e67589599c2a5ed3bd74a269f3effaa52f94975fd811a866f1fe2bbcb2edabe4/diff",
"MergedDir": "/var/lib/docker/overlay2/31c68adcd824f155d23de4197b3d0b8776b079c307c1e4c0f2f8bbc73807adc0/merged",
"UpperDir": "/var/lib/docker/overlay2/31c68adcd824f155d23de4197b3d0b8776b079c307c1e4c0f2f8bbc73807adc0/diff",
"WorkDir": "/var/lib/docker/overlay2/31c68adcd824f155d23de4197b3d0b8776b079c307c1e4c0f2f8bbc73807adc0/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:05b0f7f2a81723fd647744a7340477ef9619f5ddeba3f2ca039dac3dd143cd59",
"sha256:0c3819952093832ffd8865bf72bc17f2f5475795cffe97e2b4c4ff36e638c244",
"sha256:14fa4a9494bf9e61f83a1bb96cd9e963ab0cbbdaf8ed91ff5eec5196c5bf7b12",
"sha256:b33859b66bfd3ad176ccf3be8dbd52d6b9823de8cc26688f2efeb76a0ea24a78",
"sha256:4622c8e1bdc0716e185fa3b338fa415dfdad3724336315de0bebd173a6ceaf05",
"sha256:6427efc3a0d7bae1fe315b844703580b2095073dcdf54a6ed9c7b1c0d982d9b0",
"sha256:59cd898074ac7765bacd76a11724b8d666ed8e9c14e7806dfb20a486102f6f1e",
"sha256:ad24f18512fddb8794612f7ec5955d06dcee93641d02932d809f0640263b8e79"
]
},
"Metadata": {
"LastTagTime": "2019-06-04T05:17:01.430558997+05:30"
}
}
]
Visualizing layers of Docker Image
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock wagoodman/dive testimage
[● Layers]───────────────────────────────────────────────────────────────────────────── [Current Layer Contents]───────────────────────────────────────────────────────────────
Cmp Size Command Permission UID:GID Size Filetree
63 MB FROM e388568efdf7281 drwxr-xr-x 0:0 4.8 MB ├── bin
988 kB [ -z "$(apt-get indextargets)" ] -rwxr-xr-x 0:0 1.1 MB │ ├── bash
745 B set -xe && echo '#!/bin/sh' > /usr/sbin/policy-rc.d && echo 'exit 101' > -rwxr-xr-x 0:0 35 kB │ ├── bunzip2
7 B mkdir -p /run/systemd && echo 'docker' > /run/systemd/container -rwxr-xr-x 0:0 0 B │ ├── bzcat → bin/bunzip2
0 B mkdir -p /hello/hello -rwxrwxrwx 0:0 0 B │ ├── bzcmp → bzdiff
37 B #(nop) COPY file:666735678ded52c6f9e0693ca27b4dc3d466e3d79c585a58c3b9a91357 -rwxr-xr-x 0:0 2.1 kB │ ├── bzdiff
37 B chmod 600 /hello/hello/hello.txt -rwxrwxrwx 0:0 0 B │ ├── bzegrep → bzgrep
-rwxr-xr-x 0:0 4.9 kB │ ├── bzexe
[Layer Details]──────────────────────────────────────────────────────────────────────── -rwxrwxrwx 0:0 0 B │ ├── bzfgrep → bzgrep
-rwxr-xr-x 0:0 3.6 kB │ ├── bzgrep
Tags: (unavailable) -rwxr-xr-x 0:0 0 B │ ├── bzip2 → bin/bunzip2
Id: e388568efdf72814bd6439a80d822ce06b631689a82292a2b96382d020d63a4c -rwxr-xr-x 0:0 14 kB │ ├── bzip2recover
Digest: sha256:43c67172d1d182ca5460fc962f8f053f33028e0a3a1d423e05d91b532429e73d -rwxrwxrwx 0:0 0 B │ ├── bzless → bzmore
Command: -rwxr-xr-x 0:0 1.3 kB │ ├── bzmore
#(nop) ADD file:08e718ed0796013f5957a1be7da3bef6225f3d82d8be0a86a7114e5caad50cbc in / -rwxr-xr-x 0:0 35 kB │ ├── cat
-rwxr-xr-x 0:0 64 kB │ ├── chgrp
[Image Details]──────────────────────────────────────────────────────────────────────── -rwxr-xr-x 0:0 60 kB │ ├── chmod
-rwxr-xr-x 0:0 68 kB │ ├── chown
Total Image size: 64 MB -rwxr-xr-x 0:0 142 kB │ ├── cp
Potential wasted space: 308 B -rwxr-xr-x 0:0 121 kB │ ├── dash
Image efficiency score: 99 % -rwxr-xr-x 0:0 101 kB │ ├── date
-rwxr-xr-x 0:0 76 kB │ ├── dd
Count Total Space Path -rwxr-xr-x 0:0 85 kB │ ├── df
2 234 B /var/lib/dpkg/diversions -rwxr-xr-x 0:0 134 kB │ ├── dir
2 74 B /hello/hello/hello.txt -rwxr-xr-x 0:0 72 kB │ ├── dmesg
-rwxrwxrwx 0:0 0 B │ ├── dnsdomainname → hostname
-rwxrwxrwx 0:0 0 B │ ├── domainname → hostname
-rwxr-xr-x 0:0 35 kB │ ├── echo
-rwxr-xr-x 0:0 28 B │ ├── egrep
-rwxr-xr-x 0:0 31 kB │ ├── false
-rwxr-xr-x 0:0 28 B │ ├── fgrep
-rwxr-xr-x 0:0 65 kB │ ├── findmnt
-rwxr-xr-x 0:0 220 kB │ ├── grep
-rwxr-xr-x 0:0 2.3 kB │ ├── gunzip
-rwxr-xr-x 0:0 5.9 kB │ ├── gzexe
-rwxr-xr-x 0:0 102 kB │ ├── gzip
Lab #1: Create an image with GIT installed
Pre-requisite:
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Assignment:
- Create an image with GIT installed
- Tag your image as labs-git:v1.0
- Create a container based on that image, and run git –version to check that it is installed correctly
Creating Dockerfile
FROM alpine:3.5
RUN apk update
RUN apk add git
Build Docker Image
docker build -t ajeetraina/alpine-git .
Tagging image as labs-git
docker tag ajeetraina/alpine-git ajeetraina/labs-git:v1.0
Verify the Images
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ajeetraina/alpine-git latest cb913e37a593 16 seconds ago 26.6MB
ajeetraina/labs-git v1.0 cb913e37a593 16 seconds ago 26.6MB
Create a container
docker run -itd ajeetraina/labs-git:v1.0 /bin/sh
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3e26a5268f55 ajeetraina/labs-git:v1.0 "/bin/sh" 4 seconds ago Up 2 seconds elated_neumann
Enter into Container Shell
docker attach 3e26
Please press “Enter” key twice so as to enter into container shell
Verify if GIT is installed
/ # git --version
git version 2.13.7
Lab #2: Create an image with ADD instruction
COPY and ADD are both Dockerfile instructions that serve similar purposes. They let you copy files from a specific location into a Docker image.
COPY takes in a src and destination. It only lets you copy in a local file or directory from your host (the machine building the Docker image) into the Docker image itself.
ADD lets you do that too, but it also supports 2 other sources. First, you can use a URL instead of a local file / directory. Secondly, you can extract a tar file from the source directly into the destination.
Pre-requisite:
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Assignment:
- Create an image with ADD instruction
- Tag your image as labs-add:v1.0
- Create a container based on that image, and see the extracted tar file.
Creating Dockerfile
FROM alpine:3.5
RUN apk update
ADD http://www.vlsitechnology.org/pharosc_8.4.tar.gz .
Build Docker Image
docker build -t saiyam911/alpine-add . -f <name of dockerfile>
Tagging image as labs-git
docker tag saiyam911/alpine-add saiyam911/labs-add:v1.0
Verify the Images
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
saiyam911/alpine-add latest cdf97cb49d48 38 minutes ago 300MB
saiyam911/labs-add v1.0 cdf97cb49d48 38 minutes ago 300MB
Create a container
docker run -itd saiyam911/labs-add:v1.0 /bin/sh
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f0940750f61a saiyam911/labs-add:v1.0 "/bin/sh" 20 seconds ago Up 18 seconds distracted_darwin
Enter into Container Shell
docker attach f094
Please press “Enter” key twice so as to enter into container shell
Verify if the link has been extracted onto the container
/ # ls -ltr
-rw------- 1 root root 295168000 Sep 19 2007 pharosc_8.4.tar.gz
ADD Command lets you to add a tar directly from a link and explode to the container.
Lab #3: Create an image with COPY instruction
The COPY instruction copies files or directories from source and adds them to the filesystem of the container at destinatio.
Two form of COPY instruction
COPY [--chown=<user>:<group>] <src>... <dest>
COPY [--chown=<user>:<group>] ["<src>",... "<dest>"] (this form is required for paths containing whitespace)
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Assignment:
- Create an image with COPY instruction
- COPY instruction in Multi-stage Builds
Create an image with COPY instruction
Dockerfile
FROM nginx:alpine
LABEL maintainer="Collabnix"
COPY index.html /usr/share/nginx/html/
ENTRYPOINT ["nginx", "-g", "daemon off;"]
Lets create the index.html file
$ echo "Welcome to Dockerlabs !" > index.html
Building Docker Image
$ docker image build -t cpy:v1 .
Staring the container
$ docker container run -d --rm --name myapp1 -p 80:80 cpy:v1
Checking index file
$ curl localhost
Welcome to Dockerlabs !
COPY instruction in Multi-stage Builds
Dockerfile
FROM alpine AS stage1
LABEL maintainer="Collabnix"
RUN echo "Welcome to Docker Labs!" > /opt/index.html
FROM nginx:alpine
LABEL maintainer="Collabnix"
COPY --from=stage1 /opt/index.html /usr/share/nginx/html/
ENTRYPOINT ["nginx", "-g", "daemon off;"]
Building Docker Image
$ docker image build -t cpy:v2 .
Staring the container
$ docker container run -d --rm --name myapp2 -p 8080:80 cpy:v2
Checking index file
$ curl localhost:8080
Welcome to Docker Labs !
NOTE: You can name your stages, by adding an AS
COPY --from=nginx:latest /etc/nginx/nginx.conf /nginx.conf
Lab #4: Create an image with CMD instruction
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Creating Dockerfile
FROM alpine:3.6
RUN apk update
CMD ["top"]
Building Docker Container
docker build -t ajeetraina/lab3_cmd . -f Dockerfile_cmd
Running the Docker container
docker run ajeetraina/lab3_cmd:latest
Lab #5: Create an image with ENTRYPOINT instruction
The ENTRYPOINT instruction make your container run as an executable.
ENTRYPOINT can be configured in two forms:
- Exec Form
ENTRYPOINT [“executable”, “param1”, “param2”] - Shell Form
ENTRYPOINT command param1 param2
If an image has an ENTRYPOINT if you pass an argument it, while running container it wont override the existing entrypoint, it will append what you passed with the entrypoint.To override the existing ENTRYPOINT you should user –entrypoint flag when running container.
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Assignment:
- Create an image with ENTRYPOINT instruction(Exec Form)
- ENTRYPOINT instruction in Shell Form
- Override the existing ENTRYPOINT
Create an image with ENTRYPOINT instruction(Exec Form)
Dockerfile
FROM alpine:3.5
LABEL maintainer="Collabnix"
ENTRYPOINT ["/bin/echo", "Hi, your ENTRYPOINT instruction in Exec Form !"]
Build Docker Image
$ docker build -t entrypoint:v1 .
Verify the Image
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
entrypoint v1 1d06f06c2062 2 minutes ago 4MB
alpine 3.5 f80194ae2e0c 7 months ago 4MB
Create a container
$ docker container run entrypoint:v1
Hi, your ENTRYPOINT instruction in Exec Form !
ENTRYPOINT instruction in Shell Form
Dockerfile
$ cat Dockerfile
FROM alpine:3.5
LABEL maintainer="Collabnix"
ENTRYPOINT echo "Hi, your ENTRYPOINT instruction in Shell Form !"
Build Docker Image
$ docker build -t entrypoint:v2 .
Verify the Image
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
entrypoint v2 cde521f13080 2 minutes ago 4MB
entrypoint v1 1d06f06c2062 5 minutes ago 4MB
alpine 3.5 f80194ae2e0c 7 months ago 4MB
Create a container
$ docker container run entrypoint:v2
Hi, your ENTRYPOINT instruction in Shell Form !
Override the existing ENTRYPOINT
$ docker container run --entrypoint "/bin/echo" entrypoint:v2 "Hello, Welocme to Docker Meetup! "
Hello, Welocme to Docker Meetup!
Lab #6: Create an image with WORKDIR instruction
The WORKDIR directive in Dockerfile defines the working directory for the rest of the instructions in the Dockerfile. The WORKDIR instruction wont create a new layer in the image but will add metadata to the image config. If the WORKDIR doesn’t exist, it will be created even if it’s not used in any subsequent Dockerfile instruction. you can have multiple WORKDIR in same Dockerfile. If a relative path is provided, it will be relative to the previous WORKDIR instruction.
WORKDIR /path/to/workdir
If no WORKDIR is specified in the Dockerfile then the default path is /. The WORKDIR instruction can resolve environment variables previously set in Dockerfile using ENV.
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Assignment
- Dockerfile with WORKDIR instruction
- WORKDIR with Relative path
- WORKDIR with Absolute path
- WORKDIR with environment variables as path
Dockerfile with WORKDIR instruction
Dockerfile
FROM alpine:3.9.3
LABEL maintainer="Collabnix"
WORKDIR /opt
Building Docker image
$ docker build -t workdir:v1 .
Testing current WORKDIR by running container
$ docker run -it workdir:v1 pwd
WORKDIR with relative path
Dockerfile
FROM alpine:3.9.3
LABEL maintainer="Collabnix"
WORKDIR /opt
RUN echo "Welcome to Docker Labs" > opt.txt
WORKDIR folder1
RUN echo "Welcome to Docker Labs" > folder1.txt
WORKDIR folder2
RUN echo "Welcome to Docker Labs" > folder2.txt
Building Docker image
$ docker build -t workdir:v2 .
Testing current WORKDIR by running container
$ docker run -it workdir:v2 pwd
WORKDIR with Absolute path
Dockerfile
FROM alpine:3.9.3
LABEL maintainer="Collabnix"
WORKDIR /opt/folder1
RUN echo "Welcome to Docker Labs" > opt.txt
WORKDIR /var/tmp/
Building Docker image
$ docker build -t workdir:v3 .
Testing current WORKDIR by running container
$ docker run -it workdir:v3 pwd
WORKDIR with environment variables as path
Dockerfile
FROM alpine:3.9.3
LABEL maintainer="Collabnix"
ENV DIRPATH /myfolder
WORKDIR $DIRPATH
Building Docker image
$ docker build -t workdir:v4 .
Testing current WORKDIR by running container
$ docker run -it workdir:v4 pwd
Lab #7: Create an image with RUN instruction
The RUN instruction execute command on top of the below layer and create a new layer.
RUN instruction can be wrote in two forms:
- RUN
(shell form) - RUN [“executable”, “param1”, “param2”] (exec form)
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Assignment:
- Create an image with RUN instruction
- Combining multiple RUN instruction to one
Create an image with RUN instruction
FROM alpine:3.9.3
LABEL maintainer="Collabnix"
RUN apk add --update
RUN apk add curl
RUN rm -rf /var/cache/apk/
Building Docker image
$ docker image build -t run:v1 .
Checking layer of the image
$ docker image history run:v1
IMAGE CREATED CREATED BY SIZE
NT
5b09d7ba1736 19 seconds ago /bin/sh -c rm -rf /var/cache/apk/ 0B
192115cc597a 21 seconds ago /bin/sh -c apk add curl 1.55MB
0518580850f1 43 seconds ago /bin/sh -c apk add --update 1.33MB
8590497d994e 45 seconds ago /bin/sh -c #(nop) LABEL maintainer=Collabnix 0B
cdf98d1859c1 4 months ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B
<missing> 4 months ago /bin/sh -c #(nop) ADD file:2e3a37883f56a4a27… 5.53MB
Number of layers 6
Checking image size
$ docker image ls run:v1
REPOSITORY TAG IMAGE ID CREATED SIZE
run v1 5b09d7ba1736 4 minutes ago 8.42MB
Its 8.42MB
Combining multiple RUN instruction to one
FROM alpine:3.9.3
LABEL maintainer="Collabnix"
RUN apk add --update && \
apk add curl && \
rm -rf /var/cache/apk/
Building Docker image
$ docker image build -t run:v2 .
Checking layer of the image
$ docker image history run:v2
IMAGE CREATED CREATED BY SIZE
NT
784298155541 50 seconds ago /bin/sh -c apk add --update && apk add curl… 1.55MB
8590497d994e 8 minutes ago /bin/sh -c #(nop) LABEL maintainer=Collabnix 0B
cdf98d1859c1 4 months ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B
<missing> 4 months ago /bin/sh -c #(nop) ADD file:2e3a37883f56a4a27… 5.53MB
Number of layers 4
Checking image size
$ docker image ls run:v2
REPOSITORY TAG IMAGE ID CREATED SIZE
run v2 784298155541 3 minutes ago 7.08MB
its now 7.08MB
Lab #8: Create an image with ARG instruction
The ARG directive in Dockerfile defines the parameter name and defines its default value. This default value can be overridden by the --build-arg <parameter name>=<value> in the build command docker build.
`ARG <parameter name>[=<default>]`
The build parameters have the same effect as ENV, which is to set the environment variables. The difference is that the environment variables of the build environment set by ARG will not exist in the future when the container is running. But don’t use ARG to save passwords and the like, because docker history can still see all the values.
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Assignment
- Writing a Dockerfile with ARG instruction
- Building Docker Image with default argument
- Running container argv:v1
- Passing the argument during image build time
- Running container argv:v2
Writing a Dockerfile with ARG instruction
We are writing a Dockerfile which echo “Welcome $WELCOME_USER, to Docker World!” where default argument value for WELCOME_USER as Collabnix.
FROM alpine:3.9.3
LABEL maintainer="Collabnix"
#Setting a default value to Argument WELCOME_USER
ARG WELCOME_USER=Collabnix
RUN echo "Welcome $WELCOME_USER, to Docker World!" > message.txt
CMD cat message.txt
Building Docker Image with default argument
$ docker image build -t arg:v1 .
Running container argv:v1
$ docker run arg:v1
Welcome Collabnix, to Docker World!
Passing the argument(WELCOME_USER) during image build time using –build-arg flag
$ docker image build -t arg:v2 --build-arg WELCOME_USER=Savio .
Running container argv:v2
$ docker run arg:v2
Welcome Savio, to Docker World!
NOTE: ARG is the only one instruction which can come before FROM instruction, but then arg value can be used only by FROM.
Lab #9: Create an image with ENV instruction
The ENV instruction in Dockerfile sets the environment variable for your container when you start. The default value can be overridden by passing --env <key>=<value> when you start the container.
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Assignment
- Writing a Dockerfile with ENV instruction
- Building Docker Image
- Running container env:v1
- Override existing env while running container
Writing a Dockerfile with ENV instruction
Dockerfile
FROM alpine:3.9.3
LABEL maintainer="Collabnix"
ENV WELCOME_MESSAGE="Welcome to Docker World"
CMD ["sh", "-c", "echo $WELCOME_MESSAGE"]
Building Docker Image
$ docker build -t env:v1 .
Running container env:v1
$ docker container run env:v1
Welcome to Docker World
Override existing env while running container
$ docker container run --env WELCOME_MESSAGE="Welcome to Docker Workshop" env:v1
Welcome to Docker Workshop
Lab #10: Create an image with VOLUME instruction
Pre-requisite:
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Assignment
- Create an image with VOLUME instruction
- Finding the volume created on the host
- Testing mount working as exepected
Create an image with VOLUME instruction
Dockerfile
FROM nginx:alpine
LABEL maintainer="Collabnix"
VOLUME /myvol
CMD [ "nginx","-g","daemon off;" ]
Building Docker image
$ docker build -t volume:v1 .
Create a container based on volume:v1 image
$ docker container run --rm -d --name volume-test volume:v1
Finding the volume created on the host
Checking the volume name of the container
$ docker container inspect -f '{{ (index .Mounts 0).Name }}' volume-test
ed09456a448934218f03acbdaa31f465ebbb92e0d45e8284527a2c538cc6b016
Listout Volume in the host
$ docker volume ls
DRIVER VOLUME NAME
local ed09456a448934218f03acbdaa31f465ebbb92e0d45e8284527a2c538cc6b016
You will see the volume has been created.
Volume mount path in host
$ docker container inspect -f '{{ (index .Mounts 0).Source }}' volume-test
/var/lib/docker/volumes/ed09456a448934218f03acbdaa31f465ebbb92e0d45e8284527a2c538cc6b016/_data
Testing mount working as exepected
Create a file in this folder
$ touch /var/lib/docker/volumes/ed09456a448934218f03acbdaa31f465ebbb92e0d45e8284527a2c538cc6b016/_data/mytestfile.txt
Checking file is there in run container
$ docker container exec -it volume-test ls myvol
Lab #11: Create an image with EXPOSE instruction
The EXPOSE instruction expose a port, the protocol can be UDP or TCP associated with the indicated port, default is TCP with no specification. The EXPOSE won’t be able to map the ports on the host machine. Regardless of the EXPOSE settings, EXPOSE port can be override using -p flag while starting the container.
Pre-requisite:
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Assignment
- Create an image with EXPOSE instruction
- Inspecting the EXPOSE port in the image
- Publish all exposed port
Create an image with VOLUME instruction
Dockerfile
FROM nginx:alpine
LABEL maintainer="Collabnix"
EXPOSE 80/tcp
EXPOSE 80/udp
CMD [ "nginx","-g","daemon off;" ]
Building Docker image
$ docker build -t expose:v1 .
Create a container based on expose:v1 image
$ docker container run --rm -d --name expose-inst expose:v1
Inspecting the EXPOSE port in the image
$ docker image inspect --format={{.ContainerConfig.ExposedPorts}} expose:v1
Publish all exposed ports
We can publish all EXPOSE port using -P flag.
$ docker container run --rm -P -d --name expose-inst-Publish expose:v1
Checking the publish port
$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
24983e09bd86 expose:v1 "nginx -g 'daemon of…" 46 seconds ago Up 45 seconds 0.0.0.0:32768->80/tcp, 0.0.0.0:32768->80/udp expose-inst-Publish
Lab #12: Create an image with LABEL Instruction
You can add labels to your image to help organize images by project, record licensing information, to aid in automation, or for other reasons. For each label, add a line beginning with LABEL and with one or more key-value pairs. The following examples show the different acceptable formats.
Docker offers support to add labels into images as a way to add custom metadata on them. The label syntax on your Dockerfile is as follows:
LABEL <key>=<value> <key>=<value> <key>=<value> ...
The LABEL instruction adds metadata to an image. A LABEL is a key-value pair. To include spaces within a LABEL value, use quotes and backslashes as you would in command-line parsing. A few usage examples:
LABEL "com.example.vendor"="ACME Incorporated"
LABEL com.example.label-with-value="foo"
LABEL version="1.0"
LABEL description="This text illustrates \
that label-values can span multiple lines."
An image can have more than one label. You can specify multiple labels on a single line. Prior to Docker 1.10, this decreased the size of the final image, but this is no longer the case. You may still choose to specify multiple labels in a single instruction, in one of the following two ways:
LABEL multi.label1="value1" multi.label2="value2" other="value3"
LABEL multi.label1="value1" \
multi.label2="value2" \
other="value3"
Labels included in base or parent images (images in the FROM line) are inherited by your image. If a label already exists but with a different value, the most-recently-applied value overrides any previously-set value.
To view an image’s labels, use the docker inspect command.
"Labels": {
"com.example.vendor": "ACME Incorporated"
"com.example.label-with-value": "foo",
"version": "1.0",
"description": "This text illustrates that label-values can span multiple lines.",
"multi.label1": "value1",
"multi.label2": "value2",
"other": "value3"
},
Lab #13: Create an image with ONBUILD instruction
Format: ONBUILD <other instructions>.
ONBUILD is a special instruction, followed by other instructions, such as RUN, COPY, etc., and these instructions will not be executed when the current image is built. Only when the current image is mirrored, the next level of mirroring will be executed.
The other instructions in Dockerfile are prepared to customize the current image. Only ONBUILD is prepared to help others customize themselves.
Suppose we want to make an image of the application written by Node.js. We all know that Node.js uses npm for package management, and all dependencies, configuration, startup information, etc. are placed in the package.json file. After getting the program code, you need to do npm install first to get all the required dependencies. Then you can start the app with npm start. Therefore, in general, Dockerfile will be written like this:
FROM node:slim
RUN mkdir /app
WORKDIR /app
COPY ./package.json /app
RUN [ "npm", "install" ]
COPY . /app/
CMD [ "npm", "start" ]
Put this Dockerfile in the root directory of the Node.js project, and after building the image, you can use it to start the container. But what if we have a second Node.js project? Ok, then copy this Dockerfile to the second project. If there is a third project? Copy it again? The more copies of a file, the more difficult it is to have version control, so let’s continue to look at the maintenance of such scenarios.
If the first Node.js project is in development, I find that there is a problem in this Dockerfile, such as typing a typo, or installing an extra package, then the developer fixes the Dockerfile, builds it again, and solves the problem. The first project is ok, but the second one? Although the original Dockerfile was copied and pasted from the first project, it will not fix their Dockerfile because the first project, and the Dockerfile of the second project will be automatically fixed.
So can we make a base image, and then use the base image for each project? In this way, the basic image is updated, and each project does not need to synchronize the changes of Dockerfile. After rebuilding, it inherits the update of the base image. Ok, yes, let’s see the result. Then the above Dockerfile will become:
FROM node:slim
RUN mkdir /app
WORKDIR /app
CMD [ "npm", "start" ]
Here we take out the project-related build instructions and put them in the subproject. Assuming that the name of the base image is my-node, the own Dockerfile in each project becomes:
FROM my-node
Yes, there is only one such line. When constructing a mirror with this one-line Dockerfile in each project directory, the three lines of the previous base image ONBUILD will start executing, successfully copy the current project code into the image, and execute for this project. npm install, generate an application image.
Lab
# Dockerfile
FROM busybox
ONBUILD RUN echo "You won't see me until later"
Docker build
docker build -t me/no_echo_here .
Uploading context 2.56 kB
Uploading context
Step 0 : FROM busybox
Pulling repository busybox
769b9341d937: Download complete
511136ea3c5a: Download complete
bf747efa0e2f: Download complete
48e5f45168b9: Download complete
---> 769b9341d937
Step 1 : ONBUILD RUN echo "You won't see me until later"
---> Running in 6bf1e8f65f00
---> f864c417cc99
Successfully built f864c417cc9
Here the ONBUILD instruction is read, not run, but stored for later use.
# Dockerfile
FROM me/no_echo_here
docker build -t me/echo_here . Uploading context 2.56 kB Uploading context Step 0 : FROM cpuguy83/no_echo_here
Executing 1 build triggers
Step onbuild-0 : RUN echo "You won't see me until later"
---> Running in ebfede7e39c8
You won't see me until later
---> ca6f025712d4
---> ca6f025712d4
Successfully built ca6f025712d4
Ubutu Rails
FROM ubuntu:12.04
RUN apt-get update -qq && apt-get install -y ca-certificates sudo curl git-core
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US.UTF-8
ENV LC_ALL en_US.UTF-8
RUN curl -L https://get.rvm.io | bash -s stable
ENV PATH /usr/local/rvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
RUN /bin/bash -l -c rvm requirements
RUN source /usr/local/rvm/scripts/rvm && rvm install ruby
RUN rvm all do gem install bundler
ONBUILD ADD . /opt/rails_demo
ONBUILD WORKDIR /opt/rails_demo
ONBUILD RUN rvm all do bundle install
ONBUILD CMD rvm all do bundle exec rails server
This Dockerfile is doing some initial setup of a base image. Installs Ruby and bundler. Pretty typical stuff. At the end are the ONBUILD instructions.
ONBUILD ADD . /opt/rails_demo Tells any child image to add everything in the current directory to /opt/railsdemo. Remember, this only gets run from a child image, that is when another image uses this one as a base (or FROM). And it just so happens if you look in the repo I have a skeleton rails app in railsdemo that has it’s own Dockerfile in it, we’ll take a look at this later.
ONBUILD WORKDIR /opt/rails_demo Tells any child image to set the working directory to /opt/rails_demo, which is where we told ADD to put any project files
ONBUILD RUN rvm all do bundle install Tells any child image to have bundler install all gem dependencies, because we are assuming a Rails app here.
ONBUILD CMD rvm all do bundle exec rails server Tells any child image to set the CMD to start the rails server
Ok, so let’s see this image build, go ahead and do this for yourself so you can see the output.
git clone git@github.com:sangam14/docker_onbuild.git
cd docker_onbuild
docker build -t sangam14/docker_onbuild .
Step 0 : FROM ubuntu:12.04
---> 9cd978db300e
Step 1 : RUN apt-get update -qq && apt-get install -y ca-certificates sudo curl git-core
---> Running in b32a089b7d2d
# output supressed
ldconfig deferred processing now taking place
---> d3fdefaed447
Step 2 : RUN rm /bin/sh && ln -s /bin/bash /bin/sh
---> Running in f218cafc54d7
---> 21a59f8613e1
Step 3 : RUN locale-gen en_US.UTF-8
---> Running in 0fcd7672ddd5
Generating locales...
done
Generation complete.
---> aa1074531047
Step 4 : ENV LANG en_US.UTF-8
---> Running in dcf936d57f38
---> b9326a787f78
Step 5 : ENV LANGUAGE en_US.UTF-8
---> Running in 2133c36335f5
---> 3382c53f7f40
Step 6 : ENV LC_ALL en_US.UTF-8
---> Running in 83f353aba4c8
---> f849fc6bd0cd
Step 7 : RUN curl -L https://get.rvm.io | bash -s stable
---> Running in b53cc257d59c
# output supressed
---> 482a9f7ac656
Step 8 : ENV PATH /usr/local/rvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
---> Running in c4666b639c70
---> b5d5c3e25730
Step 9 : RUN /bin/bash -l -c rvm requirements
---> Running in 91469dbc25a6
# output supressed
Step 10 : RUN source /usr/local/rvm/scripts/rvm && rvm install ruby
---> Running in cb4cdfcda68f
# output supressed
Step 11 : RUN rvm all do gem install bundler
---> Running in 9571104b3b65
Successfully installed bundler-1.5.3
Parsing documentation for bundler-1.5.3
Installing ri documentation for bundler-1.5.3
Done installing documentation for bundler after 3 seconds
1 gem installed
---> e2ea33486d62
Step 12 : ONBUILD ADD . /opt/rails_demo
---> Running in 5bef85f266a4
---> 4082e2a71c7e
Step 13 : ONBUILD WORKDIR /opt/rails_demo
---> Running in be1a06c7f9ab
---> 23bec71dce21
Step 14 : ONBUILD RUN rvm all do bundle install
---> Running in 991da8dc7f61
---> 1547bef18de8
Step 15 : ONBUILD CMD rvm all do bundle exec rails server
---> Running in c49139e13a0c
---> 23c388fb84c1
Successfully built 23c388fb84c1
Lab #14: Create an image with HEALTHCHECK instruction
The HEALTHCHECK directive tells Docker how to determine if the state of the container is normal. This was a new directive introduced during Docker 1.12. Before the HEALTHCHECK directive, the Docker engine can only determine if the container is in a state of abnormality by whether the main process in the container exits. In many cases, this is fine, but if the program enters a deadlock state, or an infinite loop state, the application process does not exit, but the container is no longer able to provide services. Prior to 1.12, Docker did not detect this state of the container and would not reschedule it, causing some containers to be unable to serve, but still accepting user requests.
The syntax look like:
HEALTHCHECK [options] CMD <command>:
The above syntax set the command to check the health of the container
How does it work?
When a HEALTHCHECK instruction is specified in an image, the container is started with it, the initial state will be starting, and will become healthy after the HEALTHCHECK instruction is checked successfully. If it fails for a certain number of times, it will become unhealthy.
What options does HEALTHCHECK support?
--interval=<interval>: interval between two health checks, the default is 30 seconds;
--timeout=<time length>: The health check command runs the timeout period. If this time is exceeded, the health check is regarded as a failure. The default is 30 seconds.
--retries=<number>: When the specified number of consecutive failures, the container status is treated as unhealthy, the default is 3 times.
Like CMD, ENTRYPOINT, HEALTHCHECK can only appear once. If more than one is written, only the last one will take effect.
Pre-requisite:
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Assignment:
- Writing a Dockerfile with HEALTHCHECK instruction
- Build a Docker Image
- Check that the nginx config file exists
- Check if nginx is healthy
- Make Docker container Unhealthy and check
- Create the nginx.conf file and Making the container go healthy
Writing a Dockerfile with HEALTHCHECK instruction
Suppose we have a simple Web service. We want to add a health check to determine if its Web service is working. We can use curl to help determine the HEALTHCHECK of its Dockerfile:
FROM nginx:1.13
HEALTHCHECK --interval=30s --timeout=3s \
CMD curl -f http://localhost/ || exit 1
EXPOSE 80
Here we set a check every 3 seconds (here the interval is very short for the test, it should be relatively long), if the health check command does not respond for more than 3 seconds, it is considered a failure, and use curl -fs http://localhost/ || exit 1 As a health check command.
Building Docker Image
docker image build -t nginx:1.13 .
Check that the nginx config file exists
docker run --name=nginx-proxy -d \
--health-cmd='stat /etc/nginx/nginx.conf || exit 1' \
nginx:1.13
Check if nginx is healthy
docker inspect --format='{{.State.Health.Status}}' nginx-proxy
Make Docker container Unhealthy and check
docker exec nginx-proxy rm /etc/nginx/nginx.conf
Check if nginx is healthy
sleep 5; docker inspect --format='{{.State.Health.Status}}' nginx-proxy
Creating the nginx.conf file and Making the container go healthy
docker exec nginx-proxy touch /etc/nginx/nginx.conf
sleep 5; docker inspect --format='{{.State.Health.Status}}' nginx-proxy
healthy
Lab #15: Create an image with SHELL instruction
Pre-requisite:
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
How does it work?
Format: SHELL ["executable", "parameters"]
- The SHELL instruction allows the default shell used for the shell form of commands to be overridden.
The default shell on Linux is
["/bin/sh", "-c"], and on Windows is["cmd", "/S", "/C"]. The SHELL instruction must be written in JSON form in a Dockerfile. - The SHELL instruction is particularly useful on Windows where there are two commonly used and quite different native shells:
cmdandpowershell, as well as alternate shells available includingsh. - The SHELL instruction can appear multiple times. Each SHELL instruction overrides all previous SHELL instructions, and affects all subsequent instructions.
Create a Dockerfile
FROM windowsservercore
# Executed as cmd /S /C echo default
RUN echo default
# Executed as cmd /S /C powershell -command Write-Host default
RUN powershell -command Write-Host default
# Executed as powershell -command Write-Host hello
SHELL ["powershell", "-command"]
RUN Write-Host hello
# Executed as cmd /S /C echo hello
SHELL ["cmd", "/S"", "/C"]
RUN echo hello
The following instructions can be affected by the SHELL instruction when the shell form of them is used in a Dockerfile: RUN, CMD and ENTRYPOINT.
The following example is a common pattern found on Windows which can be streamlined by using the SHELL instruction:
RUN powershell -command Execute-MyCmdlet -param1 "c:\foo.txt"
The command invoked by docker will be:
cmd /S /C powershell -command Execute-MyCmdlet -param1 "c:\foo.txt"
This is inefficient for two reasons.
First, there is an un-necessary cmd.exe command processor (aka shell) being invoked.
Second, each RUN instruction in the shell form requires an extra powershell -command prefixing the command.
To make it more efficient, one of two mechanisms can be employed. One is to use the JSON form of the RUN command such as:
RUN ["powershell", "-command", "Execute-MyCmdlet", "-param1 \"c:\\foo.txt\""]
While the JSON form is unambiguous and does not use the un-necessary cmd.exe, it does require more verbosity through double-quoting and escaping.
The alternate mechanism is to use the SHELL instruction and the shell form, making a more natural syntax for Windows users, especially when combined with the escape parser directive:
# escape=`
FROM windowsservercore
SHELL ["powershell","-command"]
RUN New-Item -ItemType Directory C:\Example
ADD Execute-MyCmdlet.ps1 c:\example\
RUN c:\example\Execute-MyCmdlet -sample 'hello world'
Build an image
PS E:\docker\build\shell> docker build -t shell .
Sending build context to Docker daemon 3.584 kB
Step 1 : FROM windowsservercore
---> 5bc36a335344
Step 2 : SHELL powershell -command
---> Running in 87d7a64c9751
---> 4327358436c1
Removing intermediate container 87d7a64c9751
Step 3 : RUN New-Item -ItemType Directory C:\Example
---> Running in 3e6ba16b8df9
Directory: C:\
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 6/2/2016 2:59 PM Example
---> 1f1dfdcec085
Removing intermediate container 3e6ba16b8df9
Step 4 : ADD Execute-MyCmdlet.ps1 c:\example\
---> 6770b4c17f29
Removing intermediate container b139e34291dc
Step 5 : RUN c:\example\Execute-MyCmdlet -sample 'hello world'
---> Running in abdcf50dfd1f
Hello from Execute-MyCmdlet.ps1 - passed hello world
---> ba0e25255fda
Removing intermediate container abdcf50dfd1f
Successfully built ba0e25255fda
PS E:\docker\build\shell>
- The SHELL instruction could also be used to modify the way in which a shell operates.
For example, using
SHELL cmd /S /C /V:ON|OFFon Windows, delayed environment variable expansion semantics could be modified. - The
SHELLinstruction can also be used on Linux should an alternate shell be required suchzsh,csh,tcshand others. - The
SHELLfeature was added in Docker 1.12.
Lab 16: Create an image with USER Instruction
The USER directive is similar to WORKDIR, which changes the state of the environment and affects future layers. WORKDIR is to change the working directory, and USER is the identity of the commands such as RUN, CMD and ENTRYPOINT.
Of course, like WORKDIR, USER just helps you switch to the specified user. This user must be pre-established, otherwise it cannot be switched.
Example:
RUN groupadd -r redis && useradd -r -g redis redis
USER redis
RUN [ "redis-server" ]
If the script executed with root wants to change the identity during execution, such as wanting to run a service process with an already established user, don’t use su or sudo, which requires a more cumbersome configuration. And often in the absence of TTY environment. It is recommended to use [gosu] (https://github.com/tianon/gosu).
# Create a redis user and use gosu to change another user to execute the command
RUN groupadd -r redis && useradd -r -g redis redis
# download gosu
RUN wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/1.7/gosu-amd64" \
&& chmod +x /usr/local/bin/gosu \
&& gosu nobody true
# Set CMD and execute it as another user
CMD [ "exec", "gosu", "redis", "redis-server" ]
Pre-requisite:
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
How is ENTRYPOINT different from the RUN instruction?
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
What is ENTRYPOINT meant for?
ENTRYPOINT is meant to provide the executable while CMD is to pass the default arguments to the executable. To understand it clearly, let us consider the below Dockerfile:
If you try building this Docker image using docker build command -
Let us run this image without any argument.
Let’s run it passing a command line argument
This clearly state that ENTRYPOINT is meant to provide the executable while CMD is to pass the default arguments to the executable.
Writing Dockerfile with Hello Python Script Added
Pre-requisite:
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
[node1] (local) root@192.168.0.38 ~
$ mkdir /test
[node1] (local) root@192.168.0.38 ~
$ cd /test
[node1] (local) root@192.168.0.38 /test
$ pwd
/test
Open a file named ‘Dockerfile’ with a text editor.
[node1] (local) root@192.168.0.38 /test
$ vi Dockerfile
Writing a Dockerfile
Setting a Base Image using FROM keyword
FROM ubuntu
Thus, our image would start building taking base as Ubuntu.
Defining the Author (Optional) using MAINTAINER keyword
MAINTAINER Prashansa Kulshrestha
Running a commands on the base image to form new layers using RUN keyword
RUN apt-get update
RUN apt-get install python
Since, the base image was Ubuntu, we can run Ubuntu commands here. These commands above install python over Ubuntu.
Adding a simple Hello World printing python file to the container’s file system using ADD command
ADD hello.py /home/hello.py
ADD a.py /home/a.py
We will place our hello.py and a.py files in the newly created directory itself (/test). ADD command would copy it from /test (current working directory) of host system to container’s filesystem at /home. The destination directories in the container would be create incase they don’t exist.
Code for hello.py:
print ("Hello World")
Code for a.py:
print ("Overriden Hello")
Specifying default execution environment for the container using CMD and ENTRYPOINT
These keywords let us define the default execution environment for a container when it just initiates from an image or just starts. If a command is specified with CMD keyword, it is the first command which a container executes as soon as it instantiates from an image. However, command and arguments provided with CMD can be overridden if user specifies his own commands while running the container using ‘docker run’ command.'
ENTRYPOINT helps to create a executable container and the commands and arguments provided with this keyword are not overridden.
We can also provide the default application environment using ENTRYPOINT and default arguments to be passed to it via CMD keyword. This can be done as follows:
CMD ["/home/hello.py"]
ENTRYPOINT ["python"]
So, default application mode of container would be python and if no other filename is provided as argument to it then it will execute hello.py placed in its /home directory.
Benefit of this is that user can choose some other file to run with the same application at runtime, that is, while launching the container.
So, our overall Dockerfile currently looks like this:
FROM ubuntu
MAINTAINER Prashansa Kulshrestha
RUN apt-get update
RUN apt-get install -y python
ADD hello.py /home/hello.py
ADD a.py /home/a.py
CMD ["/home/hello.py"]
ENTRYPOINT ["python"]
Building a Dockerfile
To create an image from the Dockerfile, we need to build it. This is done as follows:
[node1] (local) root@192.168.0.38 /test
$ docker build -t pythonimage .
The option -t lets us tag our image with a name we desire. So, here we have named our image as ‘pythonimage’. The ‘.’ in the end specifies current working directory i.e. /test. We initiated our build process from here. Docker would find the file named ‘Dockerfile’ in the current directory to process the build.
Running a container from the newly built image
[node1] (local) root@192.168.0.38 /test
$ docker run --name test1 pythonimage
Hello World
[node1] (local) root@192.168.0.38 /test
$
So, here /home/hello.py file placed in the container executed and displayed the output ‘Hello World’, since it was specified as default with CMD keyword.
[node1] (local) root@192.168.0.38 /test
$ docker run --name test2 pythonimage /home/a.py
Overriden Hello
[node1] (local) root@192.168.0.38 /test
$
Here, user specified another file to be run with python (default application for this container). So, the file specified with CMD got overridden and we obtained the output from /home/a.py.
1.6 - Creating Private Docker Registry
Building a Private Docker Registry
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Create a directory to permanently store images.
$ mkdir -p /registry/data
Authenticate with DockerHub
$docker login
Start the registry container.
$ docker run -d \
-p 5000:5000 \
--name registry \
-v /registry/data:/var/lib/registry \
--restart always \
registry:2
b1a641f8d710eee34405ad575050179f5a1262f1c845806cc3c2b435dea1648c
Display running containers.
$ docker ps
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
3a056bf96c6d registry:2 "/entrypoint.sh /etc…" About an hour ago Up About an hour 0.0.0
.0:5000->5000/tcp registry
Pull Alpine 3.6 image from official repository.
$ docker pull alpine:3.6
stretch: Pulling from library/alpine
723254a2c089: Pull complete
Digest: sha256:0a5fcee6f52d5170f557ee2447d7a10a5bdcf715dd7f0250be0b678c556a501b
Status: Downloaded newer image for alpine:3.6
Tag local alpine 3.6 image with an additional tag - local repository address.
$ docker tag alpine:3.6 localhost:5000/alpine:3.6
Push image to the local repository.
[node1] (local) root@192.168.0.23 ~
$ docker push localhost:5000/alpine:3.6
The push refers to repository [localhost:5000/alpine:3.6]
90d1009ce6fe: Pushed
stretch: digest: sha256:38236c068c393272ad02db100e09cac36a5465149e2924a035ee60d6c60c38fe size: 529
[node1] (local) root@192.168.0.23 ~
Remove local images.
[node1] (local) root@192.168.0.23 ~
$ docker image remove alpine:3.6
Untagged: alpine:3.6
Untagged: alpine@sha256:df6ebd5e9c87d0d7381360209f3a05c62981b5c2a3ec94228da4082ba07c4f05
[node1] (local) root@192.168.0.23 ~
$ docker image remove localhost:5000/alpine:3.6
Untagged: localhost:5000/alpine:3.6
Untagged: localhost:5000/debian@sha256:38236c068c393272ad02db100e09cac36a5465149e2924a035ee60d6c60c38fe
Deleted: sha256:4879790bd60d439cfe39c063660eef7af525d5f6f1cbb701a14c7cfc11cbfcf7
Pull Alpine 3.6 image from local repository.
[node1] (local) root@192.168.0.23 ~
$ docker pull localhost:5000/alpine:3.6
stretch: Pulling from alpine
54f7e8ac135a: Pull complete
Digest: sha256:38236c068c393272ad02db100e09cac36a5465149e2924a035ee60d6c60c38fe
Status: Downloaded newer image for localhost:5000/alpine:3.6
List stored images.
[node1] (local) root@192.168.0.23 ~
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost:5000/alpine 3.6 4879790bd60d 12 days ago 101MB
registry 2 2e2f252f3c88 2 months ago 33.3MB
Shared local registry
Create a directory to permanently store images.
$ mkdir -p /srv/registry/data
Create a directory to permanently store certificates and authentication data.
$ mkdir -p /srv/registry/security
Store domain and intermediate certificates using /srv/registry/security/registry.crt file, private key using /srv/registry/security/registry.key file. Use valid certificate and do not waste time with self-signed one. This step is required do use basic authentication.
Install apache2-utils to use htpasswd utility.
[node1] (local) root@192.168.0.23 ~
$ apk add apache2-utils
OK: 302 MiB in 110 packages
Create initial username and password. The only supported password format is bcrypt.
$ : | sudo tee /srv/registry/security/htpasswd
[node1] (local) root@192.168.0.23 ~
$ echo "password" | sudo htpasswd -iB /srv/registry/security/htpasswd username
Adding password for user username
Adding password for user username
$
[node1] (local) root@192.168.0.23 ~
$ cat /srv/registry/security/htpasswd
username:$2y$05$q9R5FSNYpAppB4Vw/AGWb.RqMCGE8DmZ4q5HZC/1wC87oTWyvB9vy
[node1] (local) root@192.168.0.23 ~
$
Stop and Remove all old containers
$ docker stop $(docker ps -a -q)
3a056bf96c6d
[node1] (local) root@192.168.0.23 ~
$ docker rm -f $(docker ps -a -q)
3a056bf96c6d
Start the registry container.
[node1] (local) root@192.168.0.23 ~
$ docker run -d -p 443:5000 --name registry -v /srv/registry/data:/var/lib/registry -v /srv/registry/security:/etc/security -e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/security/registry.crt -e REGISTRY_HTTP_TLS_KEY=/etc/security/registry.key -e REGISTRY_AUTH=htpasswd -e REGISTRY_AUTH_HTPASSWD_PATH=/etc/security/htpasswd -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" --restart always registry:2
e7755af8cbd70ea84ab77237a87cb97fd1abb18c7726fbc116c40f081d3b7098
[node1] (local) root@192.168.0.23 ~
Display running containers.
[node1] (local) root@192.168.0.23 ~
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e7755af8cbd7 registry:2 "/entrypoint.sh /etc…" About a minute ago Restarting (1) 22 seconds ago registry
[node1] (local) root@192.168.0.23 ~
Pull Alpine image from official repository.
$ docker pull alpine:3.6
stretch: Pulling from library/alpine
723254a2c089: Pull complete
Digest: sha256:0a5fcee6f52d5170f557ee2447d7a10a5bdcf715dd7f0250be0b678c556a501b
Status: Downloaded newer image for alpine:3.6
Tag local Alpine image with an additional tag - local repository address.
$ docker tag alpine:3.6 registry.collabnix.com/alpine:3.6
This time you need to provide login credentials to use local repository.
$ docker push registry.collabnix.com/alpine:3.6
e27a10675c56: Preparing
no basic auth credentials
$ docker pull registry.collabnix.com/alpine:3.6
Error response from daemon: Get https://registry.collabnix.com/v2/alpine/manifests/3.6: no basic auth credentials
Log in to the local registry.
$ docker login --username username registry.collabnix.com
Password: ********
Login Succeeded
Push image to the local repository.
$ docker push registry.collabnix.com/alpine:3.6
The push refers to repository [registry.collabnix.com/alpine]
e27a10675c56: Pushed
stretch: digest: sha256:02741df16aee1b81c4aaff4c48d75cc2c308bade918b22679df570c170feef7c size: 529
Remove local images.
$ docker image remove alpine:3.6
Untagged: alpine:3.6
Untagged: alpine@sha256:0a5fcee6f52d5170f557ee2447d7a10a5bdcf715dd7f0250be0b678c556a501b
$ docker image remove registry.collabnix.com/alpine:3.6
Untagged: registry.collabnix.com/alpine:3.6
Untagged: registry.sl.collabnix.com/alpine@sha256:02741df16aee1b81c4aaff4c48d75cc2c308bade918b22679df570c170feef7c
Deleted: sha256:da653cee0545dfbe3c1864ab3ce782805603356a9cc712acc7b3100d9932fa5e
Deleted: sha256:e27a10675c5656bafb7bfa9e4631e871499af0a5ddfda3cebc0ac401dfe19382
Pull Debian Stretch image from local repository.
$ docker pull registry.collabnix.com/alpine:3.6
stretch: Pulling from alpine
723254a2c089: Pull complete
Digest: sha256:02741df16aee1b81c4aaff4c48d75cc2c308bade918b22679df570c170feef7c
Status: Downloaded newer image for registry.collabnix.com/alpine:3.6
List stored images.
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
registry 2 d1fd7d86a825 4 weeks ago 33.3MB
registry.collabnix.com/alpine 3.6 da653cee0545 2 months ago 100MB
hello-world latest f2a91732366c 2 months ago
Test Your Knowledge
1.7 - Docker Volumes
Managing volumes through Docker CLI
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
docker volume commands
Creating Volume called demo
docker volume create demo
Listing Docker Volumes
docker volume ls
Inspecting “demo” Docker Volume
docker inspect demo
Removing the “demo” Docker Volume
docker volume rm demo
Creating Volume Mount from docker run command & sharing same Volume Mounts among multiple containers
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 1 | 5 min |
Pre-requisite
Volumes can be shared across containers
Preparations
- Clean your docker host using the commands :
$ docker rm -f $(docker ps -a -q)
$ docker volume rm $(docker volume ls -q)
Task
-
The Task for this lab is to create a volume, call it my_volume.
-
you should than run a simple an thin container and attach a volume to it. use the image selaworkshops/busybox:latest and use any name to the mounted volume directory (e.g : data)
-
change something in the volume folder , e.g : add a file with some content.
-
create a second volume mounted to the same volume , make sure the file you created in step 3 exists !
Instructions
- Display existing volumes:
$ docker volume ls
- Create a new volume:
$ docker volume create my-volume
- Inspect the new volume to find the mountpoint (volume location):
$ docker volume inspect my-volume
[
{
"CreatedAt": "2018-06-13T20:36:15Z",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/my-volume/_data",
"Name": "my-volume",
"Options": {},
"Scope": "local"
}
]
- Let’s run a container and mount the created volume to the root:
$ docker run -it -v my-volume:/data --name my-container selaworkshops/busybox:latest
- Create a new file under /data:
$ cd /data
$ echo "hello" > hello.txt
$ ls
- Open other terminal instance and run other container with the same volume:
$ docker run -it -v my-volume:/data --name my-container-2 selaworkshops/busybox:latest
- Inspect the /data folder (the created file will be there):
$ cd data
$ ls
- Exit from both containers and delete them:
$ exit
$ docker rm -f my-container my-container-2
- Ensure the containers were deleted
$ docker ps -a
- Run a new container attaching the created volume:
$ docker run -it -v my-volume:/data --name new-container selaworkshops/busybox:latest
- Inspect the /data folder (the created file will be there):
$ cd data
$ ls
- Exit from the container and delete it:
$ exit
$ docker rm -f new-container
1.8 - Docker Networks
- Step 1 - The
docker networkcommand - Step 2 - List networks
- Step 3 - Inspect a network
- Step 4 - List network driver plugins
- Test Your Knowledge
Prerequisites
You will need all of the following to complete this lab:
- A Linux-based Docker Host running Docker 1.12 or higher
Step 1: The docker network command
The docker network command is the main command for configuring and managing container networks.
Run a simple docker network command from any of your lab machines.
$ docker network
Usage: docker network COMMAND
Manage Docker networks
Options:
--help Print usage
Commands:
connect Connect a container to a network
create Create a network
disconnect Disconnect a container from a network
inspect Display detailed information on one or more networks
ls List networks
rm Remove one or more networks
Run 'docker network COMMAND --help' for more information on a command.
The command output shows how to use the command as well as all of the docker network sub-commands. As you can see from the output, the docker network command allows you to create new networks, list existing networks, inspect networks, and remove networks. It also allows you to connect and disconnect containers from networks.
Step 2: List networks
Run a docker network ls command to view existing container networks on the current Docker host.
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
1befe23acd58 bridge bridge local
726ead8f4e6b host host local
ef4896538cc7 none null local
The output above shows the container networks that are created as part of a standard installation of Docker.
New networks that you create will also show up in the output of the docker network ls command.
You can see that each network gets a unique ID and NAME. Each network is also associated with a single driver. Notice that the “bridge” network and the “host” network have the same name as their respective drivers.
Step 3: Inspect a network
The docker network inspect command is used to view network configuration details. These details include; name, ID, driver, IPAM driver, subnet info, connected containers, and more.
Use docker network inspect to view configuration details of the container networks on your Docker host. The command below shows the details of the network called bridge.
$ docker network inspect bridge
[
{
"Name": "bridge",
"Id": "1befe23acd58cbda7290c45f6d1f5c37a3b43de645d48de6c1ffebd985c8af4b",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16",
"Gateway": "172.17.0.1"
}
]
},
"Internal": false,
"Containers": {},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]
NOTE: The syntax of the
docker network inspectcommand isdocker network inspect <network>, where<network>can be either network name or network ID. In the example above we are showing the configuration details for the network called “bridge”. Do not confuse this with the “bridge” driver.
Step 4: List network driver plugins
The docker info command shows a lot of interesting information about a Docker installation.
Run a docker info command on any of your Docker hosts and locate the list of network plugins.
$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.3
Storage Driver: aufs
<Snip>
Plugins:
Volume: local
Network: bridge host null overlay <<<<<<<<
Swarm: inactive
Runtimes: runc
<Snip>
The output above shows the bridge, host, null, and overlay drivers.
Bridge networking
In this lab you’ll learn how to build, manage, and use bridge networks.
You will complete the following steps as part of this lab.
- Step 1 - The default bridge network
- Step 2 - Connect a container to the default bridge network
- Step 3 - Test the network connectivity
- Step 4 - Configure NAT for external access
Prerequisites
You will need all of the following to complete this lab:
- A Linux-based Docker host running Docker 1.12 or higher
- The lab was built and tested using Ubuntu 16.04
Step 1: The default bridge network
Every clean installation of Docker comes with a pre-built network called bridge. Verify this with the docker network ls command.
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
1befe23acd58 bridge bridge local
726ead8f4e6b host host local
ef4896538cc7 none null local
The output above shows that the bridge network is associated with the bridge driver. It’s important to note that the network and the driver are connected, but they are not the same. In this example the network and the driver have the same name - but they are not the same thing!
The output above also shows that the bridge network is scoped locally. This means that the network only exists on this Docker host. This is true of all networks using the bridge driver - the bridge driver provides single-host networking.
All networks created with the bridge driver are based on a Linux bridge (a.k.a. a virtual switch).
Install the brctl command and use it to list the Linux bridges on your Docker host.
# Install the brctl tools
$ apt-get install bridge-utils
<Snip>
# List the bridges on your Docker host
$ brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.0242f17f89a6 no
The output above shows a single Linux bridge called docker0. This is the bridge that was automatically created for the bridge network. You can see that it has no interfaces currently connected to it.
You can also use the ip command to view details of the docker0 bridge.
$ ip a
<Snip>
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:f1:7f:89:a6 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:f1ff:fe7f:89a6/64 scope link
valid_lft forever preferred_lft forever
Step 2: Connect a container
The bridge network is the default network for new containers. This means that unless you specify a different network, all new containers will be connected to the bridge network.
Create a new container.
$ docker run -dt ubuntu sleep infinity
6dd93d6cdc806df6c7812b6202f6096e43d9a013e56e5e638ee4bfb4ae8779ce
This command will create a new container based on the ubuntu:latest image and will run the sleep command to keep the container running in the background. As no network was specified on the docker run command, the container will be added to the bridge network.
Run the brctl show command again.
$ brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.0242f17f89a6 no veth3a080f
Notice how the docker0 bridge now has an interface connected. This interface connects the docker0 bridge to the new container just created.
Inspect the bridge network again to see the new container attached to it.
$ docker network inspect bridge
<Snip>
"Containers": {
"6dd93d6cdc806df6c7812b6202f6096e43d9a013e56e5e638ee4bfb4ae8779ce": {
"Name": "reverent_dubinsky",
"EndpointID": "dda76da5577960b30492fdf1526c7dd7924725e5d654bed57b44e1a6e85e956c",
"MacAddress": "02:42:ac:11:00:02",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""
}
},
<Snip>
Step 3: Test network connectivity
The output to the previous docker network inspect command shows the IP address of the new container. In the previous example it is “172.17.0.2” but yours might be different.
Ping the IP address of the container from the shell prompt of your Docker host. Remember to use the IP of the container in your environment.
$ ping 172.17.0.2
64 bytes from 172.17.0.2: icmp_seq=1 ttl=64 time=0.069 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=64 time=0.052 ms
64 bytes from 172.17.0.2: icmp_seq=3 ttl=64 time=0.050 ms
64 bytes from 172.17.0.2: icmp_seq=4 ttl=64 time=0.049 ms
64 bytes from 172.17.0.2: icmp_seq=5 ttl=64 time=0.049 ms
^C
--- 172.17.0.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.049/0.053/0.069/0.012 ms
Press Ctrl-C to stop the ping. The replies above show that the Docker host can ping the container over the bridge network.
Log in to the container, install the ping
program and ping www.dockercon.com.
# Get the ID of the container started in the previous step.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES
6dd93d6cdc80 ubuntu "sleep infinity" 5 mins Up reverent_dubinsky
# Exec into the container
$ docker exec -it 6dd93d6cdc80 /bin/bash
# Update APT package lists and install the iputils-ping package
root@6dd93d6cdc80:/# apt-get update
<Snip>
apt-get install iputils-ping
Reading package lists... Done
<Snip>
# Ping www.dockercon.com from within the container
root@6dd93d6cdc80:/# ping www.dockercon.com
PING www.dockercon.com (104.239.220.248) 56(84) bytes of data.
64 bytes from 104.239.220.248: icmp_seq=1 ttl=39 time=93.9 ms
64 bytes from 104.239.220.248: icmp_seq=2 ttl=39 time=93.8 ms
64 bytes from 104.239.220.248: icmp_seq=3 ttl=39 time=93.8 ms
^C
--- www.dockercon.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 93.878/93.895/93.928/0.251 ms
This shows that the new container can ping the internet and therefore has a valid and working network configuration.
Step 4: Configure NAT for external connectivity
In this step we’ll start a new NGINX container and map port 8080 on the Docker host to port 80 inside of the container. This means that traffic that hits the Docker host on port 8080 will be passed on to port 80 inside the container.
NOTE: If you start a new container from the official NGINX image without specifying a command to run, the container will run a basic web server on port 80.
Start a new container based off the official NGINX image.
$ docker run --name web1 -d -p 8080:80 nginx
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
386a066cd84a: Pull complete
7bdb4b002d7f: Pull complete
49b006ddea70: Pull complete
Digest: sha256:9038d5645fa5fcca445d12e1b8979c87f46ca42cfb17beb1e5e093785991a639
Status: Downloaded newer image for nginx:latest
b747d43fa277ec5da4e904b932db2a3fe4047991007c2d3649e3f0c615961038
Check that the container is running and view the port mapping.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b747d43fa277 nginx "nginx -g 'daemon off" 3 seconds ago Up 2 seconds 443/tcp, 0.0.0.0:8080->80/tcp web1
6dd93d6cdc80 ubuntu "sleep infinity" About an hour ago Up About an hour reverent_dubinsky
There are two containers listed in the output above. The top line shows the new web1 container running NGINX. Take note of the command the container is running as well as the port mapping - 0.0.0.0:8080->80/tcp maps port 8080 on all host interfaces to port 80 inside the web1 container. This port mapping is what effectively makes the containers web service accessible from external sources (via the Docker hosts IP address on port 8080).
Now that the container is running and mapped to a port on a host interface you can test connectivity to the NGINX web server.
To complete the following task you will need the IP address of your Docker host. This will need to be an IP address that you can reach (e.g. if your lab is in AWS this will need to be the instance’s Public IP).
Point your web browser to the IP and port 8080 of your Docker host. The following example shows a web browser pointed to 52.213.169.69:8080
If you try connecting to the same IP address on a different port number it will fail.
If for some reason you cannot open a session from a web broswer, you can connect from your Docker host using the curl command.
$ curl 127.0.0.1:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<Snip>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
If you try and curl the IP address on a different port number it will fail.
NOTE: The port mapping is actually port address translation (PAT).
Test Your Knowledge
2 - Docker for intermediate
2.1 - Advanced Docker networking
Advanced network configuration
introduces some of Docker’s advanced network configurations and options.
When Docker starts, it automatically creates a docker0 virtual bridge on the host. It is actually a bridge of Linux, which can be understood as a software switch. It will be forwarded between the network ports that are mounted to it.
At the same time, Docker randomly assigns an address in a private unoccupied private network segment (defined in RFC1918 ) to the docker0 interface. For example, the typical 172.17.42.1 mask of 255.255.0.0 . The network port in the container started after this will also automatically assign an address of the same network segment ( 172.17.0.0/16 ).
When creating a Docker container, a pair of veth pair interfaces are created (when the packet is sent to one interface, the other interface can also receive the same packet). The pair is terminated in the container, eth0 ; the other end is local and mounted to the docker0 bridge, with the name starting with veth (for example, vethAQI2QT ). In this way, the host can communicate with the container and the containers can communicate with each other. Docker created a virtual shared network between the host and all containers.
we will cover all of Docker’s network custom configurations in some scenarios. And adjust, complement, or even replace Docker’s default network configuration with Linux commands.
Configuring DNS
How to customize the host name and DNS of the configuration container? The secret is that Docker uses virtual files to mount three related configuration files for the container.
Use the mount command in the container to see the mount information:
$ mount /dev/disk/by-uuid/1fec...ebdf on /etc/hostname type ext4 ... /dev/disk/by-uuid/1fec...ebdf on /etc/hosts type ext4 ... tmpfs on /etc/resolv.conf type tmpfs ...
- This mechanism allows the DNS configuration of all Docker containers to be updated immediately after the host host’s DNS information is updated via the
/etc/resolv.conffile.
Configure DNS for all containers, or add the following to the /etc/docker/daemon.json file to set it up.
{ "dns" : [ "114.114.114.114" , "8.8.8.8" ] }
This way the container DNS is automatically configured to 114.114.114.114 and 8.8.8.8 each time it is started. Use the following command to prove that it has taken effect.
$ docker run -it --rm ubuntu:18.04 cat etc/resolv.conf nameserver 114.114.114.114 nameserver 8.8.8.8
If the user wants to manually specify the configuration of the container, you can add the following parameters when starting the container with the docker run command:
-h HOSTNAME or --hostname=HOSTNAME sets the hostname of the container, which will be written to /etc/hostname and/etc/hostscontainer. But it is not docker container ls , neither in the docker container ls nor in the other container’s /etc/hosts .
--dns=IP_ADDRESS Add the DNS server to the /etc/resolv.conf of the container and let the container use this server to resolve all hostnames that are not in /etc/hosts .
--dns-search=DOMAIN sets the search domain of the container. When the search domain is set to .example.com , when searching for a host named host, DNS not only searches for host but also searches for host.example.com .
Note: If the last two parameters are not specified when the container starts, Docker will default to configuring the container with /etc/resolv.conf on the host.
Disable networking for a container
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Mac OS | 1 | 5 min |
Pre-requisite
- A linux system (here we have used macbook)
- Docker installed
If we want to disable the networking stack on a container, we can use the “–network none” flag when starting the container. Within the container, only the loopback device is created.
Steps to implement this
- Create a container
$ docker run --rm -dit --network none --name no-net-alpine alpine:latest ash
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
4fe2ade4980c: Pull complete
Digest: sha256:621c2f39f8133acb8e64023a94dbdf0d5ca81896102b9e57c0dc184cadaf5528
Status: Downloaded newer image for alpine:latest
b961be5a20f2795125b85818ea2522ebb173beb36ec43fe10ed78cbd9a1a5d9e
- Check the container’s network stack, by executing “ip link show” networking commands within the container. Notice that no eth0 was created.
$ docker exec no-net-alpine ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1
link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN qlen 1
link/tunnel6 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
- You can check that no routing table available in this container
$ docker exec no-net-alpine ip route
$
- Stop the container
$ docker stop no-net-alpine
no-net-alpine
- Container will be automatically removed while stop, because it was created with the –rm flag.
$ docker container rm no-net-alpine
Error: No such container: no-net-alpine
Without “–network none” option
If we do not use “–network none” then we can see below differences.
$ docker run --rm -dit --name no-net-alpine alpine:latest ash
8a90992643c7b75e8d4daaf6d55fd9961c264f8f1af49d3e9ed420b657706ef9
$ docker exec no-net-alpine ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1
link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN qlen 1
link/tunnel6 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
You could see eth0 related info in networking command output.
Exposing a Container Port on the Host
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Mac OS | 1 | 5 min |
Pre-requisite
- A linux system (here we have used macbook)
- Docker installed
To expose the container port on the host we use -p or –publish option. [-p host_port:container_port ]
$ docker run -dit --name my-apache-app -p 8080:80 -v "$PWD":/usr/local/apache2/htdocs/ httpd:2.4
7e14fd11385969433f3273c3af0c74f9b0d0afd5f8aa7492b9705712df421f14
Once the port exposed to host , try to reach the port via explorer or with curl commands. You should get proper output from the container
$ curl localhost:8080
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<title>Index of /</title>
</head>
<body>
<h1>Index of /</h1>
<ul></ul>
</body></html>
Finding IP address of Container
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Mac OS | 1 | 5 min |
Pre-requisite
- A linux system (here we have used macbook)
- Docker installed
If you want to get the IP address of the container running on your system “docker inspect” with –format option will be helpful. Create a container and pass the container name or id to the “docker inspect” with –format or -f option.
$ docker run --rm -dit --name no-net-alpine alpine:latest ash
8a90992643c7b75e8d4daaf6d55fd9961c264f8f1af49d3e9ed420b657706ef9
$ docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' no-net-alpine
172.17.0.2
Verifying host-level settings that impact Docker networking
Docker relies on the host being capable of performing certain functions to make Docker networking work. Namely, your Linux host must be configured to allow IP forwarding. In addition, since the release of Docker 1.7, you may now choose to use hairpin Network Address Translation (NAT) rather than the default Docker user land proxy. In this recipe, we’ll review the requirement for the host to have IP forwarding enabled. We’ll also talk about NAT hairpin and discuss the host-level requirements for that option as well. In both cases, we’ll show Docker’s default behavior with regard to its settings as well as how you can alter them.
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Ubuntu 18.04 | 1 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side
Most Linux distributions default the IP forward value to disabled or 0. Fortunately for us, in a default configuration, Docker takes care of updating this setting to the correct value when the Docker service starts. For instance, let’s take a look at a freshly rebooted host that doesn’t have the Docker service enabled at boot time. If we check the value of the setting before starting Docker, we can see that it’s disabled. Starting the Docker engine automatically enables the setting for us:
user@docker1:~$ more /proc/sys/net/ipv4/ip_forward
0
user@docker1:~$ sudo systemctl start docker
user@docker1:~$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
Linking Containers in Docker
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Mac OS | 1 | 5 min |
Pre-requisite
- A linux system (here we have used macbook)
- Docker installed
In order to connect together multiple docker containers or services running inside docker container, ‘–link’ flag can be used in order to securely connect and provide a channel to transfer information from one container to another. A simple application of using a Wordpress container linked to MySQL container, can explain this well
- Pull the latest MySql container
$ docker pull mysql:latest
latest: Pulling from library/mysql
a5a6f2f73cd8: Already exists
936836019e67: Pull complete
283fa4c95fb4: Pull complete
1f212fb371f9: Pull complete
e2ae0d063e89: Pull complete
5ed0ae805b65: Pull complete
0283dc49ef4e: Pull complete
a7e1170b4fdb: Pull complete
88918a9e4742: Pull complete
241282fa67c2: Pull complete
b0fecf619210: Pull complete
bebf9f901dcc: Pull complete
Digest: sha256:b7f7479f0a2e7a3f4ce008329572f3497075dc000d8b89bac3134b0fb0288de8
Status: Downloaded newer image for mysql:latest
- Run MySql Container in detach mode (MySQL 8 changed the password authentication method. We’re looking for the mysql_native_password plugin, hence “–default-authentication-plugin=mysql_native_password” option require)
$ docker run --name mysql01 -e MYSQL_ROOT_PASSWORD=Password1234 -d mysql --default-authentication-plugin=mysql_native_password
fdabd410a66e4b65ec959677c932ccad79542ee9081d86ad2cbd0e2fe0265f1d
- Pull Wordpress docker container
$ docker pull wordpress:latest
latest: Pulling from library/wordpress
a5a6f2f73cd8: Already exists
633e0d1cd2a3: Pull complete
fcdfdf7118ba: Pull complete
4e7dc76b1769: Pull complete
c425447c8835: Pull complete
75780b7b9977: Pull complete
33ed51bc30e8: Pull complete
7c4215700bc4: Pull complete
d4f613c1e621: Pull complete
de5465a3fde0: Pull complete
6d373ffaf200: Pull complete
991bff14f001: Pull complete
d0a8c1ecf326: Pull complete
aa3627a535bb: Pull complete
a36be75bb622: Pull complete
98ebddb8e6ca: Pull complete
ed6e19b74de1: Pull complete
18b9cc4a2286: Pull complete
dfe625c958ac: Pull complete
Digest: sha256:f431a0681072aff336acf7a3736a85266fe7b46de116f29a2ea767ff55ad8f54
Status: Downloaded newer image for wordpress:latest
- Run the wordpress container linking it to MySQL Container (will run the database container with name “mysql-wordpress” and will set root password for MySQL container)
$ docker run --name wordpress01 --link mysql01 -p 8080:80 -e WORDPRESS_DB_HOST=mysql01:3306 -e WORDPRESS_DB_USER=root -e WORDPRESS_DB_PASSWORD=Password1234 -e WORDPRESS_DB_NAME=wordpress -e WORDPRESS_TABLE_PREFIX=wp_ -d wordpress
83b1f3215b01e7640246eb945977052bbf64f500a5b7fa18bd5a27841b01289b
- Check the status of the Containers
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
83b1f3215b01 wordpress "docker-entrypoint.s…" 33 seconds ago Up 32 seconds 0.0.0.0:8080->80/tcp wordpress01
fdabd410a66e mysql "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp, 33060/tcp mysql01
- As, we have linked both the container now wordpress container can be accessed from browser using the address http://localhost:8080 and setup of wordpress can be done easily.
Docker Quick Networking Guide
Below is a list of commands related to the Docker network.
Some of these command options are only configurable when the Docker service is started and will not take effect immediately.
-b BRIDGEor--bridge=BRIDGEspecifies the bridge to which the container is mounted--bip=CIDRcustom masker0 mask-H SOCKET...or--host=SOCKET...Channel for the Docker server to receive commands--icc=true|falseWhether communication between containers is supported--ip-forward=true|falsePlease see the communication between the containers below--iptables=true|falseyou allow Docker to add iptables rules?--mtu=BYTES MTUin the –mtu=BYTES container network
The following two command options can be specified either when starting the service or when starting the container. Specifying the Docker service when it is started will become the default value, and the default value of the setting can be overwritten when the docker run is executed later.
--dns=IP_ADDRESS...Use the specified DNS server--dns-search=DOMAIN...Specify the DNS search domain
Finally, these options are only used when the docker run executed because it is specific to the container’s properties.
-h HOSTNAMEor--hostname=HOSTNAMEconfiguration container hostname--link=CONTAINER_NAME:ALIASadds a connection to another container--net=bridge|none|container:NAME_or_ID|hostconfigures the bridge mode of the container-p SPEC or --publish=SPECmaps the container port to the host host-P or --publish-all=true|falsemaps all ports of the container to the host
Contributor - Sangam Biradar
2.2 - Docker compose
Docker compose
Docker compose is a tool built by docker to ease the task to creating and configring multiple containers in a development environment counter-part of docker-compose for prodcution environment is docker swarm. Docker compose takes as input a YAML configuration file and creates the resources (containers, networks, volumes etc.) by communicating with the docker daemon through docker api.
Compose file used in examples
version: '3'
services:
web:
build: .
image: web-client
depends_on:
- server
ports:
- "8080:8080"
server:
image: akshitgrover/helloworld
volumes:
- "/app" # Anonymous volume
- "data:/data" # Named volume
- "mydata:/data" # External volume
volumes:
data:
mydata:
external: true
Refer this for configuring your compose file.
CLI Cheatsheet
Build
Used to build services specified in docker-compose.yml file with build specification.
Refer this for more details.
Note: Images build will be tagged as {DIR}_{SERVICE} unless image name is specified in the service specification.
docker-compose build [OPTIONS] [SERVICE...]
OPTIONS:
--compress | Command line flag to compress the build context, Build context is nothing but a directory where docker-compose.yml file is located. As this directory can container a lot of files, sending build context to the container can take a lot of time thus compression is needed.
--force-rm | Remove any intermediate container while building.
--no-cache | Build images without using any cached layers from previoud builds.
--pull | Allways pull newer version of the base image.
-m, --memory | Set memory limit for the container used for building the image.
--parallel | Exploit go routines to parallely build images, As docker daemon is written in go.
--build-arg key=val | Pass any varaible to the dockerfile from the command line.
SERVICE:
If you want to build any particular services instead of every service specified in the compose file pass the name (same as in the compose file) as arguments to the command.
Example:
docker-compose build --compress # Will compress the build context of service web.
Bundle
Used to generate distributed application bundle (DAB) from the compose file.
Refer this for more details about DBA.
docker-compose bundle [OPTIONS]
OPTIONS:
--push-image | Push images to the register if any service has build specifcation.
-o, --output PATH | Output path for .dab file.
Config
Used to validate the compose file
NOTE: Run this command in direcotry where docker-compose.yml file is located.
docker-compose config
Up
Creates and starts the resources as per the specification the docker-compose.yml file.
docker-compose up [OPTIONS] [SERVICE...]
OPTIONS:
-d, --detach | Run containers in background.
--build | Always build images even if it exists.
--no-deps | Avoid creating any linked services.
--force-recreate | Force recreating containers even if specification is not changed.
--no-recreate | Do not recreate containers.
--no-build | Do not build any image even if it is missing.
--no-start | Just create the containers without starting them.
--scale SERVICE=NUM | Create multiple containers for a service.
-V, --renew-anon-volumes | Recreate anonymous volumes instead of getting data from previous ones.
Example:
docker-compose up -d # Will run service containers in background
docker-compose up web # Will start service web and server because of 'depends_on' field
docker-compose up server # will start server service only.
Down
Stop and clear any resources created while lifting docker-compose.
By default only containers and networks defined in the compose file are removed. Networks and Volumes with external = true and never removed.
docker-compose down [OPTIONS]
--rmi type | Remove images Type = all (Remove every image in the compose file), local (Remove images with no custom tag)
-v, --volumes | Remove named volumes except the external ones and also remove anonymous volumes
-t, --timeout TIMEOUT | Speficy shutdown time in seconds. (default = 10)
Example:
docker-compose down # Will delete all containers of both web and server and no volume will be removed
docker-compose down -v # Will also delete anonymous and data volumes.
Scale
Scale particular services
docker-compose scale [SERVICE=NUM...]
Example:
docker-compose scale server=3 web=2
Start
Start created containers.
docker-compose start [SERVICE...]
Example:
docker-compose start # Start containers for every service.
docker-compose start web # Start containers only for service web.
Stop
Stop running containers.
docker-compose stop [SERVICE...]
Example:
docker-compose stop # Stop containers for every service.
docker-compose stop web # Stop containers only for service web.
Docker compose with swarm secrets
Start securing your swarm services using the latest compose reference that allows to specify secrets in your application stack
Getting started
Make sure your daemon is in swarm mode or initialize it as follows:
docker swarm init --advertise-addr $(hostname -i)
Automatic provision
In this example we’ll let compose automatically create our secrets and provision them through compose with the defined secret file
Create a new secret and store it in a file:
echo "shh, this is a secret" > mysecret.txt
Use your secret file in your compose stack as follows:
echo 'version: '\'3.1\''
services:
test:
image: '\'alpine\''
command: '\'cat /run/secrets/my_secret \''
secrets:
- my_secret
secrets:
my_secret:
file: ./mysecret.txt
' > docker-compose.yml
Deploy your stack service:
docker stack deploy -c docker-compose.yml secret
Results in the below output:
Creating network secret_default
Creating secret secret_my_secret
Creating service secret_test
After your stack is deployed you can check your service output:
docker service logs -f secret_test
Results in the below output (below values after secret_test.1. may vary):
secret_test.1.lcygnppmzfdp@node1 | shhh, this is a secret
secret_test.1.mg1420w2i3x4@node1 | shhh, this is a secret
secret_test.1.8osraz8yxjrb@node1 | shhh, this is a secret
secret_test.1.byh5b9uik6db@node1 | shhh, this is a secret
. . .
Using existing secrets
Create a new secret using the docker CLI:
echo "some other secret" | docker secret create manual_secret -
Define your secret as external in your compose file so it’s not created while deploying the stack
echo 'version: '\'3.1\''
services:
test:
image: '\'alpine\''
command: '\'cat /run/secrets/manual_secret \''
secrets:
- manual_secret
secrets:
manual_secret:
external: true
' > external-compose.yml
Deploy your stack as you did in the automatic section:
docker stack deploy -c external-compose.yml external_secret
Validate your secret is there by checking the service logs
docker service logs -f external_secret_test
Contributors:
@marcosnils Akshit Grover
2.3 - First Look at Docker Application Packages?
A First Look at Docker Application Packages (“docker-app”)
Consider a scenario where you have separate development, test, and production environments for your Web application. Under development environment, your team might be spending time in building up Web application(say, WordPress), developing WP Plugins and templates, debugging the issue etc. When you are in development you’ll probably want to check your code changes in real-time. The usual way to do this is mounting a volume with your source code in the container that has the runtime of your application. But for production this works differently. Before you host your web application in production environment, you might want to turn-off the debug mode and host it under the right port so as to test your application usability and accessibility. In production you have a cluster with multiple nodes, and in most of the case volume is local to the node where your container (or service) is running, then you cannot mount the source code without complex stuff that involve code synchronization, signals, etc. In nutshell, this might require multiple Docker compose files for each environment and as your number of service applications increases, it becomes more cumbersome to manage those pieces of Compose files. Hence, we need a tool which can ease the way Compose files can be shareable across different environment seamlessly.
To solve this problem, Docker, Inc recently announced a new tool called “docker-app”(Application Packages) which makes “Compose files more reusable and shareable”. This tool not only makes Compose file shareable but provide us with simplified approach to share multi-service application (not just Docker Image) directly on Dockerhub.
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 5 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Spanner Sign on the left side of the screen to bring up template of 3 Managers & 2 Worker Nodes
Verifying 5 Node Swarm Mode Cluster
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
juld0kwbajyn11gx3bon9bsct * manager1 Ready Active Leader 18.03.1-ce
uu675q2209xotom4vys0el5jw manager2 Ready Active Reachable 18.03.1-ce
05jewa2brfkvgzklpvlze01rr manager3 Ready Active Reachable 18.03.1-ce
n3frm1rv4gn93his3511llm6r worker1 Ready Active 18.03.1-ce
50vsx5nvwx5rbkxob2ua1c6dr worker2 Ready Active 18.03.1-ce
Cloning the Repository
$ git clone https://github.com/ajeetraina/app
Cloning into 'app'...remote: Counting objects: 14147, done.
remote: Total 14147 (delta 0), reused 0 (delta 0), pack-reused 14147Receiving objects: 100% (14147/14147), 17.32 MiB | 18.43 MiB/s, done.
Resolving deltas: 100% (5152/5152), done.
Installing docker-app
wget https://github.com/docker/app/releases/download/v0.3.0/docker-app-linux.tar.gz
tar xf docker-app-linux.tar.gz
cp docker-app-linux /usr/local/bin/docker-app
OR
$ ./install.sh
Connecting to github.com (192.30.253.112:443)
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (52.216.227.152:443)
docker-app-linux.tar 100% |**************************************************************| 8780k 0:00:00 ETA
[manager1] (local) root@192.168.0.13 ~/app
$
Verify docker-app version
$ docker-app version
Version: v0.3.0
Git commit: fba6a09
Built: Fri Jun 29 13:09:30 2018
OS/Arch: linux/amd64
Experimental: off
Renderers: none
The docker-app tool comes with various options as shown below:
$ docker-app
Build and deploy Docker applications.
Usage:
docker-app [command]
Available Commands:
deploy Deploy or update an application
helm Generate a Helm chart
help Help about any command
init Start building a Docker application
inspect Shows metadata and settings for a given application
ls List applications.
merge Merge the application as a single file multi-document YAML
push Push the application to a registry
render Render the Compose file for the application
save Save the application as an image to the docker daemon(in preparation for push)
split Split a single-file application into multiple files
version Print version information
Flags:
--debug Enable debug mode
-h, --help help for docker-app
Use "docker-app [command] --help" for more information about a command.
[manager1] (local) root@192.168.0.48 ~/app
WordPress Application under dev & Prod environment
Under this demo, you will see that there is a folder called wordpress.dockerapp that contains three YAML documents:
- metadata
- the compose file
- settings for your application
You can create these files using the below command:
docker-app init --single-file wordpress
For more details, you can visit https://github.com/docker/app
Listing the Wordpress Application package related files/directories
$ ls
README.md install-wp with-secrets.yml
devel prod wordpress.dockerapp
Wordpress Application Package for Dev environment
$ docker-app render wordpress -f devel/dev-settings.yml
version: "3.6"
services:
mysql:
deploy:
mode: replicated
replicas: 1
endpoint_mode: dnsrr
environment:
MYSQL_DATABASE: wordpressdata
MYSQL_PASSWORD: wordpress
MYSQL_ROOT_PASSWORD: wordpress101
MYSQL_USER: wordpress
image: mysql:5.6
networks:
overlay: null
volumes:
- type: volume
source: db_data
target: /var/lib/mysql
wordpress:
depends_on:
- mysql
deploy:
mode: replicated
replicas: 1
endpoint_mode: vip
environment:
WORDPRESS_DB_HOST: mysql
WORDPRESS_DB_NAME: wordpressdata
WORDPRESS_DB_PASSWORD: wordpress
WORDPRESS_DB_USER: wordpress
WORDPRESS_DEBUG: "true"
image: wordpress
networks:
overlay: null
ports:
- mode: ingress
target: 80
published: 8082
protocol: tcp
networks:
overlay: {}
volumes:
db_data:
name: db_data
Wordpress Application Package for Prod
Under Prod environment, I have the following content under prod/prod-settings.yml as shown :
debug: false
wordpress:
port: 80
Post rendering, you should be able to see port:80 exposed as shown below in the snippet:
image: wordpress
networks:
overlay: null
ports:
- mode: ingress
target: 80
published: 80
protocol: tcp
networks:
overlay: {}
volumes:
db_data:
name: db_data
Inspect the WordPress App
$ docker-app inspect wordpress
wordpress 1.0.0
Maintained by: ajeetraina <ajeetraina@gmail.com>
Welcome to Collabnix
Setting Default
------- -------
debug true
mysql.database wordpressdata
mysql.image.version 5.6
mysql.rootpass wordpress101
mysql.scale.endpoint_mode dnsrr
mysql.scale.mode replicated
mysql.scale.replicas 1
mysql.user.name wordpress
mysql.user.password wordpress
volumes.db_data.name db_data
wordpress.port 8081
wordpress.scale.endpoint_mode vip
wordpress.scale.mode replicated
wordpress.scale.replicas 1
[manager1] (local) root@192.168.0.13 ~/app/examples/wordpress
$
Deploying the WordPress App
$ docker-app deploy wordpress
Creating network wordpress_overlay
Creating service wordpress_mysql
Creating service wordpress_wordpress
Switching to Dev Environ
$ docker-app deploy wordpress -f devel/dev-settings.yml
Switching to Prod Environ
$ docker-app deploy wordpress -f prod/prod-settings.yml
$ [manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$ docker-app deploy -f devel/dev-settings.yml
Updating service wordpress_wordpress (id: l95b4s6xi7q5mg7vj26lhzslb)
Updating service wordpress_mysql (id: lhr4h2uaer861zz1b04pst5sh)
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$ docker-app deploy -f prod/prod-settings.yml
Updating service wordpress_wordpress (id: l95b4s6xi7q5mg7vj26lhzslb)
Updating service wordpress_mysql (id: lhr4h2uaer861zz1b04pst5sh)
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$
Pushing Application Package to Dockerhub
$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to
https://hub.docker.com to create one.
Username: ajeetraina
Password:
Login Succeeded
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$
Saving this Application Package as DOcker Image
$ [manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$ docker-app save wordpress
Saved application as image: wordpress.dockerapp:1.0.0
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$
Listing out the images
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
wordpress.dockerapp 1.0.0 c1ec4d18c16c 47 seconds ago 1.62kB
mysql 5.6 97fdbdd65c6a 3 days ago 256MB
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$
Listing out the services
$ docker stack services wordpress
ID NAME MODE REPLICAS IMAGE PORTS
l95b4s6xi7q5 wordpress_wordpress replicated 1/1 wordpress:latest *:80->80
/tcp
lhr4h2uaer86 wordpress_mysql replicated 1/1 mysql:5.6
[manager1] (local) root@192.168.0.48 ~/docker101/play-with-docker/visualizer
Using docker-app ls command to list out the application packages
$ docker-app ls
REPOSITORY TAG IMAGE ID CREATED SIZE
wordpress.dockerapp 1.0.1 299fb78857cb About a minute ago 1.62kB
wordpress.dockerapp 1.0.0 c1ec4d18c16c 16 minutes ago 1.62kB
Pushing it to Dockerhub
$ docker-app push --namespace ajeetraina --tag 1.0.1
The push refers to repository [docker.io/ajeetraina/wordpress.dockerapp]
51cfe2cfc2a8: Pushed
1.0.1: digest: sha256:14145fc6e743f09f92177a372b4a4851796ab6b8dc8fe49a0882fc5b5c1be4f9 size: 524
Say, you built WordPress application package and pushed it to Dockerhub. Now one of your colleague pull it on his development system.
Pulling it from Dockerhub
$ docker pull ajeetraina/wordpress.dockerapp:1.0.1
1.0.1: Pulling from ajeetraina/wordpress.dockerapp
a59931d48895: Pull complete
Digest: sha256:14145fc6e743f09f92177a372b4a4851796ab6b8dc8fe49a0882fc5b5c1be4f9
Status: Downloaded newer image for ajeetraina/wordpress.dockerapp:1.0.1
[manager3] (local) root@192.168.0.24 ~/app
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ajeetraina/wordpress.dockerapp 1.0.1 299fb78857cb 8 minutes ago 1.62kB
[manager3] (local) root@192.168.0.24 ~/app
$
Deploying the Application in Easy Way
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ajeetraina/wordpress.dockerapp 1.0.1 299fb78857cb 9 minutes ago 1.62kB
[manager3] (local) root@192.168.0.24 ~/app
$ docker-app deploy ajeetraina/wordpress
Creating network wordpress_overlay
Creating service wordpress_mysql
Creating service wordpress_wordpress
[manager3] (local) root@192.168.0.24 ~/app
$
Using docker-app merge option
Docker Team has introduced docker-app merge option under the new 0.3.0 release.
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$ docker-app merge -o mywordpress
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$ ls
README.md install-wp prod wordpress.dockerapp
devel mywordpress with-secrets.yml
$ cat mywordpress
version: 1.0.1
name: wordpress
description: "Welcome to Collabnix"
maintainers:
- name: ajeetraina
email: ajeetraina@gmail.com
targets:
swarm: true
kubernetes: true
--
version: "3.6"
services:
mysql:
image: mysql:${mysql.image.version}
environment:
MYSQL_ROOT_PASSWORD: ${mysql.rootpass}
MYSQL_DATABASE: ${mysql.database}
MYSQL_USER: ${mysql.user.name}
MYSQL_PASSWORD: ${mysql.user.password}
volumes:
- source: db_data
target: /var/lib/mysql
type: volume
networks:
- overlay
deploy:
mode: ${mysql.scale.mode}
replicas: ${mysql.scale.replicas}
endpoint_mode: ${mysql.scale.endpoint_mode}
wordpress:
image: wordpress
environment:
WORDPRESS_DB_USER: ${mysql.user.name}
WORDPRESS_DB_PASSWORD: ${mysql.user.password}
WORDPRESS_DB_NAME: ${mysql.database}
WORDPRESS_DB_HOST: mysql
WORDPRESS_DEBUG: ${debug}
ports:
- "${wordpress.port}:80"
networks:
- overlay
deploy:
mode: ${wordpress.scale.mode}
replicas: ${wordpress.scale.replicas}
endpoint_mode: ${wordpress.scale.endpoint_mode}
depends_on:
- mysql
volumes:
db_data:
name: ${volumes.db_data.name}
networks:
overlay:
--
debug: true
mysql:
image:
version: 5.6
rootpass: wordpress101
database: wordpressdata
user:
name: wordpress
password: wordpress
scale:
endpoint_mode: dnsrr
mode: replicated
replicas: 1
wordpress:
scale:
mode: replicated
replicas: 1
endpoint_mode: vip
port: 8081
volumes:
db_data:
name: db_data
Contributor
2.4 - Introduction to Docker Swarm
What is Docker Swarm?
Docker Swarm is a container orchestration tool built and managed by Docker, Inc. It is the native clustering tool for Docker. Swarm uses the standard Docker API, i.e., containers can be launched using normal docker run commands and Swarm will take care of selecting an appropriate host to run the container on. This also means that other tools that use the Docker API—such as Compose and bespoke scripts—can use Swarm without any changes and take advantage of running on a cluster rather than a single host.
But why do we need Container orchestration System?
Imagine that you had to run hundreds of containers. You can easily see that if they are running in a distributed mode, there are multiple features that you will need from a management angle to make sure that the cluster is up and running, is healthy and more.
Some of these necessary features include:
● Health Checks on the Containers
● Launching a fixed set of Containers for a particular Docker image
● Scaling the number of Containers up and down depending on the load
● Performing rolling update of software across containers
● and more…
Docker Swarm has capabilities to help us implement all those great features - all through simple CLIs.
Does Docker Swarm require 3rd Party tool to be installed?
Docker Swarm Mode comes integrated with Docker Platform. Starting 1.12, Docker Swarm Mode is rightly integrated which means that you don’t need to install anything outside to run Docker Swarm. Just initialize it and you can get started.
Does Docker Swarm work with Docker Machine & Docker Compose?
Yes, it works very well with the Docker command line tools like docker and docker-machine, and provides the basic ability to deploy a Docker container to a collection of machines running the Docker Engine. Docker Swarm does differ in scope, however, from what we saw when reviewing Amazon ECS.
How does Swarm Cluster look like?
The basic architecture of Swarm is fairly straightforward: each host runs a Swarm agent and one host runs a Swarm manager (on small test clusters this host may also run an agent). The manager is responsible for the orchestration and scheduling of containers on the hosts. Swarm can be run in a high-availability mode where one of etcd, Consul or ZooKeeper is used to handle fail-over to a back-up manager. There are several different methods for how hosts are found and added to a cluster, which is known as discovery in Swarm. By default, token based discovery is used, where the addresses of hosts are kept in a list stored on the Docker Hub.
A swarm is a group of machines that are running Docker and joined into a cluster. After that has happened, we continue to run the Docker commands we’re used to, but now they are executed on a cluster by a swarm manager. The machines in a swarm can be physical or virtual. After joining a swarm, they are referred to as nodes.
Swarm managers are the only machines in a swarm that can execute your commands, or authorize other machines to join the swarm as workers. Workers are just there to provide capacity and do not have the authority to tell any other machine what it can and cannot do.
Up until now, you have been using Docker in a single-host mode on your local machine. But Docker also can be switched into swarm mode, and that’s what enables the use of swarms. Enabling swarm mode instantly makes the current machine a swarm manager. From then on, Docker runs the commands you execute on the swarm you’re managing, rather than just on the current machine.
Swarm managers can use several strategies to run containers, such as “emptiest node” – which fills the least utilized machines with containers. Or “global”, which ensures that each machine gets exactly one instance of the specified container.
A swarm is made up of multiple nodes, which can be either physical or virtual machines. The basic concept is simple enough: run docker swarm init to enable swarm mode and make our current machine a swarm manager, then run docker swarm join on other machines to have them join the swarm as workers.
Next » Difference between Docker Swarm, Docker Swarm Mode and Swarmkit
Getting Started with Docker Swarm
To get started with Docker Swarm, you can use “Play with Docker”, aka PWD. It’s free of cost and open for all. You get maximum of 5 instances of Linux system to play around with Docker.
-
Open Play with Docker labs on your browser
-
Click on Icon near to Instance to choose 3 Managers & 2 Worker Nodes
-
Wait for few seconds to bring up 5-Node Swarm Cluster
We recommend you start with one of our Beginners Guides, and then move to intermediate and expert level tutorials that cover most of the features of Docker. For a comprehensive approach to understanding Docker, I have categorized it as shown below:
A Bonus… Docker Swarm Visualizer
Swarm Visualizer is a fancy tool which visualized the Swarm Cluster setup. It displays containers running on each node in the form of visuals. If you are conducting Docker workshop, it’s a perfect way to show your audience how the containers are placed under each node. Go..try it out..
Clone the Repository
git clone https://github.com/dockersamples/docker-swarm-visualizer
cd docker-swarm-visualizer
docker-compose up -d
To run in a docker swarm:
$ docker service create \
--name=viz \
--publish=8080:8080/tcp \
--constraint=node.role==manager \
--mount=type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
dockersamples/visualizer
Proceed » What is Docker Swarm
Understanding Difference between Docker Swarm(Classic), Swarm Mode & SwarmKit
Docker Swarm is an older (2014) Docker native orchestration tool. It is standalone from the Docker engine and serves to connect Docker engines together to form a cluster. It's then possible to connect to the Swarm and run containers on the cluster. Swarm has a few features:
- Allows us to specify a discovery service
- Some control over where containers are placed (using filters / constraints / distribution strategies, etc…)
- Exposes the same API as the Docker engine itself, allowing 3rd-party tools to interact seamlessly
Swarmkit is a new (2016) tool developed by the Docker team which provides functionality for running a cluster and distributing tasks to the machines in the cluster. Here are the main features:
- Distributed: SwarmKit uses the Raft Consensus Algorithm in order to coordinate and does not rely on a single point of failure to perform decisions.
- Secure: Node communication and membership within a Swarm are secure out of the box. SwarmKit uses mutual TLS for node authentication, role authorization and transport encryption, automating both certificate issuance and rotation.
- Simple: SwarmKit is operationally simple and minimizes infrastructure dependencies. It does not need an external database to operate.
Docker Swarm Mode (Version 1.12 >) uses Swarmkit libraries & functionality in order to make container orchestration over multiple hosts (a cluster) very simple & secure to operate. There is a new set of features (the main one being docker swarm) which are now built into Docker itself to allow us to initiate a new Swarm and deploy tasks to that cluster.
Docker Swarm is not being deprecated, and is still a viable method for Docker multi-host orchestration, but Docker Swarm Mode (which uses the Swarmkit libraries under the hood) is the recommended way to begin a new Docker project where orchestration over multiple hosts is required.
One of the big features in Docker 1.12 release is Swarm mode. Docker had Swarm available for Container orchestration from 1.6 release. Docker released Swarmkit as an opensource project for orchestrating distributed systems few weeks before Docker 1.12(RC) release.
"Swarm" refers to traditional Swarm functionality, "Swarm Mode" refers to new Swarm mode added in 1.12, "Swarmkit" refers to the plumbing open source orchestration project.
Swarm, Swarm Mode and Swarmkit
Following table compares Swarm and Swarm Mode :
| Swarm | Swarm Mode |
|---|---|
| Separate from Docker Engine and can run as Container | Integrated inside Docker engine |
| Needs external KV store like Consul | No need of separate external KV store |
| Service model not available | Service model is available. This provides features like scaling, rolling update, service discovery, load balancing and routing mesh |
| Communication not secure | Both control and data plane is secure |
| Integrated with machine and compose | Not yet integrated with machine and compose as of release 1.12. Will be integrated in the upcoming releases |
Following table compares Swarmkit and Swarm Mode:
| Swarmkit | Swarm Mode |
|---|---|
| Plumbing opensource project | Swarmkit used within Swarm Mode and tightly integrated with Docker Engine |
| Swarmkit needs to built and run separately | Docker 1.12 comes integrated with Swarm Mode |
| No service discovery, load balancing and routing mesh | Service discovery, load balancing and routing mesh available |
| Use swarmctl CLI | Use regular Docker CLI |
Swarmkit has primitives to handle orchestration features like node management, discovery, security and scheduling.
How Docker Swarm Mode works?
Let us first understand what is Swarm Mode and what are its key concepts.
In 1.12, Docker introduced Swarm Mode. Swarm Mode enables the ability to deploy containers across multiple Docker hosts, using overlay networks for service discovery with a built-in load balancer for scaling the services.
Swarm Mode is managed as part of the Docker CLI, making it a seamless experience to the Docker ecosystem.
Key Concepts
Docker Swarm Mode introduces three new concepts which we’ll explore in this scenario.
Node:
A Node is an instance of the Docker Engine connected to the Swarm. Nodes are either managers or workers. Managers schedules which containers to run where. Workers execute the tasks. By default, Managers are also workers.
Services:
A service is a high-level concept relating to a collection of tasks to be executed by workers. An example of a service is an HTTP Server running as a Docker Container on three nodes.
Load Balancing:
Docker includes a load balancer to process requests across all containers in the service.
Step 1 - Initialise Swarm Mode
Turn single host Docker host into a Multi-host Docker Swarm Mode. Becomes Manager By default, Docker works as an isolated single-node. All containers are only deployed onto the engine. Swarm Mode turns it into a multi-host cluster-aware engine.
The first node to initialise the Swarm Mode becomes the manager. As new nodes join the cluster, they can adjust their roles between managers or workers. You should run 3-5 managers in a production environment to ensure high availability.
Create Swarm Mode Cluster
Swarm Mode is built into the Docker CLI. You can find an overview the possibility commands via docker swarm –help
The most important one is how to initialise Swarm Mode. Initialisation is done via init.
docker swarm init
After running the command, the Docker Engine knows how to work with a cluster and becomes the manager. The results of an initialisation is a token used to add additional nodes in a secure fashion. Keep this token safe and secure for future use when scaling your cluster.
In the next step, we will add more nodes and deploy containers across these hosts.
Step 2 - Join Cluster
With Swarm Mode enabled, it is possible to add additional nodes and issues commands across all of them. If nodes happen to disappear, for example, because of a crash, the containers which were running on those hosts will be automatically rescheduled onto other available nodes. The rescheduling ensures you do not lose capacity and provides high-availability.
On each additional node, you wish to add to the cluster, use the Docker CLI to join the existing group. Joining is done by pointing the other host to a current manager of the cluster. In this case, the first host.
Docker now uses an additional port, 2377, for managing the Swarm. The port should be blocked from public access and only accessed by trusted users and nodes. We recommend using VPNs or private networks to secure access.
Task
The first task is to obtain the token required to add a worker to the cluster. For demonstration purposes, we’ll ask the manager what the token is via swarm join-token. In production, this token should be stored securely and only accessible by trusted individuals.
token=$(docker -H 172.17.0.57:2345 swarm join-token -q worker) && echo $token
On the second host, join the cluster by requesting access via the manager. The token is provided as an additional parameter.
docker swarm join 172.17.0.57:2377 --token $token
By default, the manager will automatically accept new nodes being added to the cluster. You can view all nodes in the cluster using docker node ls
Lab01 - Create Overlay Network
Swarm Mode also introduces an improved networking model. In previous versions, Docker required the use of an external key-value store, such as Consul, to ensure consistency across the network. The need for consensus and KV has now been incorporated internally into Docker and no longer depends on external services.
The improved networking approach follows the same syntax as previously. The overlay network is used to enable containers on different hosts to communicate. Under the covers, this is a Virtual Extensible LAN (VXLAN), designed for large scale cloud based deployments.
Task
The following command will create a new overlay network called skynet. All containers registered to this network can communicate with each other, regardless of which node they are deployed onto.
docker network create -d overlay collabnet
Lab02 - Deploy Service
By default, Docker uses a spread replication model for deciding which containers should run on which hosts. The spread approach ensures that containers are deployed across the cluster evenly. This means that if one of the nodes is removed from the cluster, the instances would be already running on the other nodes. There workload on the removed node would be rescheduled across the remaining available nodes.
A new concept of Services is used to run containers across the cluster. This is a higher-level concept than containers. A service allows you to define how applications should be deployed at scale. By updating the service, Docker updates the container required in a managed way.
Task
In this case, we are deploying the Docker Image katacoda/docker-http-server. We are defining a friendly name of a service called http and that it should be attached to the newly created skynet network.
For ensuring replication and availability, we are running two instances, of replicas, of the container across our cluster.
Finally, we load balance these two containers together on port 80. Sending an HTTP request to any of the nodes in the cluster will process the request by one of the containers within the cluster. The node which accepted the request might not be the node where the container responds. Instead, Docker load-balances requests across all available containers.
docker service create --name http --network collabnet --replicas 2 -p 80:80 ajeetraina/hellowhale
You can view the services running on the cluster using the CLI command docker service ls
As containers are started you will see them using the ps command. You should see one instance of the container on each host.
List containers on the first host -
docker ps
List containers on the second host -
docker ps
If we issue an HTTP request to the public port, it will be processed by the two containers
curl your_machine_ip:80
Lab03 - Inspecting State
The Service concept allows you to inspect the health and state of your cluster and the running applications.
Task
You can view the list of all the tasks associated with a service across the cluster. In this case, each task is a container,
docker service ps http
You can view the details and configuration of a service via
docker service inspect --pretty http
On each node, you can ask what tasks it is currently running. Self refers to the manager node Leader:
docker node ps self
Using the ID of a node you can query individual hosts
docker node ps $(docker node ls -q | head -n1)
In the next step, we will scale the service to run more instances of the container.
Lab04 - Scale Service
A Service allows us to scale how many instances of a task is running across the cluster. As it understands how to launch containers and which containers are running, it can easily start, or remove, containers as required. At the moment the scaling is manual. However, the API could be hooked up to an external system such as a metrics dashboard.
Task
At present, we have two load-balanced containers running, which are processing our requests curl docker
The command below will scale our http service to be running across five containers.
docker service scale http=5
docker service scale http=5
http scaled to 5
overall progress: 5 out of 5 tasks
1/5: running [==================================================>]
2/5: running [==================================================>]
3/5: running [==================================================>]
4/5: running [==================================================>]
5/5: running [==================================================>]
verify: Waiting 4 seconds to verify that tasks are stable...
verify: Service converged
[manager1] (local) root@192.168.0.4 ~
$
[manager1] (local) root@192.168.0.4 ~
On each host, you will see additional nodes being started docker ps
The load balancer will automatically be updated. Requests will now be processed across the new containers. Try issuing more commands via
curl your_machine_ip:80
Try scaling the service down to see the result.
docker service scale http=2
Next » Lab05 - Deploy Applications Components as Docker Services
Lab #05 - Deploy the application components as Docker services
Our sleep application is becoming very popular on the internet (due to hitting Reddit and HN). People just love it. So, you are going to have to scale your application to meet peak demand. You will have to do this across multiple hosts for high availability too. We will use the concept of Services to scale our application easily and manage many containers as a single entity.
Services were a new concept in Docker 1.12. They work with swarms and are intended for long-running containers.
Let’s deploy sleep as a Service across our Docker Swarm.
$ docker service create --name sleep-app ubuntu sleep infinity
k70j90k9cp5n2bxsq72tjdmxs
overall progress: 1 out of 1 tasks
1/1: running
verify: Service converged
Verify that the service create has been received by the Swarm manager.
$ docker service ls
ID NAME MODE REPLICAS IMAGE
PORTS
k70j90k9cp5n sleep-app replicated 1/1 ubuntu:latest
The state of the service may change a couple times until it is running. The image is being downloaded from Docker Store to the other engines in the Swarm. Once the image is downloaded the container goes into a running state on one of the three nodes.
At this point it may not seem that we have done anything very differently than just running a docker run. We have again deployed a single container on a single host. The difference here is that the container has been scheduled on a swarm cluster.
Well done. You have deployed the sleep-app to your new Swarm using Docker services.
Scaling the Application
Demand is crazy! Everybody loves your sleep app! It’s time to scale out.
One of the great things about services is that you can scale them up and down to meet demand. In this step you’ll scale the service up and then back down.
You will perform the following procedure from node1.
Scale the number of containers in the sleep-app service to 7 with the docker service update –replicas 7 sleep-app command. Replicas is the term we use to describe identical containers providing the same service.
$ docker service update --replicas 7 sleep-app
sleep-app
overall progress: 7 out of 7 tasks
1/7: running
2/7: running
3/7: running
4/7: running
5/7: running
6/7: running
7/7: running
verify: Service converged
$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
k70j90k9cp5n sleep-app replicated 7/7 ubuntu:latest
The Swarm manager schedules so that there are 7 sleep-app containers in the cluster. These will be scheduled evenly across the Swarm members.
We are going to use the docker service ps sleep-app command. If you do this quick enough after using the –replicas option you can see the containers come up in real time.
$ docker service ps sleep-app
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
bv6ofc6x6moq sleep-app.1 ubuntu:latest manager1 Running Running 6 minutes ago
5gj1ql7sjt14 sleep-app.2 ubuntu:latest manager2 Running Running about a minute ago
p01z0tchepwa sleep-app.3 ubuntu:latest worker2 Running Running about a minute ago
x3kwnjcwxnb0 sleep-app.4 ubuntu:latest worker2 Running Running about a minute ago
c98vxyeefmru sleep-app.5 ubuntu:latest manager1 Running Running about a minute ago
kwmey288bkhp sleep-app.6 ubuntu:latest manager3 Running Running about a minute ago
vu78hp6bhauq sleep-app.7 ubuntu:latest worker1 Running Running about a minute ago
Notice that there are now 7 containers listed. It may take a few seconds for the new containers in the service to all show as RUNNING. The NODE column tells us on which node a container is running.
Scale the service back down to just four containers with the docker service update –replicas 4 sleep-app command.
$ docker service update --replicas 4 sleep-app
sleep-app
overall progress: 4 out of 4 tasks
1/4: running
2/4: running
3/4: running
4/4: running
verify: Service converged
[manager1] (local) root@192.168.0.9 ~/dockerlabs/intermediate/swarm
$ docker service ps sleep-app
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
bv6ofc6x6moq sleep-app.1 ubuntu:latest manager1 Running Running 7 minutes ago
5gj1ql7sjt14 sleep-app.2 ubuntu:latest manager2 Running Running 2 minutes ago
p01z0tchepwa sleep-app.3 ubuntu:latest worker2 Running Running 2 minutes ago
kwmey288bkhp sleep-app.6 ubuntu:latest manager3 Running Running 2 minutes ago
You have successfully scaled a swarm service up and down.
Next » Lab06 - Drain a Node and Reschedule the Containers
Lab06 - Drain a node and reschedule the containers
Your sleep-app has been doing amazing after hitting Reddit and HN. It’s now number 1 on the App Store! You have scaled up during the holidays and down during the slow season. Now you are doing maintenance on one of your servers so you will need to gracefully take a server out of the swarm without interrupting service to your customers.
Take a look at the status of your nodes again by running docker node ls on node1.
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
swfk8vsyfe4z2zbtianz5gh2p * manager1 Ready Active Leader 18.09.3
sgyr3vxu1n99vyce9al67alwt manager2 Ready Active Reachable 18.09.3
ud3ghz1zlrmn3fbv9j930ldja manager3 Ready Active Reachable 18.09.3
v57fk367d1lw4e1ufis3jwa2h worker1 Ready Active 18.09.3
uinkvr56fq7zb711ycbifhf4f worker2 Ready Active 18.09.3
You will be taking worker2 out of service for maintenance.
Let’s see the containers that you have running on worker2.
We are going to take the ID for worker2 and run docker node update –availability drain worker2. We are using the worker2 host ID as input into our drain command. Replace yournodeid with the id of worker2.
$ docker node update --availability drain worker2
worker2
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
swfk8vsyfe4z2zbtianz5gh2p * manager1 Ready Active Leader 18.09.3
sgyr3vxu1n99vyce9al67alwt manager2 Ready Active Reachable 18.09.3
ud3ghz1zlrmn3fbv9j930ldja manager3 Ready Active Reachable 18.09.3
v57fk367d1lw4e1ufis3jwa2h worker1 Ready Active 18.09.3
uinkvr56fq7zb711ycbifhf4f worker2 Ready Drain
Node worker2 is now in the Drain state.
Switch back to node2 and see what is running there by running docker ps.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
worker2 does not have any containers running on it.
Lastly, check the service again on node1 to make sure that the container were rescheduled. You should see all four containers running on the remaining two nodes.
$ docker service ps sleep-app
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
bv6ofc6x6moq sleep-app.1 ubuntu:latest manager1 Running Running 18 minutes ago
5gj1ql7sjt14 sleep-app.2 ubuntu:latest manager2 Running Running 12 minutes ago
5aqy7jv9ojmn sleep-app.3 ubuntu:latest worker1 Running Running 3 minutes ago
p01z0tchepwa \_ sleep-app.3 ubuntu:latest worker2 Shutdown Shutdown 3 minutes ago
kwmey288bkhp sleep-app.6 ubuntu:latest manager3 Running Running 12 minutes ago
[manager1] (local) root@192.168.0.9 ~/dockerlabs/intermediate/swarm
$ docker node inspect --pretty worker2
ID: uinkvr56fq7zb711ycbifhf4f
Hostname: worker2
Joined at: 2019-03-08 15:12:03.102015148 +0000 utc
Status:
State: Ready
Availability: Drain
Address: 192.168.0.10
Platform:
Operating System: linux
Architecture: x86_64
Resources:
CPUs: 8
Memory: 31.4GiB
Plugins:
Log: awslogs, fluentd, gcplogs, gelf, journald, json-file, local, logentries, splunk
, syslog
Network: bridge, host, ipvlan, macvlan, null, overlay
Volume: local
Engine Version: 18.09.3
TLS Info:
TrustRoot:
-----BEGIN CERTIFICATE-----
MIIBajCCARCgAwIBAgIUcfR/4dysEv9qsbuPTFuIn00WbmowCgYIKoZIzj0EAwIw
EzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMTkwMzA4MTUwNzAwWhcNMzkwMzAzMTUw
NzAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH
A0IABPo7tm+Vxk+CIw9AJEGTlyW/JPotQuVqrbvi34fuK6Ak4cWYU6T1WSiJMHI0
nEGS/1zFIWQzJY0WQbT8eMaqX4ijQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB
Af8EBTADAQH/MB0GA1UdDgQWBBQ6OEYmo8HUfpFnSxJDHWkjf/wWmTAKBggqhkjO
PQQDAgNIADBFAiBy39e7JLpHBH0bONWU8rQZPmY2dtkfHjPOUQNLFBdlkAIhAIpD
Lb6ZrhbEJDcIhlnozKRcPSJi7RWy4/16THIUJdpM
-----END CERTIFICATE-----
Issuer Subject: MBMxETAPBgNVBAMTCHN3YXJtLWNh
Issuer Public Key: MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE+ju2b5XGT4IjD0AkQZOXJb8k+i1C5Wqtu+Lfh+4roCThxZhTpPVZKIkwcjScQZL/XMUhZDMljRZBtPx4xqpfiA==
Run docker node update –availability active
$ docker node update --availability active worker2
worker2
[manager1] (local) root@192.168.0.9 ~/dockerlabs/intermediate/swarm
$ docker node inspect --pretty worker2
ID: uinkvr56fq7zb711ycbifhf4f
Hostname: worker2
Joined at: 2019-03-08 15:12:03.102015148 +0000 utc
Status:
State: Ready
Availability: Active
Address: 192.168.0.10
Platform:
Operating System: linux
Architecture: x86_64
Resources:
CPUs: 8
Memory: 31.4GiB
Plugins:
Log: awslogs, fluentd, gcplogs, gelf, journald, json-file, local, logentries, splunk, syslog
Network: bridge, host, ipvlan, macvlan, null, overlay
Volume: local
Engine Version: 18.09.3
TLS Info:
TrustRoot:
-----BEGIN CERTIFICATE-----
MIIBajCCARCgAwIBAgIUcfR/4dysEv9qsbuPTFuIn00WbmowCgYIKoZIzj0EAwIw
EzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMTkwMzA4MTUwNzAwWhcNMzkwMzAzMTUw
NzAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH
A0IABPo7tm+Vxk+CIw9AJEGTlyW/JPotQuVqrbvi34fuK6Ak4cWYU6T1WSiJMHI0
nEGS/1zFIWQzJY0WQbT8eMaqX4ijQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB
Af8EBTADAQH/MB0GA1UdDgQWBBQ6OEYmo8HUfpFnSxJDHWkjf/wWmTAKBggqhkjO
PQQDAgNIADBFAiBy39e7JLpHBH0bONWU8rQZPmY2dtkfHjPOUQNLFBdlkAIhAIpD
Lb6ZrhbEJDcIhlnozKRcPSJi7RWy4/16THIUJdpM
-----END CERTIFICATE-----
Issuer Subject: MBMxETAPBgNVBAMTCHN3YXJtLWNh
Issuer Public Key: MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE+ju2b5XGT4IjD0AkQZOXJb8k+i1C5Wqtu+Lfh+4roCThxZhTpPVZKIkwcjScQZL/XMUhZDMljRZBtPx4xqpfiA==
Lab07 - Cleaning Up
Execute the docker service rm sleep-app command on manager1 to remove the service called sleep-app.
$ docker service rm sleep-app
sleep-app
[manager1] (local) root@192.168.0.9 ~/dockerlabs/intermediate/swarm
$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
Execute the docker ps command on node1 to get a list of running containers.
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
044bea1c2277 ubuntu "sleep infinity" 17 minutes ago 17 minutes ag distracted_mayer
You can use the docker kill
docker kill yourcontainerid
Finally, let’s remove node1, node2, and node3 from the Swarm. We can use the docker swarm leave –force command to do that.
Lets run docker swarm leave –force on all the nodes to leave swarm cluster.
docker swarm leave --force
Congratulations! You’ve completed this lab. You now know how to build a swarm, deploy applications as collections of services, and scale individual services up and down.
Running Cron Jobs container on Docker Swarm Cluster
A Docker Swarm consists of multiple Docker hosts which run in swarm mode and act as managers (to manage membership and delegation) and workers (which run swarm services). When you create a service, you define its optimal state (number of replicas, network and storage resources available to it, ports the service exposes to the outside world, and more). Docker works to maintain that desired state. For instance, if a worker node becomes unavailable, Docker schedules that node’s tasks on other nodes. A task is a running container which is part of a swarm service and managed by a swarm manager, as opposed to a standalone container.
Let us talk a bit more about Services…
A Swarm service is a 1st class citizen and is the definition of the tasks to execute on the manager or worker nodes. It is the central structure of the swarm system and the primary root of user interaction with the swarm. When one create a service, you specify which container image to use and which commands to execute inside running containers.Swarm mode allows users to specify a group of homogenous containers which are meant to be kept running with the docker service CLI. Its ever running process.This abstraction which is undoubtedly powerful, may not be the right fit for containers which are intended to eventually terminate or only run periodically. Hence, one might need to run some containers for specific period of time and terminate it acccordingly.
Let us consider few example:
- You are a System Administrator who wishes to allow users to submit long-running compiler jobs on a Swarm cluster
- A website which needs to process all user uploaded images into thumbnails of various sizes
- An operator who wishes to periodically run docker rmi $(docker images –filter dangling=true -q) on each machine
Today Docker Swarm doesn’t come with this feature by default. But there are various workaround to make it work. Under this tutorial, we will show you how to run on-off cron-job on 5-Node Swarm Mode Cluster.
Tested Infrastructure
| Platform | Number of Instance | Reading Time |
|---|---|---|
| Play with Docker | 5 | 5 min |
Pre-requisite
- Create an account with DockerHub
- Open PWD Platform on your browser
- Click on Spanner on the left side of the screen to bring up 5-Node Swarm Mode Cluster
Verifying 5-Node Swarm Mode Cluster
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
ENGINE VERSION
y2ewcgx27qs4qmny9840zj92p * manager1 Ready Active Leader
18.06.1-ce
qog23yabu33mpucu9st4ibvp5 manager2 Ready Active Reachable
18.06.1-ce
tq0ed0p2gk5n46ak4i1yek9yc manager3 Ready Active Reachable
18.06.1-ce
tmbcma9d3zm8jcx965ucqu2mf worker1 Ready Active
18.06.1-ce
dhht9gr8lhbeilrbz195ffhrn worker2 Ready Active
18.06.1-ce
Cloning the Repository
git clone https://github.com/crazy-max/swarm-cronjob
cd swarm-cronjob/.res/example
Bring up Swarm Cronjob
docker stack deploy -c swarm_cronjob.yml swarm_cronjob
Listing Docker Services
$ docker service ls
ID NAME MODE REPLICAS IMAGE
PORTS
qsmd3x69jds1 myswarm_app replicated 1/1 crazymax/swarm-cronjob:latest
Visualizing the Swarm Cluster running Cronjob container
git clone https://github.com/collabnix/dockerlabs
cd dockerlabs/intermediate/swarm/visualizer/
docker-compose up -d
Example #1: Running Date container every 5 seconds
Edit date.yml file to change cronjob from * to */5 to run every 5 seconds as shown:
$ cd .res/example/
$ cat date.yml
version: "3.2"
services: test: image: busybox command: date deploy:
labels:
- "swarm.cronjob.enable=true"
- "swarm.cronjob.schedule=*/5 * * * *"
- "swarm.cronjob.skip-running=false"
replicas: 0
restart_policy:
condition: none
[manager1] (local) roo
Bringing up App Stack
docker stack deploy -c date.yml date
Locking docker swarm
Docker introduced native swarm support in Docker v1.12
Docker swarm uses raft consensus algorithm to maintain consensus between the nodes of a swarm cluster. Docker engine maintains raft logs which in turn holds the data of cluster configuration, status of nodes and other sensitive data.
Docker v1.13 introduced the concept of secrets, With secrets a developer could encrypt the sensitive data and give access of decrypted data to particular swarm services using swarm stack configuration.
In v1.13, Docker also encrypts the raft logs at rest and stores the encryption key in /var/lib/docker/swarm/certificates directory in each swarm manager of the cluster. If a malicious user has access to any of the manager nodes, He can easily get the encrption key, decrypt the logs and get hands on sensitive data available in the raft logs (Secrets are also stored in the raft logs).
To bypass this possibilty of disaster and protect the encryption key, Docker introduced swarm autolock feature which allows us to take the ownership of the keys.
Note:
If you enable autolock feature, Whenever your manager node restarts you have to manually supply the key in order for the manager node to decrypt the logs.
Enabling autolock feature
There are various ways enable autolock feature.
While initializing the swarm
docker swarm init --autolock
Store the swarm unlock key in a safe place.
If swarm is already initialized
docker swarm update --autolock=true
Disabling autolock feature
If you want to disable autolock feature and the swarm is already initilized, Use the command mentioned below.
docker swarm update --autolock=false
Retrieving unlock key
If you lost the unlock key and you still have quorum of managers in the cluster, You can retrieve the unlock key by using the following command on the manager.
docker swarm unlock-key
Note: Unlock key can only be retrieved on a unlocked manager.
Unlocking a swarm
If a swarm is locked (When a manager node restarts) one has to manaually unlock the swarm using the unlock key.
docker swarm unlock
Certain scenarios
-
If a manager node is restarted it will be locked by default and has to be unlocked using the swarm unlock key.
-
If a manager node is restarted and you don’t have the unlock key but quorom of managers is maintined in the cluster. Then unlock key can be retrieved using the command mentioned above on any of the unlocked managers.
-
If a manager node is restarted and you don’t have the unlock key and quorum is also lost. Then there is no option bu for the manager is leave the swarm and join bas a new manager.
Contributor
2.5 - Multistage build
Building Dockerfiles in Multistage Build
Before Multistage Build
Keeping the image size small is one of the most difficult aspects of creating images.Every Dockerfile command adds a layer to the image, therefore before adding the next layer, remember to remove any artifacts you don’t need.Traditionally, writing an extremely effective Dockerfile required using shell tricks and other logic to keep the layers as compact as possible and to make sure that each layer only included the items it required from the previous layer and nothing else.
In reality, it was rather typical to use one Dockerfile for development (which included everything required to build your application) and a slimmed-down one for production (which only included your application and precisely what was required to run it).The “builder pattern” has been applied to this.It is not ideal to keep two Dockerfiles up to date.
The example that follows uses a simple React application that is first developed and then has its static content served by a Nginx virtual server.The two Dockerfiles that were utilized to produce the optimized image are listed below.You’ll also see a shell script that illustrates the Docker CLI instructions that must be executed to achieve this result.
Here’s an example of a Dockerfile.build, Dockerfile.main and Dockerfile which adhere to the builder pattern above:
Dockerfile.build
FROM node:alpine3.15
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build
Dockerfile.main
FROM nginx
EXPOSE 3000
COPY ./nginx/default.conf /etc/nginx/conf.d/default.conf
COPY /app/build /usr/share/nginx/html
Build.sh
#!/bin/sh
echo Building myimage/react:build
docker build -t myimage:build . -f Dockerfile.build
docker create --name extract myimage:build
docker cp extract:/app/build ./app
docker rm -f extract
echo Building myimage/react:latest
docker build --no-cache -t myimage/react:latest . -f Dockerfile.main
How to Use Multistage Builds in Docker
In Docker Engine 17.05, multi-stage build syntax was included. You use numerous FROM statements in your Dockerfile while performing multi-stage builds.Each FROM command can start a new stage of the build and may use a different base.Artifacts can be copied selectively from one stage to another, allowing you to remove anything unwanted from the final image.Let’s modify the Dockerfile from the preceding section to use multi-stage builds to demonstrate how this works.
Dockerfile
FROM node:alpine3.15 as build
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build
FROM nginx
EXPOSE 3000
COPY ./nginx/default.conf /etc/nginx/conf.d/default.conf
COPY --from=build /app/build /usr/share/nginx/html
Here only one Dockerfile is required.Additionally, you don’t require a separate build script.Simply run docker build .
There are two FROM commands in this Dockerfile, and each one represents a different build step.In this phase, the application is created and stored in the directory that the WORKDIR command specifies.The Nginx image is first pulled from Docker Hub to begin the second stage.The revised virtual server configuration is then copied to replace the stock Nginx configuration.then, the image created by the prior stage is utilised to copy only the production-related application code using the COPY -from command.
The stages are not named by default; instead, you refer to them by their integer number, which starts at 0 for the first FROM command.However, you can give your stages names by following the FROM instruction with an AS <NAME>.Here we used build as a name.By naming the stages and using the names in the COPY command, this example enhances the prior one.This means that the COPY remains intact even if the instructions in your Dockerfile are later rearranged.
Repurpose an earlier stage as a new stage.
The FROM command allows you to continue where a previous stage ended by referring to it.For instance:
FROM alpine:latest AS builder
...
...
FROM builder AS build1
...
...
FROM builder AS build2
...
...
Use an external image as a “stage”
When using multi-stage builds, you are not limited to copying from stages you created earlier in your Dockerfile. You can use the COPY --from instruction to copy from a separate image, either using the local image name, a tag available locally or on a Docker registry, or a tag ID. The Docker client pulls the image if necessary and copies the artifact from there. The syntax is:
COPY --from=sampleapp:latest home/user/app/config.json app/config.json
Demonstrating Multi-Stage Builds
For demonstration, Let us consider a nodejs project and build a binary out of it. When you execute this binary, it will call a NASA api which returns some interesting facts about today’s date.
Before: docker images
Currently we have two images which I pulled from dockerhub:
alpine (~4Mb)- Lightest version of linux osalpine-node (~70Mb)- alpine + Node/Npm and other dependency.
File structure
-
Dockerfile:- On stage 0 (alias:
builder), we have aalpine-nodeOS which hasnodeandnpmbuilt in it. Its size is~70Mb. This stage will create binary (named asnasa: Line 6) in the currentWORKDIRi.eapp/. - On stage 1, we have
alpineOS. After that, we install some necessary dependencies. InLine 14, we copiednasabinary from previous stage (builder) to current stage. So, we just copied binary and leaving all heavyalpine-nodeOS and other dependencies likenpm(node package manager) etc as binary already have required dependecies (like nodejs) built in it.
- On stage 0 (alias:
-
app/: It’s just a simple node application. It does ahttpscall and fetches data using nasa api. It hasindex.jsandpackage.json. I have usedpkgto build the node binary.
After: docker images
multistage:1.0.0 (56b102754f6d) is the final required image which we built. Its size is ~45Mb. Almost 1/4th of the intermediate image(13bac50ebc1a) built on stage 0. And almost half of alpine-node image.
So, this was a simple example to showcase multi-stage builds feature. For images having multi step (like 10-15 FROM statement), you will find this feature very useful.
2.6 - What happens when Containers are Launched?
What happens when Containers are Launched?
Here is basically what happens when a container is launched:
-
A container is created from a container image. Images are tarballs with a JSON configuration file attached. Images are often nested: for example this Libresonic image is built on top of a Tomcat image that depends (eventually) on a base Debian image. This allows for content deduplication because that Debian image (or any intermediate step) may be the basis for other containers. A container image is typically created with a command like docker build.
-
If necessary, the runtime downloads the image from somewhere, usually some “container registry” that exposes the metadata and the files for download over a simple HTTP-based protocol. It used to be only Docker Hub, but now everyone has their own registry: for example, Red Hat has one for its OpenShift project, Microsoft has one for Azure, and GitLab has one for its continuous integration platform. A registry is the server that docker pull or push talks with, for example.
-
The runtime extracts that layered image onto a copy-on-write (CoW) filesystem. This is usually done using an overlay filesystem, where all the container layers overlay each other to create a merged filesystem. This step is not generally directly accessible from the command line but happens behind the scenes when the runtime creates a container.
-
Finally, the runtime actually executes the container, which means telling the kernel to assign resource limits, create isolation layers (for processes, networking, and filesystems), and so on, using a cocktail of mechanisms like control groups (cgroups), namespaces, capabilities, seccomp, AppArmor, SELinux, and whatnot. For Docker, docker run is what creates and runs the container, but underneath it actually calls the runc command.
Those concepts were first elaborated in Docker’s Standard Container manifesto which was eventually removed from Docker, but other standardization efforts followed. The Open Container Initiative (OCI) now specifies most of the above under a few specifications:
- the Image Specification (often referred to as “OCI 1.0 images”) which defines the content of container images
- the Runtime Specification (often referred to as CRI 1.0 or Container Runtime Interface) describes the “configuration, execution environment, and lifecycle of a container”
- the Container Network Interface (CNI) specifies how to configure network interfaces inside containers, though it was standardized under the Cloud Native Computing Foundation (CNCF) umbrella, not the OCI
Implementation of those standards varies among the different projects. For example, Docker is generally compatible with the standards except for the image format. Docker has its own image format that predates standardization and it has promised to convert to the new specification soon. Implementation of the runtime interface also differs as not everything Docker does is standardized, as we shall see.
The Docker and rkt story
Since Docker was the first to popularize containers, it seems fair to start there. Originally, Docker used LXC but its isolation layers were incomplete, so Docker wrote libcontainer, which eventually became runc. Container popularity exploded and Docker became the de facto standard to deploy containers. When it came out in 2014, Kubernetes naturally used Docker, as Docker was the only runtime available at the time. But Docker is an ambitious company and kept on developing new features on its own. Docker Compose, for example, reached 1.0 at the same time as Kubernetes and there is some overlap between the two projects. While there are ways to make the two tools interoperate using tools such as Kompose, Docker is often seen as a big project doing too many things. This situation led CoreOS to release a simpler, standalone runtime in the form of rkt, that was explained this way:
Docker now is building tools for launching cloud servers, systems for clustering, and a wide range of functions: building images, running images, uploading, downloading, and eventually even overlay networking, all compiled into one monolithic binary running primarily as root on your server. The standard container manifesto was removed. We should stop talking about Docker containers, and start talking about the Docker Platform. It is not becoming the simple composable building block we had envisioned.
One of the innovations of rkt was to standardize image formats through the appc specification, something we covered back in 2015. CoreOS doesn’t yet have a fully standard implementation of the runtime interfaces: at the time of writing, rkt’s Kubernetes compatibility layer (rktlet), doesn’t pass all of Kubernetes integration tests and is still under development. Indeed, according to Brandon Philips, CTO of CoreOS, in an email exchange:
rkt has initial support for OCI image-spec, but it is incomplete in places. OCI support is less important at the moment as the support for OCI is still emerging in container registries and is notably absent from Kubernetes. OCI runtime-spec is not used, consumed, nor handled by rkt. This is because rkt execution is based on pod semantics, while runtime-spec only covers single container execution.
However, according to Dan Walsh, head of the container team at Red Hat, in an email interview, CoreOS’s efforts were vital to the standardization of the container space within the Kubernetes ecosystem: “Without CoreOS we probably would not have CNI, and CRI and would be still fighting about OCI. The CoreOS effort is under-appreciated in the market.” Indeed, according to Philips, the “CNI project and specifications originated from rkt, and were later spun off and moved to CNCF. CNI is still widely used by rkt today, both internally and for user-provided configuration.” At this point, however, CoreOS seems to be gearing up toward building its Kubernetes platform (Tectonic) and image distribution service (Quay) rather than competing in the runtime layer.
CRI-O: the minimal runtime
Seeing those new standards, some Red Hat folks figured they could make a simpler runtime that would only do what Kubernetes needed. That “skunkworks” project was eventually called CRI-O and implements a minimal CRI interface. During a talk at KubeCon Austin 2017, Walsh explained that “CRI-O is designed to be simpler than other solutions, following the Unix philosophy of doing one thing and doing it well, with reusable components.”
Started in late 2016 by Red Hat for its OpenShift platform, the project also benefits from support by Intel and SUSE, according to Mrunal Patel, lead CRI-O developer at Red Hat who hosted the talk. CRI-O is compatible with the CRI (runtime) specification and the OCI and Docker image formats. It can also verify image GPG signatures. It uses the CNI package for networking and supports CNI plugins, which OpenShift uses for its software-defined networking layer. It supports multiple CoW filesystems, like the commonly used overlay and aufs, but also the less common Btrfs.
One of CRI-O’s most notable features, however is that it supports mixed workloads between “trusted” and “untrusted” containers. For example, CRI-O can use Clear Containers for stronger isolation promises, which is useful in multi-tenant configurations or to run untrusted code. It is currently unclear how that functionality will trickle up into Kubernetes, which currently considers all backends to be the same.
CRI-O has an interesting architecture (see the diagram below from the talk slides [PDF]). It reuses basic components like runc to start containers, and software libraries like containers/image and containers/storage, created for the skopeo project, to pull container images and create container filesystems. A separate library called oci-runtime-tool prepares the container configuration. CRI-O introduces a new daemon to handle containers called conmon. According to Patel, the program was “written in C for stability and performance” and takes care of monitoring, logging, TTY allocation, and miscellaneous hazards like out-of-memory conditions.
CRI-O architecture
The conmon daemon is needed here to do all of the things that systemd doesn’t (want to) do. But even though CRI-O doesn’t use systemd directly to manage containers, it assigns containers to systemd-compatible cgroups, so that regular systemd tools like systemctl have visibility into the container resources. Since conmon (and not the CRI daemon) is the parent process of the container, it also allows parts of CRI-O to be restarted without stopping containers, which promises smoother upgrades. This is a problem for Docker deployments right now, where a Docker upgrade requires restarting all of the containers. This is usually not much trouble for Kubernetes clusters, however, because it is easy to roll out upgrades progressively by moving containers around.
CRI-O is the first implementation of the OCI standards suite that passes all Kubernetes integration tests (apart from Docker itself). Patel demonstrated those capabilities by showing a Kubernetes cluster backed by CRI-O, in what seemed to be a routine demonstration of cluster functionalities. Dan Walsh explained CRI-O’s approach in a blog post that explains how CRI-O interacts with Kubernetes:
Our number 1 goal for CRI-O, unlike other container runtimes, is to never break Kubernetes. Providing a stable and rock-solid container runtime for Kubernetes is CRI-O’s only mission in life.
According to Patel, performance is comparable to a normal Docker-based deployment, but the team is working on optimizing performance to go beyond that. Debian and RPM packages are available and deployment tools like minikube, or kubeadm also support switching to the CRI-O runtime. On existing clusters, switching runtimes is straightforward: just a single environment variable changes the runtime socket, which is what Kubernetes uses to communicate with the runtime.
CRI-O 1.0 was released in October 2017 with support for Kubernetes 1.7. Since then, CRI-O 1.8 and 1.9 were released to follow the Kubernetes 1.8 and 1.9 releases (and sync version numbers). Patel considers CRI-O to be production-ready and it is already available in beta in OpenShift 3.7, released in November 2017. Red Hat will mark it as stable in the upcoming OpenShift 3.9 release and is looking at using it by default with OpenShift 3.10, while keeping Docker as a fallback. Next steps include integrating the new Kata Containers virtual-machine-based runtime, kube-spawn support, and more storage backends like NFS or GlusterFS. The team also discussed how it could support casync or libtorrent to optimize synchronization of images between nodes.
containerd: Docker’s runtime gets an API While Red Hat was working on its implementation of OCI, Docker was also working on the standard, which led to the creation of another runtime, called containerd. The new daemon is a refactoring of internal Docker components to combine the OCI-specific bits like execution, storage, and network interface management. It was already featured in the 1.12 Docker release, but wasn’t completed until the containerd 1.0 release announced at KubeCon, which will be part of the upcoming Docker 17.12 (Docker has moved to version numbers based on year and month). And while we call containerd a “runtime”, it doesn’t directly implement the CRI interface, which is covered by another daemon called cri-containerd. So containerd needs more daemons than CRI-O for Kubernetes (five, versus three for CRI-O). Also, at the time of writing, the cri-containerd component is marked as beta but containerd itself is already used in numerous production environments through Docker, of course.
During the Node special interest group (SIG) meeting at KubeCon, Stephen Day described [Speaker Deck] containerd as “designed as a tight core of decoupled components”. Unlike CRI-O, however, containerd supports workloads outside the Kubernetes ecosystem through a Go API. The API is not considered stable yet, although containerd defines a clear release process for making changes to the API and command-line tools. Like CRI-O, containerd is feature complete and passes all Kubernetes tests, but it does not interoperate with systemd’s cgroups.
Next step for the project is to develop more tests and improve performance like memory usage and latency. Containerd developers are also working hard on improving stability. They want to provide Debian and RPM packages for easier installation, and integrate it with minikube and kops as well. There are also plans to integrate Kata Containers more smoothly: runc can already be replaced by Kata for basic integration but cri-containerd integration is not implemented yet.
Interoperability and defaults All of those options are causing a certain amount of confusion in the community. At KubeCon, which runtime to use was a recurring question to speakers. Kubernetes will likely change from Docker to a different runtime, because it doesn’t need all the features Docker provides, and there are concerns that the switch could cause compatibility issues because the new runtimes do not implement exactly the same interface as Docker. Log files, for example, are different in the CRI standard. Some programs also monitor the Docker socket directly, which has some non-standard behaviors that the new runtimes may implement differently, or not at all. All of this could cause some breakage when switching to a different runtime.
The question of which runtime Kubernetes will switch to (if it changes) is also open, which leads to some competition between the runtimes. There was a slight controversy related to that question at KubeCon because CRI-O wasn’t mentioned during the CNCF keynote, something Vincent Batts, a senior engineer at Red Hat, mentioned on Twitter:
It is bonkers that CRI implementations containerd and rktlet are covered in KubeCon keynote, but zero mention of CRI-O, which is a Kubernetes project that’s been 1.0 and actually used in production.
When I prompted him for details about this at KubeCon, Batts explained that:
Healthy competition is good, the problem is unhealthy competition. The CNCF should be better stewards of the projects under their umbrella and shouldn’t fold under pressure to promote certain projects over others.
Batts explained further that Red Hat “may be at a tipping point where some apps could start being deployed as containers instead of RPMs” citing “security concerns (namely with [security] advisory tracking, which is lacking in containers as a packaging format) as the main blocker for such transitions”. With Project Atomic, Red Hat seems to be pivoting toward containers, so the stakes are high for the Linux distribution.
When I talked to CNCF’s COO Chris Aniszczyk at KubeCon, he explained that the CNCF “current policy is to prioritize the marketing of top-level projects”:
Projects like CRIO and Helm are part of Kubernetes in some fashion and therefore part of CNCF, we just don’t market them as heavily as our top level projects which have passed the CNCF TOC [Technical Oversight Committee] approval bar.
Aniszczyk added that “we want to help, we’ve heard the feedback and plan to address things in 2018”, suggesting that one solution could be that CRI-O applies to graduate to a top-level project in CNCF.
During a container runtimes press meeting, Philips explained that the community would decide, through consensus, which runtime Kubernetes would run by default. He compared runtimes to web browsers and suggested that OCI specifications for containers be compared to the HTML5 and Javascript standards: those are evolving standards that get pushed by the different implementations. He argued that this competition is healthy and means more innovation.
A lot of the people involved in writing those new runtimes were originally Docker contributors: Patel was the initial maintainer of the original OCI implementation that led to runc, while Philips was also a core Docker contributor before starting the rkt project. Those people actively collaborate, along with Docker developers, on standardizing those interfaces and all want to see Kubernetes stabilize and improve. The goal, according to Patrick Chazenon from Docker Inc., is to “make container runtimes boring and stable to have people innovate on top of it”. The developers present at the press meeting were happy and proud of what they have accomplished: they have managed to create a single, interoperable specification for containers, and that specification is expanding.
Consolidation and standardization will continue in 2018 The current big topic in container standardization is not runtimes as much as image distribution (i.e. container registries), which is likely to standardize, again, in a specification built around Docker’s distribution system. There is also work needed to follow the Linux kernel changes, for example cgroups v2.
The reality is that each runtime has its own advantages: containerd has an API so it can be used to build customized platforms, while CRI-O is a simpler runtime specifically targeting Kubernetes. Docker and rkt are on another level, providing more than simply the runtime: they also provide ways of building containers or pushing to registries, for example.
Right now, most public cloud infrastructure still uses Docker as a runtime. In fact, even CoreOS uses Docker instead of rkt in its Tectonic platform. According to Philips, this is because “there is a substantial integration ecosystem around Docker Engine that our customers rely on and it is the most well-tested option across all existing Kubernetes products.” Philips says that CoreOS may consider supporting alternate runtimes for Tectonic, “if alternative container runtimes provide significant improvements to Kubernetes users”:
At this point containerd and CRI-O are both very young projects due to the significant amount of new code each project developed this year. Further, they need to reach the maturity of third-party integrations across the ecosystem from logging, monitoring, security, and much more.
Philips further explained CoreOS’s position in this blog post:
So far the primary benefits of the CRI for the Kubernetes community have been better code organization and improved code coverage in the Kubelet itself, resulting in a code base that’s both higher quality and more thoroughly tested than ever before. For nearly all deployments, however, we expect the Kubernetes community will continue to use Docker Engine in the near term.
During the discussion in the press meeting, Patel likewise said that “we don’t want Kubernetes users to know what the runtime is”. Indeed, as long as it works, users shouldn’t care. Besides, OpenShift, Tectonic, and other platforms abstract runtime decisions away and pick their own best default matching their users' requirements. The question of which runtime Kubernetes chooses by default is therefore not really a concern for those developers, who prefer working on building consensus on standard specifications. In a world of conflict, seeing those developers working together cordially was definitely a breath of fresh air.
Reference: https://lwn.net/Articles/741897/
3 - Using Jenkins
Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early. CI doesn’t get rid of bugs, but it does make them dramatically easier to find and remove.
CI/CD merges development with testing, allowing developers to build code collaboratively, submit it the master branch, and checked for issues. This allows developers to not only build their code, but also test their code in any environment type and as often as possible to catch bugs early in the applications development lifecycle. Since Docker can integrate with tools like Jenkins and GitHub, developers can submit code in GitHub, test the code and automatically trigger a build using Jenkins, and once the image is complete, images can be added to Docker registries. This streamlines the process, saves time on build and set up processes, all while allowing developers to run tests in parallel and automate them so that they can continue to work on other projects while tests are being run.
A Typical CI workflow look like:
- Developer pushes a commit to GitHub
- GitHub uses a webhook to notify Jenkins of the update
- Jenkins pulls the GitHub repository, including the Dockerfile describing the image, as well as the application and test code.
- Jenkins builds a Docker image on the Jenkins slave node
- Jenkins instantiates the Docker container on the slave node, and executes the appropriate tests
- If the tests are successful the image is then pushed up to Dockerhub or Docker Trusted Registry.
Under these series of blog post, I will help you get started with Continuous Integration with Jenkins, Docker & GitHub under $0. I will be using Play with Docker platform which comes for free for the general public.
Let’s get started..
Go to your browser and type – https://labs.play-with-docker.com/
Open up a “New Instance” on the left hand side of the screen.
Cloning the Repository
Let us start with a simple HTML webpage application. I have a sample GITHUB repository already created for you which contains Dockerfile and Jenkinsfile under the root of the repository as shown below:
$ git clone https://github.com/ajeetraina/webpage
What is Jenkinsfile & Jenkins Pipeline?
A Jenkinsfile is a text file that contains the definition of a Jenkins Pipeline and is checked into source control.
Jenkins Pipeline (or simply “Pipeline”) is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. It provides an extensible set of tools for modeling simple-to-complex delivery pipelines “as code”. The definition of a Jenkins Pipeline is typically written into a text file (called a Jenkinsfile) which in turn is checked into a project’s source control repository. I have created a sample Jenkinsfile which clones the repository, builds the Docker Image, test it and then push it to Dockerhub automatically using Jenkins.
node {
def app
stage('Clone repository') {
/* Let's make sure we have the repository cloned to our workspace */
checkout scm
}
stage('Build image') {
/* This builds the actual image; synonymous to
* docker build on the command line */
app = docker.build("ajeetraina/webpage")
}
stage('Test image') {
/* Ideally, we would run a test framework against our image.
* Just an example */
app.inside {
sh 'echo "Tests passed"'
}
}
stage('Push image') {
/* Finally, we'll push the image with two tags:
* First, the incremental build number from Jenkins
* Second, the 'latest' tag.
* Pushing multiple tags is cheap, as all the layers are reused. */
docker.withRegistry('https://registry.hub.docker.com', 'dockerhub') {
app.push("${env.BUILD_NUMBER}")
app.push("latest")
}
}
}
In the above Jenkinsfile, there are various stages like cloning the SCM, build, test and pushing the Docker image. You will need to change it to your respective GITHUB repo if you want to try to build your own Docker image.
Let us quickly setup Jenkins on PWD platform and believe me it’s just a matter of a single docker-compose CLI. Before we proceed, ensure that the below permission is granted so as to get Docker plugin to work flawlessly –
$chmod 666 /var/run/docker.sock
It’s time to clone the repository:
$ git clone https://github.com/ajeetraina/jenkins
Cloning into ‘jenkins’…
remote: Counting objects: 18, done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 18 (delta 2), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (18/18), done.
[node1] (local) root@192.168.0.53 ~/webpage
$ cd jenkins
[node1] (local) root@192.168.0.53 ~/webpage/jenkins
$docker-compose up
Copy the password of the jenkins starting with “5ee571..” as shown in the above screenshot.
Click on 8080 port which appear on the top of the screen and this will redirect you to Jenkins page.
Once you add the Administrator password which we copied earlier, then it will ask to install plugins. Choose “Install Suggested Plugins” on the left hand side as shown below:
Go back to Jenkins dashboard and click “New Item”. Supply any name(i.e demo-webpage) and select “Pipeline” as type of the project.
This will show up a page with Build Triggers as one of the option. Select “Pipeline script from SCM” and choose GIT under SCM. This will allow you to enter your GITHUB URL which in my case is https://github.com/ajeetraina/webpage. This will automatically add Jenkinsfile under Script Path. Click on “Apply” and then Save.
Click on “Build Now” option on the left side of the Jenkins screen. This traverse through the build pipeline i.e cloning the repository, building Docker Image , testing it and pushing it to your Dockerhub account as shown below:
Once you click on “Build Now” you will see that it initiates the build pipeline.
Jenkins Pipeline Stages View:
Click on “Console Output” to show detailed build process as shown below:
A Quick View of Stage View:
Till now, we have a new Docker Image pushed to Dockerhub automatically. Let’s go ahead and verify it by bringing up webpage container in the separate instance window as shown below:
Here you see…a static HTML page appears running on port 80. This was built as part of Jenkins pipeline.
In order to trigger this pipeline every 15 minutes, you can go back to Jenkins page for specific job and then choose “Build Periodically” under Configure page as shown below:
4 - Using Jenkins 2
Containerising your application is like shoving your app and all its dependencies into a box. Except the box is infinitely replicable. Whatever happens in the box, stays in the box - unless you explicitly take something out or put something in. And when it breaks, you’ll just throw it away and get a new one.
Containers make your app easy to run on different computers - ideally, the same image should be used to run containers in every environment stage from development to production.
This project is your guide for building a Docker image, and then setting up Jenkins 2 to build and publish the image automatically, whenever you commit changes to your code repository.
Requirements
To run through this guide, you will need the following:
1. To build and run the Docker image locally: Mac OS X or Linux, and Docker installed.
2. To set up Jenkins to build the image automatically: Access to a Jenkins 2.x installation.
Our application
For this guide, we’ll be using a very basic example: a Hello World server written with Node. We’ll also need a package.json, which tells Node some basic things about our application. To be able to build a Docker image with our app, we’ll need a Dockerfile. You can think of it as a blueprint for Docker: it tells Docker what the contents and parameters of our image should be.
Docker images are often based on other images. For this exercise, we are basing our image on the official Node Docker image. This makes our job easy, and our Dockerfile very short. The grunt work of installing Node and its dependencies in the image is already done in our base image; we’ll just need to include our application.
The Dockerfile is best stored with the code - this way any changes to it are versioned along with the actual application code.
Follow the instructions given below
1. Copy all files in your project folder.
2. Run npm install to install any dependencies for app (if we had any).
3. Specify npm start as the command Docker runs when the container starts.
Building the image locally
To build the image on your own computer, navigate to the project directory (the one with your application code and the Dockerfile), and run docker build:
docker build . -t getintodevops-hellonode:1
This instructs Docker to build the Dockerfile in the current directory with the tag getintodevops-hellonode:1. You will see Docker execute all the actions we specified in the Dockerfile
Running the image locally
If the above build command ran without errors, congratulations: your first Docker image is ready!
Let’s make sure the image works as expected by running it:
docker run -it -p 8000:8000 getintodevops-hellonode:1
The above command tells Docker to run the image interactively with a pseudo-tty, and map the port 8000 in the container to port 8000 in your machine.
You should now be able to check if the server responds in your local port 8000:
curl http://127.0.0.1:8000
Assuming it does, you can quit the docker run command with CTRL + C.
Building the image in Jenkins
Now that we know our Docker image can be built, we’ll want to do it automatically every time there is a change to the application code.
For this, we’ll use Jenkins. Jenkins is an automation server often used to build and deploy applications.
Note: this guide assumes you are running Jenkins 2.0 or newer, with the Docker Pipeline plugin and Docker installed.
If you don’t have access to a Jenkins installation, refer to https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins.
Pipelines as code: The Jenkinsfile
Just like Dockerfiles, I’m a firm believer in storing Jenkins pipeline configuration as code in a Jenkinsfile along with the application code.
It generally makes sense to have everything in the same repository; the application code, what the build artifact should look like (Dockerfile), and how said artifact is created automatically (Jenkinsfile).
Let’s think about our pipeline for a second. We can identify four stages:
Simple pipeline
We’ll need to tell Jenkins what our stages are, and what to do in each one of them. For this we’ll write a Jenkins Pipeline specification in a Jenkinsfile.
Configuring Docker Hub with Jenkins
To store the Docker image resulting from our build, we’ll be using Docker Hub. You can sign up for a free account at https://hub.docker.com.
We’ll need to give Jenkins access to push the image to Docker Hub. For this, we’ll create Credentials in Jenkins, and refer to them in the Jenkinsfile.
As you might have noticed in the above Jenkinsfile, we’re using docker.withRegistry to wrap the app.push commands - this instructs Jenkins to log in to a specified registry with the specified credential id (docker-hub-credentials).
On the Jenkins front page, click on Credentials -> System -> Global credentials -> Add Credentials.
Add your Docker Hub credentials as the type Username with password, with the ID docker-hub-credentials
Creating a job in Jenkins
The final thing we need to tell Jenkins is how to find our repository. We’ll create a Pipeline job, and point Jenkins to use a Jenkinsfile in our repository.
Here are the steps:
Click on New Item on the Jenkins front page.
Type a name for your project, and select Pipeline as the project type.
Select Poll SCM and enter a polling schedule. The example here, H/5 * * * * will poll the Git repository every five minutes.
5 - DockerTools
DockerTools was built with a purpose. It is being used by Collabnix Slack community internally to target the most popular tools and technique and coming up with the best practices around these tools.
Follow the Collabnix Twitter account for updates on new list additions.
Have Questions? Join us over Slack and get chance to be a part of 6400+ DevOps enthusiasts.
Image Slim
DockerSlim - Minify and Secure Docker containers
Minicon - Minimization of the filesystem for containers
Watchtower - A process for automating Docker container base image updates
Image Build Toolkit
BuildKit - Toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner
Dive - Tool for exploring each layer in a docker image
Docker-Squash - Squashing helps with organizing images in logical layers.
Image Security
TerraScan -Detect compliance and security violations across Infrastructure as Code to mitigate risk before provisioning cloud native infrastructure.
Clair - Vulnerability Static Analysis for Containers
Trivy - Vulnerability Scanner for Containers and other Artifacts, Suitable for CI - Aqua Security
DeepSource - Static Analysis for DockerFiles
DockerScan - A Docker analysis & hacking tools
DockerHub
DockerHub Scraper - Scraping DockerHub
Docker Volume
Flocker - Data Volume Manager for your Dockerized applications.
Runtime Security
Tracee - Linux Runtime Security and Forensics using eBPF
Container Management
Portainer - Making Docker management easy. https://www.portainer.io
Drone - continuous delivery system built on container technology
Machine Learning
Paddle Serving - A flexible, high-performance carrier for machine learning models
Development
Conan Docker - accelerating the development and Continuous Integration of C and C++ projects.
Networking
Libnetwork - Docker libnetwork plugin for Calico http://www.projectcalico.org
Libnetwork - networking for containers
Caddy Gen - Automated Caddy reverse proxy for docker containers
Swarm
Orbiter - Autoscaler for Docker Swarm
Maintainer
Contributor
Last Updated: 1 March 2022