How Do Run Jenkins As A Service ? [ Windows And Linux ]

In this comprehensive guide, I have explained what is Docker, its evolution, underlying core Linux concepts, and how information technology works.

Docker has get the defacto standard when it comes to container-based implementations. Docker is the base for container-based orchestration from modest-calibration implementations to big-calibration enterprise applications.

Docker gained so much popularity and adoption in the DevOps community quickly because it'southward developed for portability and designed for modernistic microservice architecture.

In this weblog, yous will learn,

What is Docker?
Larn about Docker and see why Docker is benign and different from other container technologies.
Docker cadre architecture and its key components
Container development and the underlying concept of Linux Containers
What is a container, and what Linux features brand it piece of work?
The departure between a process, container, and a VM

Here, the idea is to get your nuts correct to understand what Docker really is and how it works.

What is Docker?

Docker is a popular open-source project written in get and developed past Dotcloud (A PaaS Company).

It is a container engine that uses the Linux Kernel features like namespaces and control groups to create containers on top of an operating system. So y'all tin call information technology OS-level virtualization.

Docker was initially built on peak of Linux containers (LXC). Later Docker replaced LXC with its container runtime libcontainer (now role of runc). I take explained the core LXC & container concepts towards the end of the article.

Yous might ask how Docker is different from a Linux Container (LXC) as all the concepts and implementation look similar?

Besides just being a container engineering, Docker has well-defined wrapper components that brand packaging applications easy. Before Docker, it was non easy to run containers. Meaning it does all the work to decouple your application from the infrastructure by packing all application system requirements into a container.

For example, if yous accept a Coffee jar file, you can run it on any server which has java installed. Aforementioned manner, once you lot package a container with required applications using Docker, y'all tin run it on any other host which has Docker installed.

We will have containers up and running by executing a few Docker commands & parameters.

Difference Betwixt Docker & Container

Docker is a technology or a tool developed to manage containers efficiently.

So, can I run a container without Docker?

Yes! of class. Yous can use LXC technology to run containers on Linux servers. In addition, the latest tools like Podman offers like workflows similar Docker.

Things you should know most Docker:

Docker is not LXC
Docker is not a Virtual Machine Solution.
Docker is non a configuration management system and is not a replacement for Chef, Puppet, Ansible, etc.
Docker is not a platform as a service applied science.
Docker is not a container.

What Makes Docker So Great?

Docker has an efficient workflow for moving the application from the developer'due south laptop to the test environment to product. Yous will understand more well-nigh it when you wait at a practical example of packaging an application into a Docker image.

Do you know that starting a docker container takes less than a second?

It is incredibly fast, and it tin run on whatever host with compatible Linux Kernel. (Supports Windows as well)

Annotation: you cannot run a Windows container on a Linux host considering in that location is no Linux Kernel support for Windows. You can read about Windows containers from here

Docker uses a Copy-on-write spousal relationship file organisation for its image storage. Therefore, when changes are made to a container, only the changes will exist written to disk using copy on the write model.

With Re-create on write, you will have optimized shared storage layers for all your containers.

Docker Adoption Statistics

Here is the google trends data on Docker. Yous can see it has been an exploding topic for the last five years.

Here is a survey result from Datadog, which shows the rising in Docker adoption.

docker 2018 5 final v2 — source: datadog.com

Docker Core Architecture

The following sections will look at the Docker architecture and its associated components. We will besides wait at how each component works together to make Docker work.

Docker architecture has changed a few times since its inception. When I published the get-go version of this article, Docker was built on acme of LXC

Hither are some notable architectural changes that happened for the Docker

Docker moved from LXC to libcontainer in 2014
runc – a CLI for spinning upward containers that follow all OCI specifications.
containerd – Docker separated its container management component to containerd in 2016

OCI: Open Container Initiative is an open industry standard for container runtime and specifications.

When Docker was initially launched, information technology had a monolithic architecture. At present it is separated into the following 3 dissimilar components.

Docker Engine (dockerd)
docker-containerd (containerd)
docker-runc (runc)

Docker and other large organizations contributed to a standard container runtime and management layers. Hence containerd and runc are now part of the Deject Native Foundation with contributors from all the organizations.

Note: When installing Docker, all these components get installed. You don't have to install it separately. For exaplanation, we are showing it as different components.

Now let's have a looks at each Docker component.

Docker Engine

Docker engine comprises the docker daemon, an API interface, and Docker CLI. Docker daemon (dockerd) runs continuously as dockerd systemd service. It is responsible for building the docker images.

To manage images and run containers, dockerd calls the docker-containerd APIs.

docker-containerd (containerd)

containerd is another organization daemon service than is responsible for downloading the docker images and running them every bit a container. Information technology exposes its API to receive instructions from the dockerd service

docker-runc

runc is the container runtime responsible for creating the namespaces and cgroups required for a container. It and then runs the container commands inside those namespaces. runc runtime is implemented as per the OCI specification.

Read this splendid 3 part weblog post series to understand more about container runtimes.

How Does Docker Piece of work?

We take seen the core building blocks of Docker.

At present let'due south understand the Docker workflow using the Docker components.

Docker Components

The following official high-level docker architecture diagram shows the common Docker workflow.

docker architecture min — image source: docs.docker.com

Docker ecosystem is equanimous of the following four components

Docker Daemon (dockerd)
Docker Client
Docker Images
Docker Registries
Docker Containers

What is a Docker Daemon?

Docker has a customer-server compages. Docker Daemon (dockerd) or server is responsible for all the actions related to containers.

The daemon receives the commands from the Docker client through CLI or REST API. Docker client tin can exist on the aforementioned host as a daemon or present on whatever other host.

By default, the docker daemon listens to the docker.sock UNIX socket. If you accept whatsoever use example to access the docker API remotely, you need to betrayal it over a host port. One such use case is running Docker as Jenkins agents.

If yous want to run Docker inside Docker, you can employ the docker.sock from the host automobile.

What is a Docker Image?

Images are the basic edifice blocks of Docker. It contains the Os libraries, dependencies, and tools to run an application.

Images can be prebuilt with application dependencies for creating containers. For instance, if you want to run an Nginx spider web server as a Ubuntu container, you lot need to create a Docker image with the Nginx binary and all the Os libraries required to run Nginx.

What is a Dockerfile?

Docker has a concept of Dockerfile that is used for building the image. A Dockerfile a text file that contains one command (instructions) per line.

Hither is an case of a Dockerfile.

A docker image is organized in a layered fashion. Every didactics on a Dockerfile is added a layer in an image. The topmost writable layer of the prototype is a container.

Every image is created from a base of operations paradigm.

For example, if y'all can use a base epitome of Ubuntu and create another image with the Nginx application. A base of operations epitome can exist a parent image or an paradigm built from a parent image. Bank check out his docker article to know more than about information technology.

Y'all might ask where this base of operations prototype (Parent epitome) comes from? In that location are docker utilities to create the initial parent base image. Information technology takes the required Os libraries and bakes them into a base image. You don't have to do this because you will get the official base of operations images for Linux distros.

The summit layer of an image is writable and used by the running container. Other layers in the image are read-but.

What is a Docker Registry?

It is a repository (storage) for Docker images.

A registry can be public or private. For case, Docker Inc provides a hosted registry service called Docker Hub. It allows yous to upload and download images from a central location.

Note: By default, when you install docker, it looks for images from the public Docker hub unless you lot specify a custom registry in Docker settings.

Other Docker hub users can admission all your images if your repository is public. You can also create a private registry in Docker Hub.

Docker hub acts like git, where you tin build your images locally on your laptop, commit it, and and so be pushed to the Docker hub.

Tip: When using docker in enterprise networks/project, ready your own docker registries instead of using the public docker hub. All cloud providers have their own container registry services.

What is a Docker Container?

Docker Containers are created from existing images. Information technology is a writable layer of the prototype.

If you lot effort to relate paradigm layers and a container, here is how information technology looks for a ubuntu-based image.

Yous can bundle your applications in a container, commit it, and make it a gilded epitome to build more containers from it.

Containers tin be started, stopped, committed, and terminated. If you terminate a container without committing it, all the container changes will be lost.

Ideally, containers are treated every bit immutable objects, and it is not recommended to make changes to a running container. Instead, make changes to a running container simply for testing purposes.

2 or more than containers can be linked together to form tiered awarding architecture. However, hosting hight scalable applications with Docker has been made easy with the appearance of container orchestration tools like kubernetes.

Evolution of Containers

If y'all remember containerization is a new technology, information technology is non. Google has been using its container technology in its infrastructure for years.

The concept of containers started way back in the 2000s. In fact, the roots get back to 1979 when we had chroot, a concept of changing the root directory of a process.

Hither is a list of container-based projects that started in 2000.

Year	Technology
2000	FreeBSD jails introduced the container concept.
2003	Linux-V server project released with the concept of Bone-level virtualization
2005	Solaris Zones– Os-level virtualization project introduced
2007	Google released a paper on Generic Process Containers
2008	The initial release of LXC containers
2011	cloudfoundry announced warden
2013	lcmcty– Open-sourced by Google
2013	Docker project was appear by DotCloud
2014	Rocket. (rkt) announced by CoreOS
2016	Windows container preview every bit released as function of Windows server 2016

What is a Linux container (LXC)?

Now let's sympathise what a Linux Container is.

In a typical virtualized environment, one or more virtual machines run on top of a concrete server using a hypervisor like Xen, Hyper-V, etc.

On the other paw, Containers run on top of the operating organization's kernel. You can call it OS-level virtualization. Earlier getting into the underlying container concepts, y'all need to understand two key Linux concepts.

Userspace: All the lawmaking required to run user programs (applications, procedure) is called userspace. When you lot initiate a program action, for example, to create a file, the process in the userspace makes a system call to Kernal infinite.
Kernel Space: This is the heart of the operating system, where you lot have the kernel lawmaking, which interacts with the system hardware, storage, etc.

A container is a Process.

You starting time a process when you showtime an awarding, for example, an Nginx web server. A procedure itself is a cocky-independent instruction with limited isolation.

What if nosotros tin isolate the process with simply files and configuration required to run and operate. That is what a container does.

A container is a procedure with plenty isolation of userspace components to give a feeling of a dissever operating organization.

The parent container process may have a child process. And then you can say a container is also a grouping of processes.

For example, when you outset an Nginx service, it starts a parent Nginx procedure. The parent process spans its child processes like cache manager, enshroud loader, and workers.

nginx child process min — Epitome Src: https://www.nginx.com

So when you start an Nginx container, you are starting a master Nginx procedure in its isolated environment.

I will show you lot this practically in the beneath sections.

Each container has its isolated userspace, and you can run multiple containers on a single host.

Does that mean a container has the whole OS?

No. Dissimilar a VM with its kernel, a container contains the required files related to a specific distro and uses the shared host kernel.

More interestingly, you can run different Linux distros-based containers on a single host that shares the same kernel space.

For case, you tin can run an RHEL, CentOS, a SUSE-based container on an Ubuntu server. It is possible considering only the userspace is unlike for all the Linux distros, and kernel infinite is the same.

Underlying Concept of Linux Containers

The following image gives you a visual representation of Linux continers.

Containers are isolated using the ii Linux kernel features called namespaces and control groups.

A existent-globe analogy would exist an Apartment building. Fifty-fifty though information technology's a single big building, each condo/flat is isolated for individual households having their own identity with metered water, gas, and electricity. We use concrete, steel structures, and other construction materials to establish this isolation. You do not have visibility into other homes unless they allow y'all in.

Similarly, yous tin chronicle this to a single host containing multiple containers. To isolate containers with their CPU, memory, IP accost, mountain points, processes, you need two Linux kernel features called namespaces and control groups.

Linux Namespaces

A container is all nigh having a well-isolated environment to run a service (Process). To accomplish that level of isolation, a container should have its file organisation, IP address, mount points, process IDs, etc. You lot can attain this using the Linux Namespaces.

Namespaces are responsible for containers' mount points, user, IP address, procedure management, etc. And then substantially, it sets boundaries for the containers.

Following are the primal namespaces in the Linux

pid namespace: Responsible for isolating the procedure (PID: Procedure ID).
net namespace: Information technology manages network interfaces (Internet: Networking).
ipc namespace: It manages access to IPC resources (IPC: InterProcess Advice).
mnt namespace: Responsible for managing the filesystem mount points (MNT: Mount).
uts namespace: Isolates kernel and version identifiers. (UTS: Unix Timesharing Organisation).
usr namespace: Isolates user IDs. In simple words, it isolates the user ids betwixt the host and container.
Cgroup namespace: It isolates the control group information from the container process

Using the above namespaces, a container tin take its network interfaces, IP address, etc. Each container volition take its namespace and the processes running within that namespace will not have any privileges outside its namespace.

Interestingly, you lot can list the namespaces in a Linux machine using the lsns control.

Linux Command groups

We don't specify any memory or CPU limit when starting a service. Instead, we exit it to the kernel to prioritize and classify resource for the services.

Withal, you can explicitly set CPU retentiveness limits for your services using a Linux kernel feature chosen CGroups. Information technology is not a straightforward arroyo; you lot demand to brand some extra configurations and tweaks to brand information technology work.

Since you can run multiple containers inside a host, there should be a mechanism to limit resources usage, device admission, etc. Here is where command groups come into the picture.

Linux command groups manage the resource used by a container. You can restrict CPU, memory, network, and IO resource of a container Linux control group.

Then what happens if I don't limit the CPU & Memory resources of a container?

A single container might employ all the host resources, leaving other containers to crash because of resource unavailability.

Tools similar Docker abstract abroad all the complex backend configurations and let you specify these resource limits with unproblematic parameters.

Why Are Containers Better Than VMs?

Containers have some key advantages over VMs. Let'southward take a looks at those.

Resource Utilisation & Toll

You tin utilise VMs to run your applications independently, which ways one service per VM. But it can still be underutilized. And resizing a VM is not an easy task for a production application.
On the other paw, containers can run with very minimal CPU and retention requirements. Also, you tin even run multiple containers inside a VM for application segregation. Plus, resizing a container takes seconds.

Provisioning & Deployment

Provisioning a VM and deploying applications on it might take minutes to hours, depending on the workflow involved. Even rollback takes time.
But you can deploy a container in seconds and curlicue it back in seconds equally well.

Drift Direction

Drift direction in VMs is not like shooting fish in a barrel. You lot need to have total-fledged automation and processes to ensure all the environments are similar. Following immutable deployment models avoids drift in VM environments.
Once the paradigm gets backed, it will be the same in all the environments for containers. So for any changes, you demand to start making changes in dev env and re-bake the container image.

Docker FAQs

What is the difference between containerd & runc?

containerd is responsible for managing the container and runc is responsible for running the containers (create namespaces, cgroups and run commands inside the container) with the inputs from containerd

What is the deviation between the Docker engine & the Docker daemon?

Docker engine is composed of the docker daemon, rest interface, and the docker CLI. Docker daemon is the systemd dockerd service responsible for building the docker images and sending docker instructions to containerd runtime.

Conclusion

By now, you should have a practiced understanding of what Docker is and how information technology works.

The all-time feature of Docker is collaboration. Docker images tin can be pushed to a repository and pulled down to any other host to run containers from that image.

Moreover, the Docker hub has thousands of images created by users, and you can pull those images down to your hosts based on your application requirements. Besides, it is primarily used in container orchestration tools similar kubernetes

If you want to run Docker for product workloads, make certain you lot follow Docker images' recommended practices.

You can read my article on how to reduce docker image size where I take listed down all the standard approaches to optimize the docker image.

As well, if you are trying to become a DevOps engineer, I highly recommend you get hands-on feel with Docker.