Monitoring Docker (Hands On Instructions)
Monitoring on Docker
This exercise requires Docker to be installed somewhere. You could do this on your laptop, on a Vagrant box, or on a cloud provider. If installing for the first time visit https://store.docker.com/search?type=edition&offering=community to get it setup.
- Navigate to the Agent Installation page in the Datadog application (https://app.datadoghq.com/account/settings#agent/docker).
- Select Docker and copy the first command. Run it where ever you have Docker installed.
- You now have the Datadog agent running on your Docker machine.
- Run
docker exec -it dd-agent bash
to login to a bash shell on that container. - You can now run any of the Datadog commands you have on Linux. For instance, run
/etc/init.d/datadog-agent info
. - You can try other commands, or type
exit
to return to your host machine. - Start a Redis container by running the following command:
docker run -d --name ad-redis redis
. - Run the Datadog Agent Info command. Also run the Agent’s configcheck command (
docker exec -it dd-agent /etc/init.d/datadog-agent configcheck
). Is Redis being monitored? (If it isn’t try again in a few seconds.) - Create a new file called
Dockerfile
in a new directory calledredisdocker
. Edit the file and add a single line:FROM redis
- Build the new image using the following command:
docker build -t redisdocker .
- Run the redisdocker image using the command:
docker run --name dd-redis -d redisdocker
. The Redis docker container based on our modified image is now running. - Run the Datadog Agent Info command again. Is the new database being monitored? Why do you think that is?
Using Docker Compose
Let’s switch over to using docker-compose.
Create a docker-compose.yaml file in the same directory as your Dockerfile with the following contents:
version: "3" networks: main: services: datadog: image: datadog/docker-dd-agent volumes: - /var/run/docker.sock:/var/run/docker.sock - /proc:/host/proc:ro - /sys/fs/cgroup:/host/sys/fs/cgroup:ro environment: - API_KEY - SD_BACKEND=docker networks: - main frontendredis: image: redis networks: - main backendredis: build: . networks: - main
Run
docker-compose up -d
.Run the datadog info command (
docker-compose exec datadog /etc/init.d/datadog-agent info
). How many redis instances are being monitored?Modify the backendredis service to show the following:
backendredis: build: . networks: - main labels: com.datadoghq.ad.check_names: '["redisdb"]' com.datadoghq.ad.init_configs: '[{}]' com.datadoghq.ad.instances: '[{"host":"%%host%%", "port": "%%port%%"}]'
Stop and remove the containers then run
docker-compose up
again.Run the Datadog Agent Info command again, how many instances are being monitored?
Now with Kubernetes
Let’s create the same environment in Kubernetes. Your Kubernetes environment could be local using miniKube, on a Rancher or similar environment or on a cloud service such as Google Cloud. If you don’t have a Kubernetes environment, setting up miniKube is super quick and easy. The easiest install docs I have seen are here: https://github.com/kubernetes/minikube/releases. You will also need kubectl which you can find here: https://kubernetes.io/docs/tasks/tools/install-kubectl/
- Add the Datadog Agent to your Kubernetes environment by creating a dd-agent.yaml file. You can find the contents of this file, will your API KEY already filled in here: https://app.datadoghq.com/account/settings#agent/kubernetes
- Start the DaemonSet by running
kubectl apply -f dd-agent.yaml
Create a yaml file called
redisapp.yaml
with the following contents:apiVersion: v1 kind: Service metadata: name: redis-master labels: app: redis tier: backend role: master spec: ports: - port: 6379 targetPort: 6379 selector: app: redis tier: backend role: master --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: redis-master spec: replicas: 1 template: metadata: labels: app: redis role: master tier: backend spec: containers: - name: master image: gcr.io/google_containers/redis:e2e # or just image: redis resources: requests: cpu: 100m memory: 100Mi ports: - containerPort: 6379 --- apiVersion: v1 kind: Service metadata: name: redis-slave labels: app: redis tier: backend role: slave spec: ports: - port: 6379 selector: app: redis tier: backend role: slave --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: redis-slave spec: replicas: 2 template: metadata: labels: app: redis role: slave tier: backend spec: containers: - name: slave image: gcr.io/google_samples/gb-redisslave:v1 resources: requests: cpu: 100m memory: 100Mi env: - name: GET_HOSTS_FROM value: dns ports: - containerPort: 6379
Add the Deployments by running
kubectl apply -f redisapp.yaml
.Find the name of your agent pod:
kubectl get pods -l app=dd-agent
. Then run the Datadog Agent info command on that pod:kubectl exec [agent pod] /etc/init.d/datadog-agent info
.You should see that only a single redis instance is being monitored.
Update the redis-slave deployment as follows:
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: redis-slave spec: replicas: 2 template: metadata: annotations: service-discovery.datadoghq.com/slave.check_names: '["redisdb"]' service-discovery.datadoghq.com/slave.init_configs: '[{}]' service-discovery.datadoghq.com/slave.instances: '[{"host":"%%host%%","port":"%%port%%"}]' labels: app: redis role: slave tier: backend spec: containers: - name: slave image: gcr.io/google_samples/gb-redisslave:v1 resources: requests: cpu: 100m memory: 100Mi env: - name: GET_HOSTS_FROM value: dns ports: - containerPort: 6379
Apply the redisapp.yaml file again. Try the Datadog Agent Info command to see how many instances are being monitored.
Can you see where the info in the annotations come from?
Once you have the Redis container running, choose another local integration from the Integration documentation. Start the container using an image found on the Docker Hub and create a configuration. At the end of this session, lets talk about how you did it.
Presentation | Notes