Skip to main content

Docker DNS Failing - On Prem

Matt Bell
  • Updated

Overview

  The following article give’s guidance on how to troubleshoot and implement a fix should your docker DNS be broken and is unable to be remediated.

If your containers fail to communicate through DNS by default the nginx container will crash due to the requirement within the configuration to resolve the tomcat container name.

 

To view your containers run: 

 

docker ps

 

To view the logs run:

 

docker logs -f iriusrisk-nginx-ssl-prod

 

Or for later versions

 

docker logs -f iriusrisk-nginx

 

Example of the log output for the nginx container will present something similar to the following:

2022/03/14 13:40:01 [emerg] 1#1: host not found in upstream "tomcat8" in /etc/nginx/conf.d/irius.conf:23
nginx: [emerg] host not found in upstream "tomcat8" in /etc/nginx/conf.d/irius.conf:23

 

Troubleshoot

The first steps to resolve this are to try some simple first steps,

  • Restart the docker service.
sudo systemctl restart docker
sudo systemctl status docker
ÔùÅ docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2022-05-04 10:50:29 UTC; 9s ago
Docs: https://docs.docker.com
Process: 6395 ExecStartPre=/usr/libexec/docker/docker-setup-runtimes.sh (code=exited, status=0/SUCCESS)
Process: 6384 ExecStartPre=/bin/mkdir -p /run/docker (code=exited, status=0/SUCCESS)
Main PID: 6402 (dockerd)
Tasks: 41
Memory: 52.2M
CGroup: /system.slice/docker.service
Ôö£ÔöÇ6402 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --default-ulimit nofile=32768:65536
Ôö£ÔöÇ7677 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 192.168.64.2 -container-port 443
Ôö£ÔöÇ7685 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 443 -container-ip 192.168.64.2 -container-port 443
Ôö£ÔöÇ7700 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 192.168.64.2 -container-port 80
ÔööÔöÇ7706 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 80 -container-ip 192.168.64.2 -container-port 80

May 04 10:50:28 CSV4.iriusrisk.com dockerd[6402]: time="2022-05-04T10:50:28.978808969Z" level=info msg="Loading containers: done."
May 04 10:50:29 CSV4.iriusrisk.com dockerd[6402]: time="2022-05-04T10:50:29.026986975Z" level=info msg="Docker daemon" commit=b0f5bc3 graphdriver(s)=overlay2 version=20.10.7
May 04 10:50:29 CSV4.iriusrisk.com dockerd[6402]: time="2022-05-04T10:50:29.027073685Z" level=info msg="Daemon has completed initialization"
May 04 10:50:29 CSV4.iriusrisk.com systemd[1]: Started Docker Application Container Engine.
May 04 10:50:29 CSV4.iriusrisk.com dockerd[6402]: time="2022-05-04T10:50:29.058626629Z" level=info msg="ignoring event" container=470a45f956576a24a95e49211a26e45fe25baa3edef04485d112d1244e7a4162...ts.TaskDelete"
May 04 10:50:29 CSV4.iriusrisk.com dockerd[6402]: time="2022-05-04T10:50:29.059151592Z" level=info msg="API listen on /run/docker.sock"
May 04 10:50:30 CSV4.iriusrisk.com dockerd[6402]: time="2022-05-04T10:50:30.591461501Z" level=info msg="ignoring event" container=470a45f956576a24a95e49211a26e45fe25baa3edef04485d112d1244e7a4162...ts.TaskDelete"
May 04 10:50:32 CSV4.iriusrisk.com dockerd[6402]: time="2022-05-04T10:50:32.136756102Z" level=info msg="ignoring event" container=470a45f956576a24a95e49211a26e45fe25baa3edef04485d112d1244e7a4162...ts.TaskDelete"
May 04 10:50:33 CSV4.iriusrisk.com dockerd[6402]: time="2022-05-04T10:50:33.771984265Z" level=info msg="ignoring event" container=470a45f956576a24a95e49211a26e45fe25baa3edef04485d112d1244e7a4162...ts.TaskDelete"
May 04 10:50:36 CSV4.iriusrisk.com dockerd[6402]: time="2022-05-04T10:50:36.154953343Z" level=info msg="ignoring event" container=470a45f956576a24a95e49211a26e45fe25baa3edef04485d112d1244e7a4162...ts.TaskDelete"
Hint: Some lines were ellipsized, use -l to show in full.

 

  • Validate there are no other aspects preventing network connectivity i.e enter the nginx container and try to ping an the IP address of the tomcat container (in this case 172.26.0.3).

 

docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' iriusrisk-tomcat
172.26.0.3
docker exec -it iriusrisk-nginx sh
/ $ ping 172.26.0.3

If the service restart does not enable the DNS service for docker and IP addresses are responding to ping then the below solution should resolve the issue.

 

Solution

First get the IPV4 address that is currently assigned to your tomcat container.

docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' iriusrisk-tomcat
172.26.0.3

 

Define a subnet for the configured network within your docker compose YAML using the first 3 octets of the returned value from the above command (172.26.0.x) This will need placed at the top of you docker compose yaml as laid out below under the networks property.

 

version: '3.7'

networks:
   iriusrisk-frontend:
     ipam:
       config:
         - subnet: 172.26.0.0/24

services:

   nginx:
     ports:
       - "80:80"
       - "443:443"
     environment:
       - NG_SERVER_NAME=example.iriusrisk.com
     image: continuumsecurity/iriusrisk-prod:nginx-analytics
     container_name: iriusrisk-nginx
     networks:
       - iriusrisk-frontend
           ipv4_address: 172.26.0.2
     mem_reservation: 50M
     mem_limit: 200M
     cpu_shares: 128
     restart: unless-stopped
     volumes:
        - "./cert.pem:/etc/nginx/ssl/star_iriusrisk_com.crt:ro"
        - "./key.pem:/etc/nginx/ssl/star_iriusrisk_com.key:ro"
    tomcat:
     environment:
      - STARTLEFT_URL=http://startleft:8081
      - IRIUS_DB_URL=jdbc\:postgresql\://172.17.0.1\:5432/iriusprod?user\=iriusprod&password\=alongandcomplexpassword2523
      - IRIUS_EDITION=saas
      - IRIUS_EXT_URL=http\://example.iriusrisk.com
      - grails_env=production
     image: continuumsecurity/iriusrisk-prod:tomcat-4
     container_name: iriusrisk-tomcat
     networks:
      - iriusrisk-frontend
          ipv4_address: 172.26.0.3
     mem_reservation: 2G
     cpu_shares: 1024
     volumes:
       - "./logs:/usr/local/tomcat/logs"

 

Once you have set your static IP addresses you will need to overwrite the nginx configuration by manually mounting a modified version of the irius.conf file that set’s your newly defined static ip address for the tomcat container.

 

To obtain this file, run the following docker command. (docker image may vary on certain deployments, please validate this using docker ps)

 

docker cp iriusrisk-nginx:/etc/nginx/conf.d/irius.conf irius.conf

 

The next step is to then modify the irius.conf file (vim) and delete the DNS names and replace them with statically assigned IP addresses you have defined in your docker subnet and your docker-compose.yml file.

 

Before (NOTE: This is only a snippet and for illustration purposes only)

 

 

}

server {

    listen 443 ssl;

    server_name example.iriusrisk.com;

    proxy_set_header Host $host:$server_port;

    proxy_set_header X-Forwarded-Proto https;

    proxy_set_header X-Real-IP $remote_addr;

    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;




    # ---------------- MAINTAINANCE CONFIG------------#

    #location / {

    #     root   /var/www/html/;

    #     index 50x.html;

    #}

    #--------------------------------------------




    # ---------------- PRODUCTION CONFIG ---------------#

    location / {

        proxy_pass http://tomcat:8080/;

        proxy_cookie_path ~.* "/;secure;";

        proxy_intercept_errors on;

    }




    location /api {

        proxy_pass http://tomcat:8080/api;

        proxy_cookie_path ~.* /;

        proxy_intercept_errors off;

    }




    location /VAADIN/ {

        proxy_pass http://tomcat:8080/VAADIN/;

        proxy_intercept_errors on;

    }


    # Error pages

    location ~ ^/[54][0][04x].(html|png)$ {

        root   /var/www/html/;

        proxy_redirect off;

    }

After 

}

server {

    listen 443 ssl;

    server_name example.iriusrisk.com;

    proxy_set_header Host $host:$server_port;

    proxy_set_header X-Forwarded-Proto https;

    proxy_set_header X-Real-IP $remote_addr;

    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;




    # ---------------- MAINTAINANCE CONFIG------------#

    #location / {

    #     root   /var/www/html/;

    #     index 50x.html;

    #}

    #--------------------------------------------




    # ---------------- PRODUCTION CONFIG ---------------#

    location / {

        proxy_pass http:// 172.26.0.3:8080/;

        proxy_cookie_path ~.* "/;secure;";

        proxy_intercept_errors on;

    }




    location /api {

        proxy_pass http:// 172.26.0.3:8080/api;

        proxy_cookie_path ~.* /;

        proxy_intercept_errors off;

    }




    location /VAADIN/ {

        proxy_pass http:// 172.26.0.3:8080/VAADIN/;

        proxy_intercept_errors on;

    }

    # Error pages

    location ~ ^/[54][0][04x].(html|png)$ {

        root   /var/www/html/;

        proxy_redirect off;

    }

 

Save the file and exit.

 

Once you have the updated file saved, you will then need to mount the newly defined config in the nginx container, to do this you will need to mount this as a volume within the docker-compose config as laid out below, ensure the irius.conf file is in the same directory as your docker-compose.yml

 

- "./irius.conf:/etc/nginx/conf.d/irius.conf"
version: '3.7'

networks:
  iriusrisk-frontend:
    ipam:
      config:
        - subnet: 172.26.0.0/24

services:

  nginx:
    ports:
      - "80:80"
      - "443:443"
    environment:
      - NG_SERVER_NAME=example.iriusrisk.com
    image: continuumsecurity/iriusrisk-prod:nginx-analytics
    container_name: iriusrisk-nginx
    networks:
      - iriusrisk-frontend
          ipv4_address: 172.26.0.2
    mem_reservation: 50M
    mem_limit: 200M
    cpu_shares: 128
    restart: unless-stopped
    volumes:
      - "./cert.pem:/etc/nginx/ssl/star_iriusrisk_com.crt:ro"
      - "./key.pem:/etc/nginx/ssl/star_iriusrisk_com.key:ro"
      - "./irius.conf:/etc/nginx/conf.d/irius.conf"

 

Once set, restart your docker-compose configuration and you should now see the nginx container communicating with tomcat.

 

docker-compose down && docker-compose up -d
9e963e53d749 continuumsecurity/iriusrisk-prod:nginx "/nginx-entrypoint.sh" 6 seconds ago Up 3 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp iriusrisk-nginx
1d01c862e4ab continuumsecurity/iriusrisk-prod:tomcat-4 "/entrypoint/dynamic" 8 seconds ago Up 5 seconds iriusrisk-tomcat

 

Related articles

Comments

0 comments

Article is closed for comments.

Powered by Zendesk