BKM: NGINX, Redirect, and DNS Resolver, under Docker Swarm and Kubernetes - OpenVisualCloud/Smart-City-Sample GitHub Wiki

There are two modes of name resolution going on when NGINX works within a container environment:

• Static Name Resolution: If a service name is explicitly specified in the NGINX configuration file, for example, proxypass http://backend-service;, NGINX uses the system DNS resolver to resolve the service names. Under docker swarm or Kubernetes, the service names can be any deployed services. Namespace is applied automatically as specified under /etc/resolv.conf.

Note that if you use NGINX as your container entry point or command, your container instance may fail a few times (depending on the service startup order) until the dependent service is up and running (thus the service name can be resolved.) This is the expected behavior.

• Dynamic Name Resolution: If a service name is constructed at runtime (typically used as part of the X-Accel-Redirect request), NGINX uses its own internal DNS resolver to cache the DNS data and resolve the names. We need to specify the cluster DNS IP address in the NGINX configuration as well as taking care of the namespace expansion. The rest sections describe the BKM of making X-Accel-Redirect work under Docker Swarm and Kubernetes.

X-Accel-Redirect is a flexible feature that allows a service be redirected to any backend servers. For example, the frontend service receives a request to serve a video file. The service parses the video path to decide which backend server may host the video file and then forward the request to the backend server, which actually delivers the file.

The frontend service implementation is as simple as:

    @gen.coroutine
    def get(self):
        ...
        self.add_header('X-Accel-Redirect','/redirect/'+protocol+'/'+host+'/'+path)
        self.set_status(200,'OK')

where the service implementation figures out the backend service URL and issues a X-Accel-Redirect request. NGINX will forward the request to the specified URL. The tricky part is in the NGINX configuration.

See Also: redirect.py

The NGINX configuration must define a redirect block to specify where to pass on the request.

        location ~* ^/redirect/(.*?)/(.*?)/(.*)$ {
            internal;
            include resolv.conf;
            proxy_set_header Host $2;
            proxy_set_header Authorization '';
            proxy_set_header Cookie '';
            proxy_max_temp_file_size 0;
            proxy_connect_timeout 60s;
            proxy_pass $1://$2/$3$is_args$args;
        }

where /etc/nginx/resolv.conf is a file we will construct to tell NGINX where to look for external DNS. The proxy_set_header and proxy_pass lines ensure that NGINX redirects the request to the specified service.

If you only plan to use Docker Swarm, then you can replace the line include resolv.conf with resolver 127.0.0.11;. Kubernetes requires more attention thus we must construct the resolver configuration at runtime.

See Also: nginx.conf

While Docker Swarm uses a hard-coded DNS IP address 127.0.0.11, the Kubernetes DNS IP address is not fixed. To write for both Docker Swarm and Kubernetes, we need to figure out the DNS IP address at runtime, after the container is started and before NGINX starts.

def setup_nginx_resolver():
    with open("/etc/resolv.conf","rt") as fd:
        for line in fd:
            if not line.startswith("nameserver"): continue
            with open("/etc/nginx/resolv.conf","wt") as fdr:
                fdr.write("resolver "+line.strip().split(" ")[1]+";")
            return

where the setup_nginx_resolver function extracts the DNS IP address from the system /etc/resolv.conf and then writes to the NGINX configuration file /etc/nginx/resolv.conf. The setup_nginx_resolver function must run before NGINX starts.

Needlessly to say, you need to set permission to make /etc/nginx/resolv.conf writable.

See Also: Setup NGINX Resolver

NGINX does not understand Kubernetes namespace thus the service name must be specified as a fully qualified name in the request. For example, for the default name space, the service name must be specified as <service>.default.svc.cluster.local.