Implement IP White Listing through K8s Ingress - CloudCommandos/JohnChan GitHub Wiki

Assumption:
You are using HAProxy as your load balancer in front of Nginx Ingress Controller.

The problem with using a load balancer in front of Nginx Ingress Controller is that the client IP is replaced with the load balancer's IP. This renders the IP white-listing functionality of K8s Ingress as useless. We want to be able to white-list specific public IPs to access specific applications.

Nginx Ingress Controller from quay.io

For the Nginx Ingress Controller from quay.io's container image, edit the nginx-configuration's ConfigMap object as such:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  use-proxy-protocol: "true"
  proxy-real-ip-cidr: "10.0.1.100, 10.0.1.101, 10.0.1.102"

K8s Ingress Configuration

Add nginx.ingress.kubernetes.io/whitelist-source-range into your Kubernetes ingress manifest file with as many CIDRs as such:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: name-virtual-host-ingress
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "1.1.1.1/32, 2.2.2.2/32"
    ...

HAProxy Configuration

Modify your haproxy.cfg as such:

# For http:
frontend http-ingress
    bind 0.0.0.0:58080
    mode http
    option tcplog
    default_backend http-ingress
backend http-ingress
    mode http
    option tcplog
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
        server svr1 10.0.1.103:80 check fall 3 rise 2 send-proxy
        server svr2 10.0.1.104:80 check fall 3 rise 2 send-proxy

# For https:
frontend https-ingress
    bind 0.0.0.0:50443
    mode tcp
    option tcplog
    default_backend https-ingress
backend https-ingress
    mode tcp
    option tcplog
    option ssl-hello-chk
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
        server svr1 10.0.1.103:443 check fall 3 rise 2 send-proxy
        server svr2 10.0.1.104:443 check fall 3 rise 2 send-proxy

# For other tcp protocols (such as mqtt):
frontend other-tcp-ingress
    bind 0.0.0.0:51883
    mode tcp
    option tcplog
    default_backend other-tcp-ingress
backend other-tcp-ingress
    mode tcp
    option tcplog
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
        server svr1 10.0.1.103:443 check fall 3 rise 2
        server svr2 10.0.1.104:443 check fall 3 rise 2

We are not using X Headers forwarding because https does not support it. X headers forwarding can be applied to http to pass on the source IP, however, Nginx Ingress Controller can only operate either with real_ip_header X_Forwarded_For or real_ip_header proxy_protocol. Therefore we go with the one that works for both http and https --- Proxy Protocol. For other tcp protocols the send-proxy option should be omitted.