Proxy protocol not rewriting remote_addr from DOKS load balancer

[Syntaf]

Describe the bug
My remote address remains an internal address even after setting up and accepting the proxy protocol from my DOKS (Digital Ocean Kubernetes) load balancer.

To Reproduce
Steps to reproduce the behavior:

1. I followed (only) step 4 of DO's instructions to configure my load balancer to use the proxy protocol.
2. I followed the this repositories docs to setup my ingress to handle the proxy protocol

   # values.yaml
   
   # ingress resource & server snippet
   ingress:
     enabled: true
     annotations:
       nginx.org/server-snippets: |
         gzip on;
         gzip_types text/plain text/html application/json;
   
         server_name *.embolt.app;
   
         if ($host = "www.embolt.app") {
           return 301 $scheme://embolt.app$request_uri;
         }
      host:
         # ...
   
   # nginx-ingress configuration
   nginx-ingress:
     controller:
       # ...
       enableSnippets: true
       config:
         entries:
           proxy-protocol: "True"
           real-ip-header: "proxy_protocol"
           set-real-ip-from: "<internal IP>"
      service:
        annotations:
          service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"

3. I'm still only able to see the internal IP when viewing ingress logs from NGINX using the default log format

   10.120.0.5 - - [13/Feb/2024:02:05:49 +0000] "GET / HTTP/1.1" 200 4781 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36" "-"
   

Expected behavior
I would have expected to see either the initial address (remote_addr) or final segment of the logs (x_forwarded_for) to have contained the client's IP address.

Your environment

• Version of the Ingress Controller - 0.17.1
• Version of Kubernetes - 1.26.7-do.0
• Kubernetes platform - Digital Ocean (DOKS)
• Using NGINX or NGINX Plus - NGINX

Additional context

• I'm able to confirm that my LB is using the proxy protocol, for example when I removed the proxy protocol configuration from my nginx config but retained the load balancer annotations I received the following error confirming it was using the protocol:

  10.120.0.5 - - [11/Feb/2024:22:24:58 +0000] "PROXY TCP4 68.224.237.201 138.68.36.61 54359 443" 400 157 "-" "-" "-"
  

• My application works fine using the configuration above, I just simply do not seem to have any client IP information
• I'm wondering if this might mean my NGINX server snippets or configuration elsewhere is invalid if the proxy protocol is being read fine but my client addresses still aren't showing up.

Thank you for anyone that might have ideas on this!

[github-actions]

Hi @Syntaf thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this 🙂

Cheers!

[j1m-ryan]

Hi @Syntaf. I set up NGINX Ingress Controller on DOKS today to look into this.

I deployed NGINX Ingress Controller v3.4.2 (this has helm chart version 1.1.2) along with the example in examples/ingress-resources/complete-example

I made one modification to this example, deleting the TLS stuff in cafe-ingress.yaml, otherwise it's identical.

I used the following in my config map

config:
  entries:
    proxy-protocol: "True"
      real-ip-header: "proxy_protocol"
      set-real-ip-from: "0.0.0.0/0"

And added the required annotation you mentioned from the Digital Ocean Kubernetes docs, on the NGINX Ingress Controller service, just as you have done

service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"

I then sent a curl request to my digital ocean kubernetes cluster
curl -H "Host: cafe.example.com" http://redacted/coffee

I then looked in the NIC logs, and saw my personal ip address which I will also redact.

XXX.XXX.XXX.XXX - - [13/Feb/2024:16:52:55 +0000] "GET /coffee HTTP/1.1" 200 161 "-" "curl/8.2.1" "-"

So it does appear to be working on v3.4.2

I saw that you are using the helm chart version 0.17.1 which corresponds to NIC v3.1.1 which was released in May. I uninstalled NIC v3.4.2. and pulled the helm chart version 0.17.1, adjusted the values.yaml as above and installed NIC again. I then ran the curl command again against the new external IP of the service

curl -H "Host: cafe.example.com" http://redacted/coffee

I got the same result again in the NIC logs, I could see my own IP address.

The only real differences I can see from our setups, is the snippet that you are using. I do not suspect this would cause this issue though. And the other difference being the that you are using an internal ip for set-real-ip-from.

[Syntaf]

@j1m-ryan Huge thank you for jumping in to assist, you're right that I was incorrectly using an internal IP for set-real-ip-from. Switching this to "0.0.0.0/0" worked like a charm and my nginx logs are now showing the client IP 🍾

I think I misunderstood the documentation and assumed I'd need the internal-IP of the load balancer, though I probably should have tried something like 0.0.0.0/0 ahead of time in hindsight before throwing in the towel.

Considering my ingress is behind load balancer, do you have any opinions on whether 0.0.0.0/0 is safe to use as it's essentially saying any incoming IP address can rewrite the remote address? I want to say initially I'm OK with this as the ingress is not directly accepting external traffic, though I'm not sure if there's any security implications I should be concerned about.

[j1m-ryan]

No bother @Syntaf. 0.0.0.0/0 is safe here, if you trust whats directly in front of NGINX Ingress Controller, which in this case will be the digital ocean load balancer.

[vepatel]

@Syntaf is it okay to close this one now?

[shaun-nx]

Hi @Syntaf
I'm going to close this issue as it looks like what @j1m-ryan provided resolved the original issue.
Please feel free to open another issue if the problem persists, and thanks for reaching out!

Also, if you ever want to discuss issues and discussions with the team, we run a bi-weekly zoom call where we will traige issue and discussions. It's a great opportunity to talk directly to the team!

All the details about the call is here on our main README Hope to see you there some time! 😄