Enable ENABLE_V6_EGRESS on Clusters with Mixed IPv6/IPv4 Subnets

[sergeylanzman]

What type of PR is this?
feature

Which issue does this PR fix:
N/A

What does this PR do / Why do we need it:
Today, the inability to enable ENABLE_V6_EGRESS on clusters with both IPv6 and IPv4 subnets is addressed in this PR. This enhancement allows for the coexistence of nodes operating exclusively on IPv6, where pods can utilize IPv6 egress, alongside nodes functioning solely on IPv4 without ENABLE_V6_EGRESS being active. This integration enhances network flexibility and functionality.

If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:

Will this PR introduce any new dependencies?:
N/A

Will this break upgrades or downgrades? Has updating a running cluster been tested?:
No

Does this change require updates to the CNI daemonset config files to work?:
No

If this change does not work with a "kubectl patch" of the image tag, please explain why.
No

Does this PR introduce any user-facing change?:

Enable ENABLE_V6_EGRESS on Clusters with Mixed IPv6/IPv4 Subnets

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jdn5126]

@sergeylanzman I am confused by this PR. EKS only supports a single Kubernetes IP family for pods, so is this an IPv4 cluster that you are trying to migrate to IPv6? Are pods using IPv4 or IPv6 addresses?

Note that ENABLE_V6_EGRESS just controls whether the egress-cni plugin is chained to the VPC CNI plugin. It is up to the application whether it connects to an IPv4 or IPv6 endpoint.

[sergeylanzman]

@jdn5126 My use case involves multiple groups of nodes. In one of these groups, there is a node that supports both IPv6 and IPv4 (private), and all pods are operating on IPv4. My requirement is to enable pods from the node group with IPv6 to communicate with IPv6 destinations. EKS offers the ENABLE_V6_EGRESS feature for this purpose. However, there's a complication: enabling ENABLE_V6_EGRESS on a node without IPv6 leads to the aws-cni pod crashing upon start-up. Since my cluster has different types of node groups, I'm unable to use ENABLE_V6_EGRESS for just the specific node group with IPv6.

currently I need use IPv6 on one of group nodes, but's it's can be usable for slowly migration to ipv6

[jdn5126]

If this is an IPv4 cluster, how do you have nodes with IPv6-only subnets? Or maybe I am misreading and it is an IPv4 cluster where only some nodes have IPv6 addresses assigned.

If it is the latter, we unfortunately have a limitation where ENABLE_V6_EGRESS requires all nodes to have an IPv6 address assigned to work.

[sergeylanzman]

@jdn5126 second option. Some node with ipv6

Why this limitation? Ipv6 not in internal cluster network only egress. So it’s per node configuration. We can support v6_egress only on node with ipv6. On others nodes(ipv4 only) it’s still disabled.

[jdn5126]

@jdn5126 second option. Some node with ipv6

Why this limitation? Ipv6 not in internal cluster network only egress. So it’s per node configuration. We can support v6_egress only on node with ipv6. On others nodes(ipv4 only) it’s still disabled.

The limitation was added to enforce consistency, as we cannot detect between the case where a node is supposed to have an IPv6 address (but is misconfigured) vs the case where the node should not have an IPv6 address.

I see your point, though. We could instead skip adding the egress plugin and log a warning here if the IPv6 address cannot be found.

I'll need to talk to my team about this to see if this is a change we are ok with.

[sergeylanzman]

Exactly, that's the functionality introduced by the PR. Here's the breakdown:

If ENABLE_V6_EGRESS is enabled and the node has IPv6: IPv6 egress is enabled.
If ENABLE_V6_EGRESS is enabled but the node does not have IPv6: IPv6 egress remains disabled.
If ENABLE_V6_EGRESS is disabled: IPv6 egress is disabled, regardless of the node's IPv6 capabilities.

https://github.com/aws/amazon-vpc-cni-k8s/pull/2754/files#diff-29a659a6a9cbfa72a4a83016bef85bdf10a07402f34f7d73fa7fa5eb275a8ec1R876-R882

I waiting for your team's anwser
Thank you

[jdn5126]

Thanks for explaining @sergeylanzman. I think we can make this PR a bit simpler by just keeping egressEnabled as false when we fail to get an IPv6 address on an IPv4 node. Will respond after discussing with our project manager

[jdn5126]

@sergeylanzman we are good to take an enhancement here where we log a warning and continue when ENABLE_V6_EGRESS is set and the node does not have an IPv6 address. I think we need to rework this PR to only a 5 to 10-line change, though. Lmk if you are interested in doing that, otherwise I can take it up soon.

[sergeylanzman]

I am interesting. What need?

[jdn5126]

I am interesting. What need?

So all we should need to do is change this error to a warning and set egressEnabled = false: https://github.com/aws/amazon-vpc-cni-k8s/blob/master/cmd/aws-vpc-cni/main.go#L273

[sergeylanzman]

done https://github.com/aws/amazon-vpc-cni-k8s/pull/2754/files#diff-cc76031459dec8376271a9ac940081b48726527a60aa7bc293d8686a46091893R273

what about other places where used ENABLE_V6_EGRESS like cmd/aws-vpc-cni-init/main.go or pkg/networkutils/network.go ?

[jdn5126]

done https://github.com/aws/amazon-vpc-cni-k8s/pull/2754/files#diff-cc76031459dec8376271a9ac940081b48726527a60aa7bc293d8686a46091893R273

what about other places where used ENABLE_V6_EGRESS like cmd/aws-vpc-cni-init/main.go or pkg/networkutils/network.go ?

Usage in both cases should be harmless. In the init container, it will enable IPv6 forwarding on the node, which will not impact anything if you are not sending IPv6 traffic.

And in pkg/networkutils/network.go, it adds iptables rules to prevent node-local traffic for IPv6, which is fine.

[sergeylanzman]

Done
can you review and approve?

[jdn5126]

The change generally looks good, thank you! I will pull it tomorrow and do some further testing.

[sergeylanzman]

any updates?

[jdn5126]

any updates?

Sorry for the delay, I added this validation to my current sprint and plan to finish by the end of the week

[sergeylanzman]

Thank you

[jdn5126]

Testing that I carried out:

1. I created an IPv4 cluster with AZs and 6 subnets (1 public, 1 private in each AZ)
2. I associated an IPv6 CIDR with the cluster VPC and routed all egress traffic through an EIGW with a default IPv6 route
3. I associated the IPv6 CIDR to the public and private subnets in AZ us-west-2a
4. I assigned IPv6 addresses to the primary ENI for all nodes in us-west-2a
5. I enabled ENABLE_V6_EGRESS on the aws-node daemonset and validated that all pods came up. The pods running on nodes without an IPv6 address assigned returned the following warning:

time="2024-01-24T20:14:56Z" level=warning msg="To support IPv6 egress, node primary ENI must have a global IPv6 address, error: get instance metadata: failed to retrieve ipv6 - EC2MetadataError: failed to make EC2Metadata request\n<?xml version=\"1.0\" encoding=\"iso-8859-1\"?>\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"\n\t\t \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n<html xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"en\" lang=\"en\">\n <head>\n  <title>404 - Not Found</title>\n </head>\n <body>\n  <h1>404 - Not Found</h1>\n </body>\n</html>\n\n\tstatus code: 404, request id: "
time="2024-01-24T20:14:56Z" level=info msg="Successfully copied CNI plugin binary and config file."

6. I validated that pods in us-west-2a can reach out to IPv6 endpoints:

% kubectl run curl --restart=Never --rm -i --tty --image=curlimages/curl:latest --overrides='{"spec": { "nodeSelector": {"kubernetes.io/hostname": "ip-192-168-8-210.us-west-2.compute.internal"}}}' --command -- curl -6 icanhazip.com
2600:1f14:1916:a000:2a75:8a1f:a21a:a2df
pod "curl" deleted

7. I validated that pods in other AZs cannot reach IPv6 endpoints:

% kubectl run curl --restart=Never --rm -i --tty --image=curlimages/curl:latest --overrides='{"spec": { "nodeSelector": {"kubernetes.io/hostname": "ip-192-168-83-246.us-west-2.compute.internal"}}}' --command -- curl -6 icanhazip.com
curl: (7) Failed to connect to icanhazip.com port 80 after 5 ms: Couldn't connect to server
pod "curl" deleted
pod default/curl terminated (Error)

8. I validated that basic IPv4 traffic, including service routing, works on all nodes.
9. I ran VPC CNI integration tests and validated that they passed.

@sergeylanzman can you share some of the test coverage that you got?

[sergeylanzman]

yes, my test lab for now:

1. ipv4 cluster 1.28 with 2 autoscaling group multi az(3 az)
2. 1 autoscaling group running on ipv4 only subnets and second on ipv4+ipv6 subnets. 3 subnets for each group
3. All subnets are private, working with nat gateway.
4. IPv6 subnets has route ipv6 to vpc endpoint
5. I running ping from pods running on nodes with ipv6 to ipv6 subnet from the route table - working
6. ipv4 nodes working with 100+ pods regular with many deployments 24 hours.
7. Cluster includes fargate node - no impact

│ aws-node Installed /host/opt/cni/bin/aws-cni                                                                                                                            │
│ aws-node time="2024-01-23T19:36:28Z" level=info msg="Starting IPAM daemon... "                                                                                          │
│ aws-node time="2024-01-23T19:36:28Z" level=info msg="Checking for IPAM connectivity... "                                                                                │
│ aws-node Installed /host/opt/cni/bin/egress-cni                                                                                                                         │
│ aws-node time="2024-01-23T19:36:30Z" level=info msg="Copying config file... "                                                                                           │
│ aws-node time="2024-01-23T19:36:30Z" level=info msg="Successfully copied CNI plugin binary and config file."                                                            │
│ aws-eks-nodeagent {"level":"info","ts":"2024-01-23T19:36:39Z","msg":"version","GitVersion":"","GitCommit":"","BuildDate":""}                                            │
│ aws-vpc-cni-init time="2024-01-23T19:36:20Z" level=info msg="Copying CNI plugin binaries ..."                                                                           │
│ aws-vpc-cni-init time="2024-01-23T19:36:20Z" level=info msg="Copied all CNI plugin binaries to /host/opt/cni/bin"                                                       │
│ aws-vpc-cni-init time="2024-01-23T19:36:20Z" level=info msg="Found primaryMAC 02:b0:07:e3:fc:ed"                                                                        │
│ aws-vpc-cni-init time="2024-01-23T19:36:20Z" level=info msg="Found primaryIF eth0"                                                                                      │
│ aws-vpc-cni-init time="2024-01-23T19:36:20Z" level=info msg="Updated net/ipv4/conf/eth0/rp_filter to 2\n"                                                               │
│ aws-vpc-cni-init time="2024-01-23T19:36:20Z" level=info msg="Updated net/ipv4/tcp_early_demux to 1\n"                                                                   │
│ aws-vpc-cni-init time="2024-01-23T19:36:20Z" level=info msg="Updated net/ipv6/conf/all/forwarding to 1\n"                                                               │
│ aws-vpc-cni-init time="2024-01-23T19:36:20Z" level=info msg="Updated net/ipv6/conf/eth0/accept_ra to 2\n"                                                               │
│ aws-vpc-cni-init time="2024-01-23T19:36:20Z" level=info msg="CNI init container done"                                                                                   

init container set net/ipv6/conf/ because ENABLE_V6_EGRESS enabled for init container too

log from ipv4 nodes

│ aws-node Installed /host/opt/cni/bin/aws-cni                                                                                                                            │
│ aws-node Installed /host/opt/cni/bin/egress-cni                                                                                                                         │
│ aws-node time="2024-01-23T19:29:25Z" level=info msg="Starting IPAM daemon... "                                                                                          │
│ aws-node time="2024-01-23T19:29:25Z" level=info msg="Checking for IPAM connectivity... "                                                                                │
│ aws-node time="2024-01-23T19:29:28Z" level=info msg="Copying config file... "                                                                                           │
│ aws-vpc-cni-init time="2024-01-23T19:29:24Z" level=info msg="Copying CNI plugin binaries ..."                                                                           │
│ aws-vpc-cni-init time="2024-01-23T19:29:24Z" level=info msg="Copied all CNI plugin binaries to /host/opt/cni/bin"                                                       │
│ aws-vpc-cni-init time="2024-01-23T19:29:24Z" level=info msg="Found primaryMAC 12:3e:ce:49:92:8d"                                                                        │
│ aws-vpc-cni-init time="2024-01-23T19:29:24Z" level=info msg="Found primaryIF eth0"                                                                                      │
│ aws-vpc-cni-init time="2024-01-23T19:29:24Z" level=info msg="Updated net/ipv4/conf/eth0/rp_filter to 2\n"                                                               │
│ aws-vpc-cni-init time="2024-01-23T19:29:24Z" level=info msg="Updated net/ipv4/tcp_early_demux to 1\n"                                                                   │
│ aws-vpc-cni-init time="2024-01-23T19:29:24Z" level=info msg="Updated net/ipv6/conf/all/forwarding to 1\n"                                                               │
│ aws-vpc-cni-init time="2024-01-23T19:29:24Z" level=info msg="Updated net/ipv6/conf/eth0/accept_ra to 2\n"                                                               │
│ aws-vpc-cni-init time="2024-01-23T19:29:24Z" level=info msg="CNI init container done"                                                                                   │
│ aws-eks-nodeagent {"level":"info","ts":"2024-01-23T19:29:25Z","msg":"version","GitVersion":"","GitCommit":"","BuildDate":""}                                            │
│ aws-node time="2024-01-23T19:29:28Z" level=warning msg="To support IPv6 egress, node primary ENI must have a global IPv6 address, error: get instance metadata: failed  │
│ aws-node time="2024-01-23T19:29:28Z" level=info msg="Successfully copied CNI plugin binary and config file."

[jdn5126]

@sergeylanzman awesome, thank you for confirming! I am going to do a bit more testing to confirm that the sysctls have no adverse impact, then I can merge this in tomorrow as everything looks good to me.

[jdn5126]

@sergeylanzman actually, sorry, one more thing: can we add a comment by this change that says:

// When ENABLE_V6_EGRESS is set, but the node is lacking an IPv6 address, log a warning and disable the egress-v6-cni plugin. This allows IPv4-only nodes to function while still alerting the customer to the possibility of a misconfiguration.

[sergeylanzman]

Added. I think you mean this place?

[jdn5126]

Added. I think you mean this place?

Yep, thank you! Just approved. Once merged, this will ship in the next VPC CNI release, which is currently targeting mid-February.