Kmesh request routing example not working as expected

[cr7258]

What happened:

I installed kmesh following this guide: https://kmesh.net/en/docs/setup/quickstart/. When I tried request routing example, it didn't work. Ideally, all requests with end-user:jason header should be sent to Server 2, but they were sent to both Server 1 and Server 2.

kubectl describe pod netutils-685bdc88b8-l2ffs |grep Annotations
Annotations:      kmesh.net/redirection: enabled

kubectl get virtualservices.networking.istio.io fortio -o yaml
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"fortio","namespace":"default"},"spec":{"hosts":["fortio"],"http":[{"match":[{"headers":{"end-user":{"exact":"jason"}}}],"route":[{"destination":{"host":"fortio","subset":"v2"}}]},{"route":[{"destination":{"host":"fortio","subset":"v1"}}]}]}}
  creationTimestamp: "2025-01-31T13:37:20Z"
  generation: 2
  name: fortio
  namespace: default
  resourceVersion: "2173"
  uid: 321a2cd3-566a-48ec-aed7-2df29cbfc268
spec:
  hosts:
  - fortio
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: fortio
        subset: v2
  - route:
    - destination:
        host: fortio
        subset: v1

# All requests with `end-user:jason` header should be sent to Server 2
for i in {1..10}; do kubectl exec -it $(kubectl get pod | grep netutils | awk '{print $1}') -- curl \--header "end-user:jason" -v $(kubectl get svc -owide | grep fortio | awk '{print $3}'):80 | grep "Server:"; done
< Server: 1
< Server: 2
< Server: 2
< Server: 1
< Server: 2
< Server: 2
< Server: 1
< Server: 1
< Server: 1
< Server: 1

What you expected to happen:

All requests with end-user:jason header should be sent to Server 2.

How to reproduce it (as minimally and precisely as possible):

# deploy istio
istioctl install

# install kmesh
helm install kmesh ./deploy/charts/kmesh-helm -n kmesh-system --create-namespace

# Enable Kmesh for the specified namespace
kubectl label namespace default istio.io/dataplane-mode=Kmesh

Anything else we need to know?:

Kmesh daemonset logs:

root@seven-dev-vm:~/kmesh# kubectl logs -n kmesh-system kmesh-2v2mm 
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --bpf-fs-path=\"/sys/fs/bpf\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --cgroup2-path=\"/mnt/kmesh_cgroup2\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --cni-etc-path=\"/etc/cni/net.d\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --conflist-name=\"\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --enable-bypass=\"false\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --enable-ipsec=\"false\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --enable-mda=\"false\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --enable-secret-manager=\"false\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --help=\"false\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --mode=\"dual-engine\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --monitoring=\"true\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --plugin-cni-chained=\"true\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="FLAG: --profiling=\"false\"" subsys=manager
time="2025-01-31T13:36:36Z" level=info msg="kmesh start with Normal" subsys=bpf
time="2025-01-31T13:36:39Z" level=info msg="bpf loader start successfully" subsys=manager
time="2025-01-31T13:36:39Z" level=info msg="start kmesh manage controller successfully" subsys=controller
time="2025-01-31T13:36:39Z" level=info msg="proxy ztunnel~10.244.0.5~kmesh-2v2mm.kmesh-system~kmesh-system.svc.cluster.local connect to discovery address istiod.istio-system.svc:15012" subsys=controller/config
time="2025-01-31T13:36:39Z" level=info msg="controller start successfully" subsys=manager
time="2025-01-31T13:36:39Z" level=info msg="start write CNI config" subsys="cni installer"
time="2025-01-31T13:36:39Z" level=info msg="kmesh cni use chained\n" subsys="cni installer"
time="2025-01-31T13:36:39Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:39Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:39Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:39Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:39Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:39Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:39Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:39Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:39Z" level=info msg="Copied /usr/bin/kmesh-cni to /opt/cni/bin." subsys="cni installer"
time="2025-01-31T13:36:39Z" level=info msg="wrote kubeconfig file /etc/cni/net.d/kmesh-cni-kubeconfig" subsys="cni installer"
time="2025-01-31T13:36:39Z" level=info msg="cni config file: /etc/cni/net.d/10-kindnet.conflist" subsys="cni installer"
time="2025-01-31T13:36:39Z" level=info msg="start cni successfully" subsys=manager
time="2025-01-31T13:36:39Z" level=info msg="start watching file /var/run/secrets/kubernetes.io/serviceaccount/token" subsys="cni installer"
time="2025-01-31T13:36:39Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:39Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:39Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:40Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:40Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:40Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:40Z" level=info msg="grpc reconnect succeed" subsys=controller
time="2025-01-31T13:36:40Z" level=info msg="grpc reconnect succeed" subsys=controller

Environment:

• Kmesh version: ghcr.io/kmesh-net/kmesh:latest
• Kmesh mode(kmesh has Kernel-Native Mode and Duel-Engine Mode): dual-engine
• Istio version: 1.24.2
• Kernel version: Debian 6.1.123-1
• Others:

[hzxuzhonghu]

Seems you are installing kmesh with kerner-native (ads) mode, in this mode the L7 feature rely on enhanced kernel. I guess this is the reason it not works

[cr7258]

If I install Kmesh in Duel-Engine mode, does the L7 feature don't require an enhanced kernel?
If so, how to install Kmesh in Duel-Engine mode?

[hzxuzhonghu]

Yes, dual-engine does not require enhanced kernel patch. The install guide https://kmesh.net/en/docs/setup/quickstart/

And before trying L7 you should install the waypoint beforehand, then try https://kmesh.net/en/docs/userguide/try_request_routing/