Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dubbo-xds not support WeightedCluster #271

Open
AYue-94 opened this issue May 31, 2023 · 7 comments
Open

dubbo-xds not support WeightedCluster #271

AYue-94 opened this issue May 31, 2023 · 7 comments

Comments

@AYue-94
Copy link

AYue-94 commented May 31, 2023 •
edited
Loading

Environment

  • Dubbo version: 3.2.0
  • Operating System version: xxx
  • Java version: 1.8

Steps to reproduce this issue

1）use dubbo-samples/3-extensions/registry/dubbo-samples-xds
2）kubectl label pods -n dubbo-demo {podName1} version=v1 && kubectl label pods -n dubbo-demo {podName2} version=v2
3）kubectl apply -f VirtualService.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: dubbo-samples-xds-provider
  namespace: dubbo-demo
spec:
  hosts:
    - dubbo-samples-xds-provider.dubbo-demo.svc.cluster.local
  http:
    - match:
        - port: 50051
      route:
        - destination:
            host: dubbo-samples-xds-provider.dubbo-demo.svc.cluster.local
            subset: v1
            port:
              number: 50051
          weight: 50
        - destination:
            host: dubbo-samples-xds-provider.dubbo-demo.svc.cluster.local
            subset: v2
            port:
              number: 50051
          weight: 50

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: dubbo-samples-xds-provider
  namespace: dubbo-demo
spec:
  host: dubbo-samples-xds-provider.dubbo-demo.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

Expected Behavior

consumer can find 2 invokers.

Actual Behavior

no invokers found.

May 31, 2023 5:47:26 AM org.apache.dubbo.rpc.cluster.directory.AbstractDirectory warn
WARNING:  [DUBBO] No provider available after connectivity filter for the service org.apache.dubbo.samples.api.GreetingService:1.0.0 All validInvokers' size: 0 All routed invokers' size: 0 All invokers' size: 0 from registry istiod.istio-system.svc:15012 on the consumer 10.1.3.76 using the dubbo version 3.2.0., dubbo version: 3.2.0, current host: 10.1.3.76, error code: 2-2. This may be caused by provider server or registry center crashed, go to https://dubbo.apache.org/faq/2/2 to find instructions.
org.apache.dubbo.rpc.RpcException: Failed to invoke the method sayHello in the service org.apache.dubbo.samples.api.GreetingService. No provider available for the service org.apache.dubbo.samples.api.GreetingService:1.0.0 from registry istiod.istio-system.svc:15012 on the consumer 10.1.3.76 using the dubbo version 3.2.0. Please check if the providers have been started and registered.
	at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.checkInvokers(AbstractClusterInvoker.java:367)
	at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:60)
	at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:341)
	at org.apache.dubbo.rpc.cluster.router.RouterSnapshotFilter.invoke(RouterSnapshotFilter.java:46)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
	at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:101)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
	at org.apache.dubbo.rpc.cluster.filter.support.MetricsClusterFilter.invoke(MetricsClusterFilter.java:51)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
	at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:52)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
	at org.apache.dubbo.rpc.cluster.filter.support.ObservationSenderFilter.invoke(ObservationSenderFilter.java:61)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
	at org.apache.dubbo.spring.security.filter.ContextHolderParametersSelectedTransferFilter.invoke(ContextHolderParametersSelectedTransferFilter.java:41)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
	at org.apache.dubbo.rpc.cluster.filter.support.ConsumerClassLoaderFilter.invoke(ConsumerClassLoaderFilter.java:40)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
	at org.apache.dubbo.rpc.cluster.filter.support.ConsumerContextFilter.invoke(ConsumerContextFilter.java:118)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:194)
	at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$ClusterFilterInvoker.invoke(AbstractCluster.java:91)
	at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:103)
	at org.apache.dubbo.rpc.cluster.support.wrapper.ScopeClusterInvoker.invoke(ScopeClusterInvoker.java:131)
	at org.apache.dubbo.registry.client.migration.MigrationInvoker.invoke(MigrationInvoker.java:284)
	at org.apache.dubbo.rpc.proxy.InvocationUtil.invoke(InvocationUtil.java:57)
	at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:75)
	at org.apache.dubbo.samples.api.GreetingServiceDubboProxy0.sayHello(GreetingServiceDubboProxy0.java)
	at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
	at com.sun.proxy.$Proxy27.sayHello(Unknown Source)
	at org.apache.dubbo.samples.action.GreetingServiceConsumer.doSayHello(GreetingServiceConsumer.java:34)
	at org.apache.dubbo.samples.ConsumerBootstrap.main(ConsumerBootstrap.java:38)
May 31, 2023 5:47:26 AM org.apache.dubbo.rpc.cluster.SingleRouterChain warn
WARNING:  [DUBBO] No provider available after route for the service org.apache.dubbo.samples.api.GreetingService:1.0.0 from registry 10.1.3.76 on the consumer 10.1.3.76 using the dubbo version 3.2.0. Router snapshot is below:
[ Parent (Input: 0) (Current Node Output: 0) (Chain Node Output: 0) ] Input: Empty -> Chain Node Output: Empty
  [ MockInvokersSelector (Input: 0) (Current Node Output: 0) (Chain Node Output: 0) Router message: Empty invokers. Directly return. ] Current Node Output: Empty, dubbo version: 3.2.0, current host: 10.1.3.76, error code: 2-2. This may be caused by No provider available after route for the service, go to https://dubbo.apache.org/faq/2/2 to find instructions.
May 31, 2023 5:47:26 AM org.apache.dubbo.rpc.cluster.directory.AbstractDirectory warn
WARNING:  [DUBBO] No provider available after connectivity filter for the service org.apache.dubbo.samples.api.GreetingService:1.0.0 All validInvokers' size: 0 All routed invokers' size: 0 All invokers' size: 0 from registry istiod.istio-system.svc:15012 on the consumer 10.1.3.76 using the dubbo version 3.2.0., dubbo version: 3.2.0, current host: 10.1.3.76, error code: 2-2. This may be caused by provider server or registry center crashed, go to https://dubbo.apache.org/faq/2/2 to find instructions.
@AYue-94
Copy link
Author

AYue-94 commented May 31, 2023 •
edited
Loading

我大概看了一下，主要有两个地方可能有问题。
1）org.apache.dubbo.registry.xds.util.protocol.impl.RdsProtocol
解析的时候没有解析weightcluster。
image
但是却提供了xdsrouter相关功能，但是实际上底层invokers并不会存在。
2）org.apache.dubbo.rpc.cluster.router.xds.XdsRouter#computeSubset
这里在路由的时候，构造bitlist应该有问题，是不是应该这样做？
image

除了上面两个问题以外，我想问一下proxyless模式现在是不是非生产可用？

@AlbumenJ
Copy link
Member

AlbumenJ commented Jun 2, 2023

@aamingaa PTAL

@aamingaa
Copy link

aamingaa commented Jun 2, 2023

怎么直接apply virtual.yml了，前面还有好几步呢，参考下下面的命令
mvn clean package -DskipTests -U
cd ./dubbo-samples-xds-provider/
docker build -t aamingaa/dubbo-demo:dubbo-samples-xds-provider_0.0.1 .
cd ../dubbo-samples-xds-consumer/
docker build -t aamingaa/dubbo-demo:dubbo-samples-xds-consumer_0.0.1 .
cd ../
kubectl apply -f ./deploy/Namespace.yml
kubens dubbo-demo
cd ./dubbo-samples-xds-provider/src/main/resources/k8s
kubectl apply -f Deployment.yml
kubectl apply -f Deployment-2.yml
kubectl apply -f Service.yml
kubectl apply -f Virtual-Service.yml
cd ../../../../../dubbo-samples-xds-consumer/src/main/resources/k8s
kubectl apply -f Deployment.yml
kubectl apply -f Service.yml

@AYue-94
Copy link
Author

AYue-94 commented Jun 5, 2023 •
edited
Loading

怎么直接apply virtual.yml了，前面还有好几步呢，参考下下面的命令 mvn clean package -DskipTests -U cd ./dubbo-samples-xds-provider/ docker build -t aamingaa/dubbo-demo:dubbo-samples-xds-provider_0.0.1 . cd ../dubbo-samples-xds-consumer/ docker build -t aamingaa/dubbo-demo:dubbo-samples-xds-consumer_0.0.1 . cd ../ kubectl apply -f ./deploy/Namespace.yml kubens dubbo-demo cd ./dubbo-samples-xds-provider/src/main/resources/k8s kubectl apply -f Deployment.yml kubectl apply -f Deployment-2.yml kubectl apply -f Service.yml kubectl apply -f Virtual-Service.yml cd ../../../../../dubbo-samples-xds-consumer/src/main/resources/k8s kubectl apply -f Deployment.yml kubectl apply -f Service.yml

1）我省略了前面Deployment和Service的部署，这几步肯定要做的。
2）dubbo-samples-xds并没有提供virtualService.yaml，所以我把sample项目部署、打标、VirtualService资源的定义分开描述了。
3）sidecar模式确实没问题，问题是proxyless模式

@aamingaa
Copy link

aamingaa commented Jun 5, 2023

这有一份proxyless下模式的virtualService.yml的配置文件，你对照着试一下？apache/dubbo-samples@master...aamingaa:dubbo-samples:test/xds_debug_3.2.0-beta.7-SNAPSHOT

@AYue-94
Copy link
Author

AYue-94 commented Jun 6, 2023 •
edited
Loading

这有一份proxyless下模式的virtualService.yml的配置文件，你对照着试一下？apache/[email protected]:dubbo-samples:test/xds_debug_3.2.0-beta.7-SNAPSHOT

@aamingaa

这个我也试了，不行。因为sidecar模式和proxyless模式，除了application.properties中几个配置项不一样以外。其他部署上是没有区别的（除了namespace不打标注入istioproxy以外），所以sidecar能通，proxyless我这边也不会部署有什么问题。

如果我没理解错的话，运行时通过PilotExchanger.getInstance().getEndpoints("dubbo-samples-xds-provider")应该能获取到provider的所有Endpoint，包括v1的和v2的，但是现在也获取不到。

@songxiaosheng
Copy link
Member

songxiaosheng commented Sep 23, 2023 •
edited
Loading

这周也遇到了同样的异常ProxyLess模式，

  • 个人mac电脑安装的k8s环境参考如下教程安装的xds通信一切正常，
  • 公司环境遇到了相同的异常(完全使用案例中的Demo)，消费者无法发现提供者，
    - 排查发现istio的LDS和RDS日志都有，并且kiali控制台均能看到消费者和提供者应用信息，但是使用3.1.0 3.1.8 3.2.6版本时消费者均无法发现提供者服务 ，提供者Service也有配置，是否有思路排查来观测消费者查询到的提供者服务
image

@CrazyHZM CrazyHZM transferred this issue from apache/dubbo Dec 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@AlbumenJ @AYue-94 @aamingaa @songxiaosheng