Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

istio ingress gw doesn't reach running on remote cluster #23859

Closed
linsun opened this issue May 14, 2020 · 14 comments
Closed

istio ingress gw doesn't reach running on remote cluster #23859

linsun opened this issue May 14, 2020 · 14 comments

Comments

@linsun
Copy link
Member

linsun commented May 14, 2020

Bug description

$ k get pods -n istio-system                                
NAME                                   READY   STATUS    RESTARTS   AGE
istio-ingressgateway-ffcc5c67b-6md9l   0/1     Running   0          13m
istiod-7897db8c8f-r64qf                1/1     Running   0          87s
prometheus-65f7544b8f-sh4qm            2/2     Running   0          13m
(⎈ |linistio14/boft3r6w0p5blanmm18g:default)
~/go/src/istio.io/testingdir ⌚ 11:56:14
$ k exec -it istio-ingressgateway-ffcc5c67b-6md9l -n istio-system sh                                    
# curl localhost:15021
curl: (7) Failed to connect to localhost port 15021: Connection refused
# curl localhost:15020
404 page not found

The log look fine, I think it is just an status for the port failed.

Expected behavior
ingress gw reaches running

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
1.6.0 rc.0
How was Istio installed?
Followed the example here on the primary cluster: istio/istio.io#7243, step 1.

Environment where bug was observed (cloud vendor, OS, etc)
IKS 1.16

@linsun
Copy link
Member Author

linsun commented May 14, 2020

$ k logs istio-ingressgateway-ffcc5c67b-6md9l -n istio-system                
2020-05-14T15:42:33.708919Z	info	FLAG: --concurrency="0"
2020-05-14T15:42:33.708958Z	info	FLAG: --disableInternalTelemetry="false"
2020-05-14T15:42:33.708967Z	info	FLAG: --domain="istio-system.svc.cluster.local"
2020-05-14T15:42:33.708972Z	info	FLAG: --help="false"
2020-05-14T15:42:33.708976Z	info	FLAG: --id=""
2020-05-14T15:42:33.708981Z	info	FLAG: --ip=""
2020-05-14T15:42:33.708992Z	info	FLAG: --log_as_json="false"
2020-05-14T15:42:33.708996Z	info	FLAG: --log_caller=""
2020-05-14T15:42:33.709001Z	info	FLAG: --log_output_level="default:info"
2020-05-14T15:42:33.709005Z	info	FLAG: --log_rotate=""
2020-05-14T15:42:33.709010Z	info	FLAG: --log_rotate_max_age="30"
2020-05-14T15:42:33.709014Z	info	FLAG: --log_rotate_max_backups="1000"
2020-05-14T15:42:33.709019Z	info	FLAG: --log_rotate_max_size="104857600"
2020-05-14T15:42:33.709023Z	info	FLAG: --log_stacktrace_level="default:none"
2020-05-14T15:42:33.709035Z	info	FLAG: --log_target="[stdout]"
2020-05-14T15:42:33.709042Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2020-05-14T15:42:33.709047Z	info	FLAG: --mixerIdentity=""
2020-05-14T15:42:33.709067Z	info	FLAG: --outlierLogPath=""
2020-05-14T15:42:33.709072Z	info	FLAG: --pilotIdentity=""
2020-05-14T15:42:33.709076Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2020-05-14T15:42:33.709080Z	info	FLAG: --proxyLogLevel="warning"
2020-05-14T15:42:33.709085Z	info	FLAG: --serviceCluster="istio-ingressgateway"
2020-05-14T15:42:33.709089Z	info	FLAG: --serviceregistry="Kubernetes"
2020-05-14T15:42:33.709094Z	info	FLAG: --stsPort="0"
2020-05-14T15:42:33.709098Z	info	FLAG: --templateFile=""
2020-05-14T15:42:33.709103Z	info	FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2020-05-14T15:42:33.709107Z	info	FLAG: --trust-domain="cluster.local"
2020-05-14T15:42:33.709208Z	info	Version 2ef332f29a03351b512df036c412dc09429ead17-dirty-2ef332f29a03351b512df036c412dc09429ead17-dirty-Clean
2020-05-14T15:42:33.709412Z	info	Obtained private IP [172.30.24.170 fe80::c04:b1ff:fe4b:428c]
2020-05-14T15:42:33.709515Z	info	Apply mesh config from file accessLogEncoding: TEXT
accessLogFile: ""
accessLogFormat: ""
defaultConfig:
  concurrency: 2
  configPath: ./etc/istio/proxy
  connectTimeout: 10s
  controlPlaneAuthPolicy: NONE
  discoveryAddress: istiod.istio-system.svc:15012
  drainDuration: 45s
  parentShutdownDuration: 1m0s
  proxyAdminPort: 15000
  proxyMetadata:
    DNS_AGENT: ""
  serviceCluster: istio-proxy
  tracing:
    zipkin:
      address: zipkin.istio-system:9411
disableMixerHttpReports: true
disablePolicyChecks: true
enablePrometheusMerge: false
ingressClass: istio
ingressControllerMode: STRICT
ingressService: istio-ingressgateway
protocolDetectionTimeout: 100ms
reportBatchMaxEntries: 100
reportBatchMaxTime: 1s
sdsUdsPath: unix:/etc/istio/proxy/SDS
trustDomain: cluster.local
trustDomainAliases: null
2020-05-14T15:42:33.711222Z	info	Effective config: binaryPath: /usr/local/bin/envoy
configPath: ./etc/istio/proxy
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
  DNS_AGENT: ""
serviceCluster: istio-ingressgateway
statNameLength: 189
statusPort: 15020
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2020-05-14T15:42:33.711268Z	info	Proxy role: &model.Proxy{ClusterID:"", Type:"router", IPAddresses:[]string{"172.30.24.170", "fe80::c04:b1ff:fe4b:428c"}, ID:"istio-ingressgateway-ffcc5c67b-6md9l.istio-system", Locality:(*envoy_api_v2_core.Locality)(nil), DNSDomain:"istio-system.svc.cluster.local", ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), PrevSidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), IstioVersion:(*model.IstioVersion)(nil), ipv6Support:false, ipv4Support:false, GlobalUnicastIP:"", XdsResourceGenerator:model.XdsResourceGenerator(nil), Active:map[string]*model.WatchedResource(nil)}
2020-05-14T15:42:33.711280Z	info	JWT policy is third-party-jwt
2020-05-14T15:42:33.711314Z	info	PilotSAN []string{"istiod.istio-system.svc"}
2020-05-14T15:42:33.711327Z	info	MixerSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account"}
2020-05-14T15:42:33.711334Z	info	serverOptions.CAEndpoint == istiod.istio-system.svc:15012
2020-05-14T15:42:33.711339Z	info	Using user-configured CA istiod.istio-system.svc:15012
2020-05-14T15:42:33.711350Z	info	istiod uses self-issued certificate
2020-05-14T15:42:33.711387Z	info	the CA cert of istiod is: -----BEGIN CERTIFICATE-----
MIID7TCCAtWgAwIBAgIJAOIRDhOcxsx6MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD
VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTESMBAGA1UEBwwJU3Vubnl2YWxl
MQ4wDAYDVQQKDAVJc3RpbzENMAsGA1UECwwEVGVzdDEQMA4GA1UEAwwHUm9vdCBD
QTEiMCAGCSqGSIb3DQEJARYTdGVzdHJvb3RjYUBpc3Rpby5pbzAgFw0xODAxMjQx
OTE1NTFaGA8yMTE3MTIzMTE5MTU1MVowgYsxCzAJBgNVBAYTAlVTMRMwEQYDVQQI
DApDYWxpZm9ybmlhMRIwEAYDVQQHDAlTdW5ueXZhbGUxDjAMBgNVBAoMBUlzdGlv
MQ0wCwYDVQQLDARUZXN0MRAwDgYDVQQDDAdSb290IENBMSIwIAYJKoZIhvcNAQkB
FhN0ZXN0cm9vdGNhQGlzdGlvLmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEA38uEfAatzQYqbaLou1nxJ348VyNzumYMmDDt5pbLYRrCo2pS3ki1ZVDN
8yxIENJFkpKw9UctTGdbNGuGCiSDP7uqF6BiVn+XKAU/3pnPFBbTd0S33NqbDEQu
IYraHSl/tSk5rARbC1DrQRdZ6nYD2KrapC4g0XbjY6Pu5l4y7KnFwSunnp9uqpZw
uERv/BgumJ5QlSeSeCmhnDhLxooG8w5tC2yVr1yDpsOHGimP/mc8Cds4V0zfIhQv
YzfIHphhE9DKjmnjBYLOdj4aycv44jHnOGc+wvA1Jqsl60t3wgms+zJTiWwABLdw
zgMAa7yxLyoV0+PiVQud6k+8ZoIFcwIDAQABo1AwTjAdBgNVHQ4EFgQUOUYGtUyh
euxO4lGe4Op1y8NVoagwHwYDVR0jBBgwFoAUOUYGtUyheuxO4lGe4Op1y8NVoagw
DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEANXLyfAs7J9rmBamGJvPZ
ltx390WxzzLFQsBRAaH6rgeipBq3dR9qEjAwb6BTF+ROmtQzX+fjstCRrJxCto9W
tC8KvXTdRfIjfCCZjhtIOBKqRxE4KJV/RBfv9xD5lyjtCPCQl3Ia6MSf42N+abAK
WCdU6KCojA8WB9YhSCzza3aQbPTzd26OC/JblJpVgtus5f8ILzCsz+pbMimgTkhy
AuhYRppJaQ24APijsEC9+GIaVKPg5IwWroiPoj+QXNpshuvqVQQXvGaRiq4zoSnx
xAJz+w8tjrDWcf826VN14IL+/Cmqlg/rIfB5CHdwVIfWwpuGB66q/UiPegZMNs8a
3g==
-----END CERTIFICATE-----

2020-05-14T15:42:33.711573Z	info	parsed scheme: ""
2020-05-14T15:42:33.711592Z	info	scheme "" not registered, fallback to default scheme
2020-05-14T15:42:33.711613Z	info	ccResolverWrapper: sending update to cc: {[{istiod.istio-system.svc:15012  <nil> 0 <nil>}] <nil> <nil>}
2020-05-14T15:42:33.711626Z	info	ClientConn switching balancer to "pick_first"
2020-05-14T15:42:33.711631Z	info	Channel switches to new LB policy "pick_first"
2020-05-14T15:42:33.711670Z	info	Subchannel Connectivity change to CONNECTING
2020-05-14T15:42:33.711758Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc00066fcd0, {CONNECTING <nil>}
2020-05-14T15:42:33.712164Z	info	Channel Connectivity change to CONNECTING
2020-05-14T15:42:33.712176Z	info	Subchannel picks a new address "istiod.istio-system.svc:15012" to connect
2020-05-14T15:42:33.735427Z	info	Subchannel Connectivity change to READY
2020-05-14T15:42:33.735462Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc00066fcd0, {READY <nil>}
2020-05-14T15:42:33.735470Z	info	Channel Connectivity change to READY
2020-05-14T15:42:33.763286Z	info	Starting gateway SDS
2020-05-14T15:42:33.800844Z	warn	secretfetcher	failed load server cert/key pair from secret cacerts: server cert or private key is empty
2020-05-14T15:42:33.865588Z	info	sds	SDS gRPC server for workload UDS starts, listening on "./etc/istio/proxy/SDS" 

2020-05-14T15:42:33.865702Z	info	sds	SDS gRPC server for ingress gateway controller starts, listening on "/var/run/ingress_gateway/sds" 

2020-05-14T15:42:33.865720Z	info	sds	Start SDS grpc server
2020-05-14T15:42:33.865772Z	info	Starting proxy agent
2020-05-14T15:42:33.865788Z	info	sds	Start SDS grpc server for ingress gateway proxy
2020-05-14T15:42:33.865822Z	info	Received new config, creating new Envoy epoch 0
2020-05-14T15:42:33.865861Z	info	Opening status port 15020

2020-05-14T15:42:33.865995Z	info	Epoch 0 starting
2020-05-14T15:42:33.875658Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~172.30.24.170~istio-ingressgateway-ffcc5c67b-6md9l.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --log-format %Y-%m-%dT%T.%fZ	%l	envoy %n	%v -l warning --component-log-level misc:error]
2020-05-14T15:42:33.951506Z	warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-05-14T15:42:33.951586Z	warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-05-14T15:42:33.961835Z	info	sds	resource:default new connection
generateSecret called.
2020-05-14T15:42:34.217763Z	warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-05-14T15:42:34.217809Z	warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-05-14T15:42:34.242309Z	info	cache	Root cert has changed, start rotating root cert for SDS clients
2020-05-14T15:42:34.242370Z	info	cache	GenerateSecret default
2020-05-14T15:42:34.242533Z	info	sds	resource:default pushed key/cert pair to proxy
2020-05-14T15:42:34.873861Z	warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-05-14T15:42:34.873909Z	warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-05-14T15:42:49.091553Z	info	sds	resource:ROOTCA new connection
2020-05-14T15:42:49.091921Z	info	cache	Loaded root cert from certificate ROOTCA
2020-05-14T15:42:49.092216Z	info	sds	resource:ROOTCA pushed root cert to proxy
2020-05-14T15:54:54.136228Z	warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 13, 
2020-05-14T15:54:54.138516Z	info	Subchannel Connectivity change to CONNECTING
2020-05-14T15:54:54.138651Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc00066fcd0, {CONNECTING <nil>}
2020-05-14T15:54:54.138660Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-05-14T15:54:54.138683Z	info	Channel Connectivity change to CONNECTING
2020-05-14T15:54:54.138704Z	info	Subchannel picks a new address "istiod.istio-system.svc:15012" to connect
2020-05-14T15:54:54.147555Z	info	grpc: addrConn.createTransport failed to connect to {istiod.istio-system.svc:15012  <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 172.21.174.85:15012: connect: connection refused". Reconnecting...
2020-05-14T15:54:54.147587Z	info	Subchannel Connectivity change to TRANSIENT_FAILURE
2020-05-14T15:54:54.147646Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc00066fcd0, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp 172.21.174.85:15012: connect: connection refused"}
2020-05-14T15:54:54.147663Z	info	Channel Connectivity change to TRANSIENT_FAILURE
2020-05-14T15:54:54.155393Z	warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-05-14T15:54:55.147804Z	info	Subchannel Connectivity change to CONNECTING
2020-05-14T15:54:55.147873Z	info	Subchannel picks a new address "istiod.istio-system.svc:15012" to connect
2020-05-14T15:54:55.148126Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc00066fcd0, {CONNECTING <nil>}
2020-05-14T15:54:55.148139Z	info	Channel Connectivity change to CONNECTING
2020-05-14T15:54:55.172306Z	info	Subchannel Connectivity change to READY
2020-05-14T15:54:55.172342Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc00066fcd0, {READY <nil>}
2020-05-14T15:54:55.172351Z	info	Channel Connectivity change to READY

@linsun
Copy link
Member Author

linsun commented May 14, 2020

what exactly is our status port, the log has

serviceCluster: istio-ingressgateway
statNameLength: 189
statusPort: 15020

@linsun linsun added this to the 1.6 milestone May 14, 2020
@howardjohn
Copy link
Member

I think you have the wrong version running

@linsun
Copy link
Member Author

linsun commented May 14, 2020

That was it, sorry forgot to remove hub and tag from my io. Thank you @howardjohn !!

@linsun linsun closed this as completed May 14, 2020
@linsun
Copy link
Member Author

linsun commented May 14, 2020

I am still seeing issues with my ingress gw on remote cluster:

$ k get pods -n istio-system
NAME                                   READY   STATUS    RESTARTS   AGE
istio-ingressgateway-dddd9c8b4-dqfvq   0/1     Running   0          5m49s
Events:
  Type     Reason     Age                  From                     Message
  ----     ------     ----                 ----                     -------
  Normal   Scheduled  <unknown>            default-scheduler        Successfully assigned istio-system/istio-ingressgateway-dddd9c8b4-dqfvq to 10.188.142.230
  Normal   Pulling    110s                 kubelet, 10.188.142.230  Pulling image "docker.io/istio/proxyv2:1.6.0-rc.0"
  Normal   Pulled     107s                 kubelet, 10.188.142.230  Successfully pulled image "docker.io/istio/proxyv2:1.6.0-rc.0"
  Normal   Created    107s                 kubelet, 10.188.142.230  Created container istio-proxy
  Normal   Started    106s                 kubelet, 10.188.142.230  Started container istio-proxy
  Warning  Unhealthy  65s (x21 over 105s)  kubelet, 10.188.142.230  Readiness probe failed: Get http://172.30.95.7:15021/healthz/ready: dial tcp 172.30.95.7:15021: connect: connection refused

(⎈ |linistio10/7caab3af9f514f028081a8180c107b69:default)
~/go/src/istio.io/istio on  lin-fix ⌚ 15:40:53
$ k exec -it istio-ingressgateway-dddd9c8b4-dqfvq -n istio-system sh                                    
# curl localhost:15021/
# curl 172.30.95.7:15021/healthz/ready

@linsun linsun reopened this May 14, 2020
@linsun
Copy link
Member Author

linsun commented May 14, 2020

Not sure if anyone else hit it... I think the ingress gw did come up it just take slightly longer (60+s) thus it was not marked as ready.

@linsun
Copy link
Member Author

linsun commented May 14, 2020

may need to tweak this slightly longer for ingress gw as it does take a while to come up - https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-startup-probes

@howardjohn
Copy link
Member

Lin the time it takes does not matter, it could take 1 hr and once it becomes ready it will be marked ready. Can you post the full logs when it's slow to get ready maybe?

@linsun
Copy link
Member Author

linsun commented May 14, 2020

I need to test this without centralIstiod but I think the ingress gw doesn't reach running was due to citadel agent on the gw can't get the certs via CSR. @irisdingbj I think you forgot to cherry pick https://github.com/istio/istio/pull/22989/files to release-1.6.

2020-05-14T21:28:37.834927Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 169.60.74.107:15012: connect: connection timed out"
2020-05-14T21:28:38.018905Z	warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-05-14T21:28:39.310788Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected

cc @brian-avery @GregHanson

@linsun linsun changed the title istio ingress gw doesn't reach running istio ingress gw doesn't reach running when CA is not on the cluster May 14, 2020
@irisdingbj
Copy link
Member

I need to test this without centralIstiod but I think the ingress gw doesn't reach running was due to citadel agent on the gw can't get the certs via CSR. @irisdingbj I think you forgot to cherry pick https://github.com/istio/istio/pull/22989/files to release-1.6.

2020-05-14T21:28:37.834927Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 169.60.74.107:15012: connect: connection timed out"
2020-05-14T21:28:38.018905Z	warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-05-14T21:28:39.310788Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected

cc @brian-avery @GregHanson

@linsun 22989 is in 1.6 . It is checked in before we cut 1.6 branch . So it does not need a cherry-pick PR

@linsun linsun changed the title istio ingress gw doesn't reach running when CA is not on the cluster istio ingress gw doesn't reach running May 15, 2020
@linsun linsun changed the title istio ingress gw doesn't reach running istio ingress gw doesn't reach running on remote cluster May 15, 2020
@linsun
Copy link
Member Author

linsun commented May 15, 2020

Found out this is caused by port 15012 is not opened up by default. By having the following in the yaml will solve it.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    gateways:
      istio-ingressgateway:
        meshExpansionPorts:
          - port: 15017
            targetPort: 15017
            name: tcp-webhook
          - port: 15012
            targetPort: 15012
            name: tcp-istiod

can update doc for this.

@linsun
Copy link
Member Author

linsun commented May 15, 2020

It appears some change from #23102 broke my sleep container on remote cluster from reaching running. @irisdingbj is going to follow up with @hzxuzhonghu on this.

$ k get pods
NAME                    READY   STATUS    RESTARTS   AGE
sleep-f8cbf5b76-ncxjg   1/2     Running   0          73s
2020-05-15T02:50:42.397385Z	warn	serverca	Authentication failed: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: the service account authentication returns an error: [invalid bearer token, square/go-jose: error in cryptographic primitive]. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. ```

@linsun
Copy link
Member Author

linsun commented May 15, 2020

#23879

@linsun
Copy link
Member Author

linsun commented May 20, 2020

This ingress gw issue is fixed by adding caAddress in the remote yaml config (see istio/istio.io#7243). With that, ingress gw can reach running.

The sidecar not reach running issue is tracked under #23879

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants