Skip to content

Commit

Permalink
fix 977 (#1540)
Browse files Browse the repository at this point in the history
  • Loading branch information
wind57 authored Dec 21, 2023
1 parent 30584ae commit bf5b1fd
Show file tree
Hide file tree
Showing 26 changed files with 1,187 additions and 41 deletions.
63 changes: 48 additions & 15 deletions docs/src/main/asciidoc/discovery-client.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -229,32 +229,65 @@ NOTE: `spring.application.name` has no effect as far as the name registered for

'''

Spring Cloud Kubernetes can also watch the Kubernetes service catalog for changes and update the
`DiscoveryClient` implementation accordingly. By "watch" we mean that we will publish a heartbeat event every `spring.cloud.kubernetes.discovery.catalog-services-watch-delay`
milliseconds (by default it is `30000`). The heartbeat event will contain the target references (and their namespaces of the addresses of all endpoints
Spring Cloud Kubernetes can also watch the Kubernetes service catalog for changes and update the `DiscoveryClient` implementation accordingly. In order to enable this functionality you need to add
`@EnableScheduling` on a configuration class in your application. By "watch", we mean that we will publish a heartbeat event every `spring.cloud.kubernetes.discovery.catalog-services-watch-delay`
milliseconds (by default it is `30000`). For the http discovery server this must be an environment variable set in deployment yaml:

----
containers:
- name: discovery-server
image: springcloud/spring-cloud-kubernetes-discoveryserver:3.0.5-SNAPSHOT
env:
- name: SPRING_CLOUD_KUBERNETES_DISCOVERY_CATALOGSERVICESWATCHDELAY
value: 3000
----

The heartbeat event will contain the target references (and their namespaces of the addresses of all endpoints
(for the exact details of what will get returned you can take a look inside `KubernetesCatalogWatch`). This is an implementation detail, and listeners of the heartbeat event
should not rely on the details. Instead, they should see if there are differences between two subsequent heartbeats via `equals` method. We will take care to return a correct implementation that adheres to the equals contract.
The endpoints will be queried in either :

- all namespaces (enabled via `spring.cloud.kubernetes.discovery.all-namespaces=true`)
- `all-namespaces` (enabled via `spring.cloud.kubernetes.discovery.all-namespaces=true`)

- `selective namespaces` (enabled via `spring.cloud.kubernetes.discovery.namespaces`), for example:

- specific namespaces (enabled via `spring.cloud.kubernetes.discovery.namespaces`), for example:
- `one namespace` via xref:property-source-config.adoc#namespace-resolution[Namespace Resolution] if the above two paths are not taken.

NOTE: If, for any reasons, you want to disable catalog watcher, you need to set `spring.cloud.kubernetes.discovery.catalog-services-watch.enabled=false`. For the http discovery server, this needs to be an environment variable set in deployment for example:

[source]
----
spring:
cloud:
kubernetes:
discovery:
namespaces:
- namespace-a
- namespace-b
SPRING_CLOUD_KUBERNETES_DISCOVERY_CATALOGSERVICESWATCH_ENABLED=FALSE
----

The functionality of catalog watch works for all 3 discovery clients that we support, with some caveats that you need to be aware of in case of the http client.

- The first is that this functionality is disabled by default, and it needs to be enabled in two places:

* in discovery server via an environment variable in the deployment manifest, for example:
+
----
containers:
- name: discovery-server
image: springcloud/spring-cloud-kubernetes-discoveryserver:3.0.5-SNAPSHOT
env:
- name: SPRING_CLOUD_KUBERNETES_HTTP_DISCOVERY_CATALOG_WATCHER_ENABLED
value: "TRUE"
----
+

* in discovery client, via a property in your `application.properties` for example:
+
----
spring.cloud.kubernetes.http.discovery.catalog.watcher.enabled=true
----
+

- The second point is that this is only supported since version `3.0.6` and upwards.
- Since http discovery has _two_ components : server and client, we strongly recommend to align versions between them, otherwise things might not work.
- If you decide to disable catalog watcher, you need to disable it in both server and client.

- we will use: xref:property-source-config.adoc#namespace-resolution[Namespace Resolution] if the above two paths are not taken.

In order to enable this functionality you need to add
`@EnableScheduling` on a configuration class in your application.

By default, we use the `Endpoints`(see https://kubernetes.io/docs/concepts/services-networking/service/#endpoints) API to find out the current state of services. There is another way though, via `EndpointSlices` (https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/). Such support can be enabled via a property: `spring.cloud.kubernetes.discovery.use-endpoint-slices=true` (by default it is `false`). Of course, your cluster has to support it also. As a matter of fact, if you enable this property, but your cluster does not support it, we will fail starting the application. If you decide to enable such support, you also need proper Role/ClusterRole set-up. For example:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import org.springframework.core.log.LogAccessor;
import org.springframework.scheduling.annotation.Scheduled;

import static org.springframework.cloud.kubernetes.commons.discovery.KubernetesDiscoveryConstants.CATALOG_WATCH_PROPERTY_WITH_DEFAULT_VALUE;
import static org.springframework.cloud.kubernetes.commons.discovery.KubernetesDiscoveryConstants.DISCOVERY_GROUP;
import static org.springframework.cloud.kubernetes.commons.discovery.KubernetesDiscoveryConstants.DISCOVERY_VERSION;
import static org.springframework.cloud.kubernetes.commons.discovery.KubernetesDiscoveryConstants.ENDPOINT_SLICE;
Expand Down Expand Up @@ -67,7 +68,7 @@ public void setApplicationEventPublisher(ApplicationEventPublisher publisher) {
this.publisher = publisher;
}

@Scheduled(fixedDelayString = "${spring.cloud.kubernetes.discovery.catalogServicesWatchDelay:30000}")
@Scheduled(fixedDelayString = "${" + CATALOG_WATCH_PROPERTY_WITH_DEFAULT_VALUE + "}")
void catalogServicesWatch() {
try {

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
/*
* Copyright 2019-2023 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.springframework.cloud.kubernetes.commons.discovery;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Inherited;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;

/**
* Provides a more succinct conditional for:
* <code>spring.cloud.kubernetes.http.discovery.client.catalog.watcher.enabled</code>.
*
* @author wind57
*/
@Target({ ElementType.TYPE, ElementType.METHOD })
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@ConditionalOnProperty(value = "spring.cloud.kubernetes.http.discovery.catalog.watcher.enabled", matchIfMissing = false)
public @interface ConditionalOnHttpDiscoveryCatalogWatcherEnabled {

}
Original file line number Diff line number Diff line change
Expand Up @@ -82,4 +82,20 @@ private KubernetesDiscoveryConstants() {
*/
public static final String SECURED = "secured";

/**
* catalog watch delay property name.
*/
public static final String CATALOG_WATCH_PROPERTY_NAME = "spring.cloud.kubernetes.discovery.catalogServicesWatchDelay";

/**
* default delay for the configuration watcher.
*/
public static final String CATALOG_WATCHER_DEFAULT_DELAY = "30000";

/**
* catalog watch delay property name with default value.
*/
public static final String CATALOG_WATCH_PROPERTY_WITH_DEFAULT_VALUE = CATALOG_WATCH_PROPERTY_NAME + ":"
+ CATALOG_WATCHER_DEFAULT_DELAY;

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
/*
* Copyright 2013-2023 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.springframework.cloud.kubernetes.discoveryserver;

import java.util.List;

import reactor.core.publisher.Mono;

import org.springframework.cloud.kubernetes.commons.discovery.ConditionalOnHttpDiscoveryCatalogWatcherEnabled;
import org.springframework.cloud.kubernetes.commons.discovery.ConditionalOnKubernetesCatalogEnabled;
import org.springframework.cloud.kubernetes.commons.discovery.EndpointNameAndNamespace;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

/**
* @author wind57
*/
@RestController
@ConditionalOnKubernetesCatalogEnabled
@ConditionalOnHttpDiscoveryCatalogWatcherEnabled
class DiscoveryCatalogWatcherController {

private final HeartBeatListener heartBeatListener;

DiscoveryCatalogWatcherController(HeartBeatListener heartBeatListener) {
this.heartBeatListener = heartBeatListener;
}

@GetMapping("/state")
Mono<List<EndpointNameAndNamespace>> state() {
return Mono.defer(() -> Mono.just(heartBeatListener.lastState().get()));
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,13 @@

import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.scheduling.annotation.EnableScheduling;

/**
* @author Ryan Baxter
*/
@SpringBootApplication
@EnableScheduling
public class DiscoveryServerApplication {

public static void main(String[] args) {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
/*
* Copyright 2013-2023 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.springframework.cloud.kubernetes.discoveryserver;

import java.util.List;
import java.util.concurrent.atomic.AtomicReference;

import org.apache.commons.logging.LogFactory;

import org.springframework.cloud.client.discovery.event.HeartbeatEvent;
import org.springframework.cloud.kubernetes.commons.discovery.ConditionalOnHttpDiscoveryCatalogWatcherEnabled;
import org.springframework.cloud.kubernetes.commons.discovery.ConditionalOnKubernetesCatalogEnabled;
import org.springframework.cloud.kubernetes.commons.discovery.EndpointNameAndNamespace;
import org.springframework.context.ApplicationListener;
import org.springframework.core.env.Environment;
import org.springframework.core.log.LogAccessor;
import org.springframework.stereotype.Component;

import static org.springframework.cloud.kubernetes.commons.discovery.KubernetesDiscoveryConstants.CATALOG_WATCHER_DEFAULT_DELAY;
import static org.springframework.cloud.kubernetes.commons.discovery.KubernetesDiscoveryConstants.CATALOG_WATCH_PROPERTY_NAME;

/**
* Listener for a HeartbeatEvent that comes from KubernetesCatalogWatch.
*
* @author wind57
*/
@Component
@ConditionalOnKubernetesCatalogEnabled
@ConditionalOnHttpDiscoveryCatalogWatcherEnabled
class HeartBeatListener implements ApplicationListener<HeartbeatEvent> {

private static final LogAccessor LOG = new LogAccessor(LogFactory.getLog(HeartBeatListener.class));

private final AtomicReference<List<EndpointNameAndNamespace>> lastState = new AtomicReference<>(List.of());

HeartBeatListener(Environment environment) {
String watchDelay = environment.getProperty(CATALOG_WATCH_PROPERTY_NAME);
if (watchDelay != null) {
LOG.debug("using delay : " + watchDelay);
}
else {
LOG.debug("using default watch delay : " + CATALOG_WATCHER_DEFAULT_DELAY);
}
}

@Override
@SuppressWarnings("unchecked")
public void onApplicationEvent(HeartbeatEvent event) {
LOG.debug(() -> "received heartbeat event");
List<EndpointNameAndNamespace> state = (List<EndpointNameAndNamespace>) event.getValue();
LOG.debug(() -> "state received : " + state);
lastState.set(state);
}

AtomicReference<List<EndpointNameAndNamespace>> lastState() {
return lastState;
}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
/*
* Copyright 2013-2023 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.springframework.cloud.kubernetes.discoveryserver;

import java.util.List;
import java.util.concurrent.atomic.AtomicReference;

import org.junit.jupiter.api.Test;
import org.mockito.Mockito;
import reactor.test.StepVerifier;

import org.springframework.cloud.kubernetes.commons.discovery.EndpointNameAndNamespace;

/**
* @author wind57
*/
class DiscoveryCatalogWatcherControllerTests {

private final HeartBeatListener heartBeatListener = Mockito.mock(HeartBeatListener.class);

@Test
void test() {
Mockito.when(heartBeatListener.lastState())
.thenReturn(new AtomicReference<>(List.of(new EndpointNameAndNamespace("one", "two"))));

DiscoveryCatalogWatcherController catalogWatcherController = new DiscoveryCatalogWatcherController(
heartBeatListener);

StepVerifier.create(catalogWatcherController.state())
.expectNext(List.of(new EndpointNameAndNamespace("one", "two"))).verifyComplete();
}

}
Loading

0 comments on commit bf5b1fd

Please sign in to comment.