make k8s ingestion core (#17614)

* make k8s ingestion core

* add redirects

* missing licenses

* Add disclaimer for druid 28

* Fix redirect

.github/labeler.yml

@@ -88,7 +88,7 @@
 'Kubernetes':
   - changed-files:
     - any-glob-to-any-file:
-      - 'extensions-contrib/kubernetes-overlord-extensions/**'
+      - 'extensions-core/kubernetes-overlord-extensions/**'
 
 'GHA':
   - changed-files:

distribution/pom.xml

@@ -254,6 +254,8 @@
                                         <argument>-c</argument>
                                         <argument>org.apache.druid.extensions:druid-kubernetes-extensions</argument>
                                         <argument>-c</argument>
+                                        <argument>org.apache.druid.extensions:druid-kubernetes-overlord-extensions</argument>
+                                        <argument>-c</argument>
                                         <argument>org.apache.druid.extensions:druid-catalog</argument>
                                         <argument>${druid.distribution.pulldeps.opts}</argument>
                                     </arguments>
@@ -413,8 +415,6 @@
                                         <argument>-c</argument>
                                         <argument>org.apache.druid.extensions.contrib:kafka-emitter</argument>
                                         <argument>-c</argument>
-                                        <argument>org.apache.druid.extensions.contrib:druid-kubernetes-overlord-extensions</argument>
-                                        <argument>-c</argument>
                                         <argument>org.apache.druid.extensions.contrib:materialized-view-maintenance</argument>
                                         <argument>-c</argument>
                                         <argument>org.apache.druid.extensions.contrib:materialized-view-selection</argument>

docs/configuration/extensions.md

@@ -62,6 +62,7 @@ Core extensions are maintained by Druid committers.
 |simple-client-sslcontext|Simple SSLContext provider module to be used by Druid's internal HttpClient when talking to other Druid processes over HTTPS.|[link](../development/extensions-core/simple-client-sslcontext.md)|
 |druid-pac4j|OpenID Connect authentication for druid processes.|[link](../development/extensions-core/druid-pac4j.md)|
 |druid-kubernetes-extensions|Druid cluster deployment on Kubernetes without Zookeeper.|[link](../development/extensions-core/kubernetes.md)|
+|druid-kubernetes-overlord-extensions|Support for launching tasks in k8s without Middle Managers|[link](../development/extensions-core/k8s-jobs.md)|
 
 ## Community extensions
 
@@ -100,7 +101,6 @@ All of these community extensions can be downloaded using [pull-deps](../operati
 |druid-tdigestsketch|Support for approximate sketch aggregators based on [T-Digest](https://github.com/tdunning/t-digest)|[link](../development/extensions-contrib/tdigestsketch-quantiles.md)|
 |gce-extensions|GCE Extensions|[link](../development/extensions-contrib/gce-extensions.md)|
 |prometheus-emitter|Exposes [Druid metrics](../operations/metrics.md) for Prometheus server collection (<https://prometheus.io/>)|[link](../development/extensions-contrib/prometheus.md)|
-|druid-kubernetes-overlord-extensions|Support for launching tasks in k8s without Middle Managers|[link](../development/extensions-contrib/k8s-jobs.md)|
 |druid-spectator-histogram|Support for efficient approximate percentile queries|[link](../development/extensions-contrib/spectator-histogram.md)|
 |druid-rabbit-indexing-service|Support for creating and managing [RabbitMQ](https://www.rabbitmq.com/) indexing tasks|[link](../development/extensions-contrib/rabbit-stream-ingestion.md)|
 |druid-ranger-security|Support for access control through Apache Ranger.|[link](../development/extensions-contrib/druid-ranger-security.md)|

docs/design/architecture.md

@@ -107,7 +107,7 @@ forking separate JVM processes per-task, the Indexer runs tasks as individual th
 
 The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system and to better enable resource sharing across tasks, which can help streaming ingestion. The Indexer is currently designated [experimental](../development/experimental.md).
 
-Typically, you would deploy one of the following: MiddleManagers, [MiddleManager-less ingestion using Kubernetes](../development/extensions-contrib/k8s-jobs.md), or Indexers. You wouldn't deploy more than one of these options.
+Typically, you would deploy one of the following: MiddleManagers, [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md), or Indexers. You wouldn't deploy more than one of these options.
 
 ## Colocation of services
 

docs/design/indexer.md

@@ -24,7 +24,7 @@ sidebar_label: "Indexer"
   -->
 
 :::info
- The Indexer is an optional and experimental feature. If you're primarily performing batch ingestion, we recommend you use either the MiddleManager and Peon task execution system or [MiddleManager-less ingestion using Kubernetes](../development/extensions-contrib/k8s-jobs.md). If you're primarily doing streaming ingestion, you may want to try either [MiddleManager-less ingestion using Kubernetes](../development/extensions-contrib/k8s-jobs.md) or the Indexer service.
+ The Indexer is an optional and experimental feature. If you're primarily performing batch ingestion, we recommend you use either the MiddleManager and Peon task execution system or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md). If you're primarily doing streaming ingestion, you may want to try either [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md) or the Indexer service.
 :::
 
 The Apache Druid Indexer service is an alternative to the Middle Manager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process.

docs/development/extensions-contrib/k8s-jobs.md → docs/development/extensions-core/k8s-jobs.md

@@ -696,6 +696,10 @@ roleRef:
 ```
 
 ## Migration/Kubernetes and Worker Task Runner
+:::info
+This feature is only available starting in Druid 28. If you require a rolling update to enable Kubernetes-based ingestion, first update your cluster to Druid 28 then apply the overlord configurations mentioned in this section.
+:::
+
 If you are running a cluster with tasks running on middle managers or indexers and want to do a zero downtime migration to mm-less ingestion, the mm-less ingestion system is capable of running in migration mode by reading tasks from middle managers/indexers and Kubernetes and writing tasks to either middle managers or to Kubernetes.
 
 To do this, set the following property.

extensions-contrib/kubernetes-overlord-extensions/pom.xml → extensions-core/kubernetes-overlord-extensions/pom.xml

@@ -22,7 +22,7 @@
          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
   <modelVersion>4.0.0</modelVersion>
 
-  <groupId>org.apache.druid.extensions.contrib</groupId>
+  <groupId>org.apache.druid.extensions</groupId>
   <artifactId>druid-kubernetes-overlord-extensions</artifactId>
   <name>druid-kubernetes-overlord-extensions</name>
   <description>druid-kubernetes-overlord-extensions</description>

licenses.yaml

@@ -853,12 +853,33 @@ libraries:
 
 name: kubernetes fabric java client
 license_category: binary
-module: extensions-contrib/kubernetes-overlord-extensions
+module: extensions-core/kubernetes-overlord-extensions
 license_name: Apache License version 2.0
 version: 6.7.2
 libraries:
-  - io.fabric8: kubernetes-client
-
+  - io.fabric8: kubernetes-client-api
+  - io.fabric8: kubernetes-model-batch
+  - io.fabric8: kubernetes-model-core
+  - io.fabric8: kubernetes-model-admissionregistration
+  - io.fabric8: kubernetes-model-apiextensions
+  - io.fabric8: kubernetes-model-apps
+  - io.fabric8: kubernetes-model-autoscaling
+  - io.fabric8: kubernetes-model-certificates
+  - io.fabric8: kubernetes-model-common
+  - io.fabric8: kubernetes-model-coordination
+  - io.fabric8: kubernetes-model-discovery
+  - io.fabric8: kubernetes-model-events
+  - io.fabric8: kubernetes-model-extensions
+  - io.fabric8: kubernetes-model-flowcontrol
+  - io.fabric8: kubernetes-model-gatewayapi
+  - io.fabric8: kubernetes-model-metrics
+  - io.fabric8: kubernetes-model-networking
+  - io.fabric8: kubernetes-model-node
+  - io.fabric8: kubernetes-model-policy
+  - io.fabric8: kubernetes-model-rbac
+  - io.fabric8: kubernetes-model-resource
+  - io.fabric8: kubernetes-model-scheduling
+  - io.fabric8: kubernetes-model-storageclass
 ---
 
 name: kubernetes official java client
@@ -1026,6 +1047,26 @@ libraries:
 
 ---
 
+name: org.snakeyaml snakeyaml-engine
+license_category: binary
+module: extensions-core/druid-kubernetes-overlord-extensions
+license_name: Apache License version 2.0
+version: 2.6
+libraries:
+  - org.snakeyaml: snakeyaml-engine
+
+---
+
+name: org.yaml snakeyaml
+license_category: binary
+module: extensions-core/druid-kubernetes-overlord-extensions
+license_name: Apache License version 2.0
+version: 1.33
+libraries:
+  - org.yaml: snakeyaml
+
+---
+
 name: org.yaml snakeyaml
 license_category: binary
 module: extensions/druid-kubernetes-extensions

pom.xml

@@ -194,6 +194,7 @@
         <module>cloud/gcp-common</module>
         <!-- Core extensions -->
         <module>extensions-core/kubernetes-extensions</module>
+        <module>extensions-core/kubernetes-overlord-extensions</module>
         <module>extensions-core/avro-extensions</module>
         <module>extensions-core/azure-extensions</module>
         <module>extensions-core/datasketches</module>
@@ -250,7 +251,6 @@
         <module>extensions-contrib/aliyun-oss-extensions</module>
         <module>extensions-contrib/prometheus-emitter</module>
         <module>extensions-contrib/opentelemetry-emitter</module>
-        <module>extensions-contrib/kubernetes-overlord-extensions</module>
         <module>extensions-contrib/grpc-query</module>
         <module>extensions-contrib/druid-iceberg-extensions</module>
         <module>extensions-contrib/druid-deltalake-extensions</module>

website/redirects.js

@@ -294,6 +294,10 @@ const Redirects=[
     "from": "/docs/latest/development/extensions-contrib/google.html",
     "to": "/docs/latest/development/extensions-core/google"
   },
+  {
+    "from": "/docs/latest/development/extensions-contrib/k8s-jobs",
+    "to": "/docs/latest/development/extensions-core/k8s-jobs"
+  },
   {
     "from": "/docs/latest/development/integrating-druid-with-other-technologies.html",
     "to": "/docs/latest/ingestion/"