This repository has been archived by the owner on Jul 1, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: Chris Giblin <[email protected]>
- Loading branch information
1 parent
dcbc222
commit f115598
Showing
1 changed file
with
125 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,125 @@ | ||
# Getting started | ||
|
||
Requirements: | ||
- a Kafka instance, version 2.8.0 or older, which you can configure | ||
- Java 11 | ||
- Apache maven installed in your command line environment | ||
- git command | ||
|
||
|
||
The steps for getting started with this initial version of topic encryption are outlined below: | ||
1. Clone the repository and set your working path. | ||
2. Compile | ||
3. Configure the Kafka advertised address | ||
4. Configure the proxy | ||
5. Run the proxy | ||
6. Start kafka | ||
7. Run kafka clients | ||
|
||
Each of these steps is described in detail below with an example. | ||
|
||
## Scenario | ||
|
||
In the scenario to get started, all components run on the same system, `localhost`. The Kafka broker can also run remotely. The minimum requirement is that one can update the broker configuration file and restart the broker. In this example, however, we run the broker locally. | ||
|
||
The proxy will listen on port 1234 and the broker listens on its standard port 9092 as depicted below: | ||
|
||
``` | ||
Kafka client Proxy Kafka broker | ||
o------------o 1234 o------------o 9092 | ||
``` | ||
|
||
The clients are reconfigured to use port 1234 (details below). | ||
|
||
A policy to encrypt all topics with the same key, along with a test key management system (KMS) which returns a hard-coded AES key, is used. | ||
|
||
The following sections provide details for each step in running the encrypting proxy. | ||
|
||
### 1. Clone the repository and set your working path | ||
``` | ||
git clone [email protected]:strimzi/topic-encryption.git | ||
cd topic-encryption | ||
``` | ||
|
||
### 2. Compile | ||
|
||
``` | ||
mvn install | ||
``` | ||
|
||
### 3. Configure the Kafka broker's listeners | ||
The address advertised by Kafka must be that of the proxy, not the broker itself. | ||
|
||
Modify the `advertised.listeners` property in `server.properties` to point to the proxy host and port, as shown in the snippet below: | ||
|
||
``` | ||
# The address the socket server listens on. It will get the value returned from | ||
# java.net.InetAddress.getCanonicalHostName() if not configured. | ||
# FORMAT: | ||
# listeners = listener_name://host_name:port | ||
# EXAMPLE: | ||
# listeners = PLAINTEXT://your.host.name:9092 | ||
listeners=PLAINTEXT://:9092 | ||
# Hostname and port the broker will advertise to producers and consumers. If not set, | ||
# it uses the value for "listeners" if configured. Otherwise, it will use the value | ||
# returned from java.net.InetAddress.getCanonicalHostName(). | ||
advertised.listeners=PLAINTEXT://127.0.0.1:1234 | ||
``` | ||
Stop the Kafka broker and start it after the proxy is running. | ||
|
||
### 4. Configure the proxy | ||
Set the working directory to the proxy's target folder: | ||
``` | ||
$ cd vertx-proxy/target/ | ||
``` | ||
|
||
Create a configuration file, `config.json` and add the following JSON contents: | ||
|
||
``` | ||
{ | ||
"listening_port" : 1234, | ||
"kafka_broker" : "localhost:9092", | ||
"policy_repo" : "test" | ||
} | ||
``` | ||
### 5. Run the proxy | ||
With the current path set to the target directory, run the proxy with the following Java invocation: | ||
|
||
``` | ||
$ java -cp vertx-proxy-0.0.1-SNAPSHOT-fat.jar io.strimzi.kafka.proxy.vertx.VertRunner | ||
``` | ||
|
||
If successfully started, the following output appears: | ||
``` | ||
$ java -cp vertx-proxy-0.0.1-SNAPSHOT-fat.jar io.strimzi.kafka.proxy.vertx.VertRunner | ||
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. | ||
2022-04-13 10:30:12 INFO KafkaProxyVerticle:46 35 - Kafka version: 2.8.0 | ||
2022-04-13 10:30:12 INFO KafkaProxyVerticle:75 35 - Listening on port 1234 | ||
``` | ||
|
||
### 6. Start Kafka broker | ||
|
||
Now start the Kafka broker, for example: | ||
``` | ||
bin/kafka-server-start.sh config/server.properties | ||
``` | ||
|
||
### 7. Run Kafka clients | ||
Start the Kafka console producer (note the proxy address in the broker list): | ||
|
||
``` | ||
bin/kafka-console-producer.sh --broker-list localhost:1234 --topic enctest --producer.config config/producer.properties | ||
``` | ||
|
||
Start the Kafka console consumer, like the producer, specifying the proxy host and port: | ||
``` | ||
bin/kafka-console-consumer.sh --bootstrap-server localhost:1234 --consumer.config config/consumer.properties --topic enctest --from-beginning | ||
``` | ||
|
||
Enter arbitry data in the producer and verify that it appears in consumer. | ||
|
||
Inspect the topic segment files and verify they indeed are encrypted. | ||
``` | ||
hexdump /tmp/kafka-logs/enctest-0/00000000000000000000.log | ||
``` |