How to get ordered delivery of messages at consumer side?

[shreyasarani23]

Hi @scholzj
Following is my topic configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: test-topic
  labels:
    strimzi.io/cluster: my-kafka-cluster
spec:
  partitions: 3
  replicas: 2
  config:
    min.insync.replicas: 1
    retention.ms: -1
    segment.bytes: 1073741824

I have produced records into the topic by using following properties

enable.idempotence=true
max.in.flight.requests.per.connection=5
acks=all
retries=2147483647

[kafka@my-kafka-cluster-kafka-0 kafka]$ bin/kafka-console-producer.sh --topic test-topic --bootstrap-server my-kafka-cluster-kafka-bootstrap:9092 --producer-property acks=all --producer-property enable.idempotence=true --producer-property max.in.flight.requests.per.connection=5 --producer-property retries=2147483647
>1
>2
>3
>4
>5
>6
>7
[kafka@my-kafka-cluster-kafka-0 kafka]$ bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server my-kafka-cluster-kafka-bootstrap:9092
3
6
1
2
4
5
7

Here the problem is that even though using enable.idempotence=true but at the consumer side I am not getting ordered delivery of messages. At the consumer side I want the output to be 1234567. Please help is there additional configuration required?

[scholzj]

Kafka guarantees ordering only on a per-partition basis. The messages you send will be split between 3 partitions. And when you consume them, you will receive them in block from the different partitions. So for example:

• Messages 3 and 6 are in partition 0
• Messages 1, 2 and 4 in partition 1
• And messages 5 and 7 in partition 2

As you can see, the per partition ordering is still maintained. (just an example -> they might be split differently in your case)

This is really a Kafka architecture thing, nothing to do with Strimzi per se. The Kafka Definitive Guide book (should be available for free to download AFAIK) explains the Kafka architecture and how it works quite well if you are interested in more details.

[shreyasarani23]

Oh you are saying that when consumer consumes messages it is rendered from all partitions in round robin fashion which is the default behaviour/architecture of kafka. In the strimzi documentation it is mentioned that if we use idempotence it preserves the message ordering. I thought the same message ordering will be reflected at the consumer side. Then what is the actual use of idempotence, what is its role and in which scenario is it useful?

[tomncooper]

Idempotence prevents retries by the producer resulting in duplicate messages (more details here). This can happen if an ack is lost but the 1st message was actually commited. The idempotent producer will prevent a retried message which has already been commited being committed again.

[scholzj]

Well, it does preserve the message ordering ... but as explained above, message ordering in Kafka is always only within a partition. Not within the whole topic.

[shreyasarani23]

Ok got it thanks @tomncooper @scholzj