Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

For eachBatch(), call to resolveOffset for each message followed by commitOffsetsIfNecessary does not commit messages unless autoCommitInterval or autoCommitThreshold is set #1729

Open
emranho95 opened this issue Jan 2, 2025 · 0 comments

Comments

@emranho95
Copy link

emranho95 commented Jan 2, 2025

Describe the bug
Call to resolveOffset() for each message followed by commitOffsetsIfNecessary() does not commit messages unless autoCommitInterval or autoCommitThreshold is set. But with the same configuration, if I set autoCommitInterval to a lower value that is less than the message processing time in the consumer side, ex: 10ms, the messages will be committed.

To Reproduce

Consumer config

  1. Use eachBatch() to handle messages
    2. autoCommit = false, eachBatchAutoResolve = false, eachBatchAutoResolve = false
    Resolve messages after each message processing and then call commitOffsetsIfNecessary()
Inside eachBatch() method ->
 for (const message of messages) {
          resolveOffset(message.rawMessage.offset);
          await heartbeat();
 }
 await commitOffsetsIfNecessary();

  1. Produce small amount of messages than will be consumed within few milliseconds, ex: 5
  2. Consume the messages using one consumer

Expected behavior

  1. The committed offsets fetched by admin.fetchOffsets() should match the length of the messages.

Observed behavior
None of the messages are committed and so it returns -1.

Environment:

  • OS: Mac OS
  • KafkaJS version [e.g. 2.2.4]
  • NodeJS version [npm v10.5.0]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant