PCIP-3 Pulsar Extended Transaction API Enhancement Proposal

[liangyepianzhou]

Motivation

The motivation behind this proposal is to address the issue of duplicate message consumption when using Fail-over subscription mode with cumulative Ack in Pulsar. Despite the use of transactions, achieving exactly-once semantics has been problematic due to the inherent behavior of cumulative Ack. This issue is particularly challenging to resolve within the constraints of the Pulsar main repository for several reasons:

• Complexity and Longevity: Fixing this issue without altering the Client API is complex and time-consuming.
• Confusing API Usage: The existing transaction APIs in Pulsar are often confusing, with methods like abort() and commit() appearing synchronous but functioning asynchronously. This leads to incorrect usage patterns, such as needing to call abort().get() and commit().get() for proper operation.
• API Modification Challenges: Modifying the Client API in the Pulsar main repository is difficult due to the uncertainty of whether a new solution will be perfect and the lengthy API update cycle.

Modifications

The proposed solution involves designing a new transaction API in the contributor repository that wraps the original Transaction API. This approach offers several benefits:

• Concise Solution: By wrapping the original API, the problem can be solved in a straightforward and concise manner.
• Best Practice Reference: This wrapping serves as a best practice example that does not disrupt existing users' usage while providing a reference solution for those encountering similar issues.
• Enhanced Usability: The new API aims to clarify and optimize the transaction methods, making them more intuitive and less prone to misuse, thus improving the overall usability and clarity of transactional operations in Pulsar.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

[eolivelli]

I think that it is definitively worth to push this initiative to the main repo

[liangyepianzhou]

I think that it is definitively worth to push this initiative to the main repo

I understand your point that the initiative aims to address an existing bug in Pulsar, which indeed warrants consideration for integration into the main repository. However, modifying the public API in the Pulsar main repository is a challenging and lengthy process, and we cannot guarantee that the new API will be flawless and not require further modifications. Therefore, it makes sense to first iterate on the contributor repository, allowing some users to adopt it and solve their issues before considering a merge into the main repository.

[AuroraTwinkle]

The CI is failed.

[StevenLuMT]

Change the format of the title. Do you need to change it to:

[improve][pcip] PCIP-3 Pulsar Extended Transaction API Enhancement Proposal

[AuroraTwinkle]

LGTM

[eolivelli]

+1 to keep this here for now