Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: kafkaesque-io/pulsar-client-go
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 0.3.0-pulsar-perf
Choose a base ref
...
head repository: kafkaesque-io/pulsar-client-go
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
Loading
Showing with 39,606 additions and 7,351 deletions.
  1. +22 −0 .github/workflows/bot.yml
  2. +13 −12 .github/workflows/go.yml
  3. +5 −4 .github/workflows/project.yml
  4. +0 −2 .golangci.yml
  5. +135 −0 CHANGELOG.md
  6. +2 −2 Dockerfile
  7. +30 −0 README.md
  8. +1 −1 VERSION
  9. +2 −1 docker-ci.sh
  10. +31 −21 docs/release-process.md
  11. +15 −6 go.mod
  12. +164 −10 go.sum
  13. +3 −1 integration-tests/license_test.go
  14. +23 −0 integration-tests/pb/build.sh
  15. +100 −0 integration-tests/pb/hello.pb.go
  16. +25 −0 integration-tests/pb/hello.proto
  17. +4 −1 integration-tests/standalone.conf
  18. +120 −0 oauth2/auth.go
  19. +65 −0 oauth2/auth_suite_test.go
  20. +338 −0 oauth2/authorization_tokenretriever.go
  21. +338 −0 oauth2/authorization_tokenretriever_test.go
  22. +142 −0 oauth2/cache/cache.go
  23. +158 −0 oauth2/client_credentials_flow.go
  24. +183 −0 oauth2/client_credentials_flow_test.go
  25. +74 −0 oauth2/client_credentials_provider.go
  26. +98 −0 oauth2/clock/clock.go
  27. +275 −0 oauth2/clock/testing/fake_clock.go
  28. +57 −0 oauth2/config_tokenprovider.go
  29. +91 −0 oauth2/config_tokenprovider_test.go
  30. +203 −0 oauth2/device_code_flow.go
  31. +230 −0 oauth2/device_code_flow_test.go
  32. +133 −0 oauth2/device_code_provider.go
  33. +12 −0 oauth2/go.mod
  34. +117 −0 oauth2/go.sum
  35. +58 −0 oauth2/oidc_endpoint_provider.go
  36. +92 −0 oauth2/oidc_endpoint_provider_test.go
  37. +194 −0 oauth2/store/keyring.go
  38. +87 −0 oauth2/store/memory.go
  39. +45 −0 oauth2/store/store.go
  40. +2 −3 perf/perf-consumer.go
  41. +9 −4 perf/perf-producer.go
  42. +32 −1 perf/pulsar-perf-go.go
  43. +44 −0 pulsar/batcher_builder.go
  44. +25 −1 pulsar/client.go
  45. +35 −8 pulsar/client_impl.go
  46. +121 −1 pulsar/client_impl_test.go
  47. +25 −2 pulsar/consumer.go
  48. +178 −36 pulsar/consumer_impl.go
  49. +51 −0 pulsar/consumer_interceptor.go
  50. +46 −15 pulsar/consumer_multitopic.go
  51. +270 −109 pulsar/consumer_partition.go
  52. +25 −13 pulsar/consumer_partition_test.go
  53. +38 −27 pulsar/consumer_regex.go
  54. +10 −29 pulsar/consumer_regex_test.go
  55. +729 −4 pulsar/consumer_test.go
  56. +111 −0 pulsar/default_router.go
  57. +45 −0 pulsar/default_router_bench_test.go
  58. +144 −0 pulsar/default_router_test.go
  59. +13 −7 pulsar/dlq_router.go
  60. +10 −9 pulsar/helper.go
  61. +85 −27 pulsar/impl_message.go
  62. +49 −0 pulsar/impl_message_bench_test.go
  63. +6 −6 pulsar/impl_message_test.go
  64. +145 −0 pulsar/internal/auth/oauth2.go
  65. +142 −0 pulsar/internal/auth/oauth2_test.go
  66. +3 −0 pulsar/internal/auth/provider.go
  67. +10 −0 pulsar/internal/auth/tls.go
  68. +1 −1 pulsar/internal/auth/token.go
  69. +165 −79 pulsar/internal/batch_builder.go
  70. +12 −38 pulsar/internal/blocking_queue.go
  71. +22 −7 pulsar/internal/blocking_queue_test.go
  72. +7 −6 pulsar/internal/buffer.go
  73. +50 −35 pulsar/internal/commands.go
  74. +21 −12 pulsar/internal/compression/compression.go
  75. +122 −0 pulsar/internal/compression/compression_bench_test.go
  76. +23 −10 pulsar/internal/compression/compression_test.go
  77. +46 −18 pulsar/internal/compression/lz4.go
  78. +31 −8 pulsar/internal/compression/noop.go
  79. +1,795 −0 pulsar/internal/compression/test_data_sample.txt
  80. +25 −11 pulsar/internal/compression/zlib.go
  81. +10 −35 pulsar/internal/compression/zstd.go
  82. +81 −0 pulsar/internal/compression/zstd_cgo.go
  83. +72 −0 pulsar/internal/compression/zstd_go.go
  84. +195 −88 pulsar/internal/connection.go
  85. +50 −16 pulsar/internal/connection_pool.go
  86. +10 −10 pulsar/internal/connection_reader.go
  87. +0 −80 pulsar/internal/default_router.go
  88. +0 −102 pulsar/internal/default_router_test.go
  89. +237 −0 pulsar/internal/key_based_batch_builder.go
  90. +14 −8 pulsar/internal/lookup_service.go
  91. +14 −11 pulsar/internal/lookup_service_test.go
  92. +364 −0 pulsar/internal/metrics.go
  93. +0 −6,188 pulsar/internal/pb/PulsarApi.pb.go
  94. +25,852 −0 pulsar/internal/pulsar_proto/PulsarApi.pb.go
  95. +941 −0 pulsar/internal/pulsar_proto/PulsarApi.proto
  96. +13 −0 pulsar/internal/pulsar_proto/README.md
  97. +42 −34 pulsar/internal/rpc_client.go
  98. +73 −19 pulsar/internal/semaphore.go
  99. +60 −0 pulsar/internal/semaphore_test.go
  100. +5 −1 pulsar/internal/topic_name.go
  101. +7 −0 pulsar/internal/topic_name_test.go
  102. +107 −0 pulsar/key_shared_policy.go
  103. +91 −0 pulsar/key_shared_policy_test.go
  104. +52 −0 pulsar/log/log.go
  105. +67 −0 pulsar/log/logger.go
  106. +140 −0 pulsar/log/wrapper_logrus.go
  107. +21 −0 pulsar/message.go
  108. +17 −9 pulsar/negative_acks_tracker.go
  109. +11 −8 pulsar/negative_acks_tracker_test.go
  110. +316 −0 pulsar/primitiveSerDe.go
  111. +143 −0 pulsar/primitiveSerDe_test.go
  112. +51 −1 pulsar/producer.go
  113. +93 −39 pulsar/producer_impl.go
  114. +43 −0 pulsar/producer_interceptor.go
  115. +312 −86 pulsar/producer_partition.go
  116. +249 −8 pulsar/producer_test.go
  117. +21 −1 pulsar/reader.go
  118. +80 −18 pulsar/reader_impl.go
  119. +222 −0 pulsar/reader_test.go
  120. +139 −0 pulsar/retry_router.go
  121. +502 −0 pulsar/schema.go
  122. +58 −0 pulsar/schema_def_test.go
  123. +422 −0 pulsar/schema_test.go
  124. +1 −1 stable.txt
22 changes: 22 additions & 0 deletions .github/workflows/bot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
name: Bot tests
on:
issue_comment:
types: [created]

jobs:
bot:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Bot actions
uses: zymap/bot@v1.0.1
env:
GITHUB_TOKEN: ${{ secrets.GO_CLIENT_BOT_TOKEN }}
with:
repo_owner: apache
repo_name: pulsar-client-go
rerun_cmd: rerun failure checks
comment: ${{ github.event.comment.body }}
25 changes: 13 additions & 12 deletions .github/workflows/go.yml
Original file line number Diff line number Diff line change
@@ -1,21 +1,22 @@
name: Go
on: [pull_request]
jobs:

build:
name: Build
runs-on: ubuntu-latest
strategy:
matrix:
go-version: [1.11, 1.12, 1.13, 1.14]
steps:
- name: Set up Go
uses: actions/setup-go@v1
with:
go-version: ${{ matrix.go-version }}
id: go

- name: Set up Go 1.12
uses: actions/setup-go@v1
with:
go-version: 1.12
id: go

- name: Check out code into the Go module directory
uses: actions/checkout@v2
- name: Check out code into the Go module directory
uses: actions/checkout@v2

- name: Test
run: |
./docker-ci.sh
- name: Test
run: |
./docker-ci.sh ${{ matrix.go-version }}
9 changes: 5 additions & 4 deletions .github/workflows/project.yml
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
name: Project Style Check
on: [pull_request]
jobs:

build:
name: Build
runs-on: ubuntu-latest
strategy:
matrix:
go-version: [1.11, 1.12, 1.13, 1.14]
steps:

- name: Set up Go 1.12
- name: Set up Go
uses: actions/setup-go@v1
with:
go-version: 1.12
go-version: ${{ matrix.go-version }}
id: go

- name: Check out code into the Go module directory
2 changes: 0 additions & 2 deletions .golangci.yml
Original file line number Diff line number Diff line change
@@ -13,10 +13,8 @@ linters:
- golint
- gosimple
- govet
- ineffassign
- interfacer
- misspell
- staticcheck
- structcheck
- stylecheck
- typecheck
135 changes: 135 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -23,6 +23,141 @@

All notable changes to this project will be documented in this file.

[0.3.0] 2020-11-11

## Feature

* Support retry letter topic in Go client, see [PR-359](https://github.com/apache/pulsar-client-go/pull/359).
* Support limit the retry number of reconnectToBroker, see [PR-360](https://github.com/apache/pulsar-client-go/pull/360).
* Support key shared policy in Go client, see [PR-363](https://github.com/apache/pulsar-client-go/pull/363).
* Add schema logic in producer and consumer, see [PR-368](https://github.com/apache/pulsar-client-go/pull/368).


## Improve

* Fix panic on receiverQueueSize set to `-1`, see [PR-361](https://github.com/apache/pulsar-client-go/pull/361).
* Fix may lead to panic test case, see [PR-369](https://github.com/apache/pulsar-client-go/pull/369).
* Send delay message individually even batching is enabled, see [PR-372](https://github.com/apache/pulsar-client-go/pull/372).
* Fixed buffer resize when writing request on connection, see [PR-374](https://github.com/apache/pulsar-client-go/pull/374).
* Fixed deadlock in DLQ ack processing, see [PR-375](https://github.com/apache/pulsar-client-go/pull/375).
* Fix deadlock when connection closed, see [PR-376](https://github.com/apache/pulsar-client-go/pull/376).
* Fix producer deadlock after write failure, see [PR-378](https://github.com/apache/pulsar-client-go/pull/378).
* Fix maxMessageSize not effective even if aligned with broker, see [PR-381](https://github.com/apache/pulsar-client-go/pull/381).
* Update default router to switch partition on all batching thresholds, see [PR-383](https://github.com/apache/pulsar-client-go/pull/383).
* Replaced `github.com/DataDog/zstd` with `github.com/datadog/zstd`, see [PR-385](https://github.com/apache/pulsar-client-go/pull/385).
* Fix retry policy not effective with non-FQDN topics, see [PR-386](https://github.com/apache/pulsar-client-go/pull/386).


[0.2.0] 2020-08-28

## Feature

* Expose BatchingMaxSize from ProducerOptions, see [PR-280](https://github.com/apache/pulsar-client-go/pull/280).
* Allow applications to configure the compression level, see [PR-290](https://github.com/apache/pulsar-client-go/pull/290).
* Support producer name for Message, see [PR-299](https://github.com/apache/pulsar-client-go/pull/299).
* Support oauth2 authentication for pulsar-client-go, see [PR-313](https://github.com/apache/pulsar-client-go/pull/313).
* Add interceptor feature for Go client, see [PR-314](https://github.com/apache/pulsar-client-go/pull/314).
* Export client metrics to Prometheus, see [PR-317](https://github.com/apache/pulsar-client-go/pull/317).
* Add Name method to Consumer interface, see [PR-321](https://github.com/apache/pulsar-client-go/pull/321).
* Add oauth2 to the provider, see [PR-338](https://github.com/apache/pulsar-client-go/pull/338).
* Support specified the oauth2 private key with prefix `file://` and `data://`, see [PR-343](https://github.com/apache/pulsar-client-go/pull/343).
* Fix the keyfile unmarshal error, see [PR-339](https://github.com/apache/pulsar-client-go/pull/339).
* Add a new method to create auth provider from tls cert supplier, see [PR-347](https://github.com/apache/pulsar-client-go/pull/347).
* Add seek logic for reader, see [PR-356](https://github.com/apache/pulsar-client-go/pull/356).

## Improve

* Use .asf.yaml to configure github repo, see [PR-216](https://github.com/apache/pulsar-client-go/pull/216).
* Auto update the client to handle changes in number of partitions, see [PR-221](https://github.com/apache/pulsar-client-go/pull/221).
* Clean callbacks of connection after run loop stopped, see [PR-248](https://github.com/apache/pulsar-client-go/pull/248).
* Fix unable to close consumer after unsubscribe in Shared Subscription, see [PR-283](https://github.com/apache/pulsar-client-go/pull/283).
* Introduced lifecycle for compression providers, see [PR-284](https://github.com/apache/pulsar-client-go/pull/284).
* Use maxPendingMessages for sizing producer eventsChan, see [PR-285](https://github.com/apache/pulsar-client-go/pull/285).
* Avoid contention on producer mutex on critical path, see [PR-286](https://github.com/apache/pulsar-client-go/pull/286).
* Switched to DataDog zstd wrapper, reusing the compression ctx, see [PR-287](https://github.com/apache/pulsar-client-go/pull/287).
* Fix panic when creating consumer with ReceiverQueueSize set to -1, see [PR-289](https://github.com/apache/pulsar-client-go/pull/289).
* Used pooled buffering for compression and batch serialization, see [PR-292](https://github.com/apache/pulsar-client-go/pull/292).
* Use gogofast to have in-place protobuf serialization, see [PR-294](https://github.com/apache/pulsar-client-go/pull/294).
* Added semaphore implementation with lower contention, see [PR-298](https://github.com/apache/pulsar-client-go/pull/298).
* Fixed pooled buffer lifecycle, see [PR-300](https://github.com/apache/pulsar-client-go/pull/300).
* Removed blocking queue iterator, see [PR-301](https://github.com/apache/pulsar-client-go/pull/301).
* Fix panic in CreateReader API using custom MessageID for ReaderOptions, see [PR-305](https://github.com/apache/pulsar-client-go/pull/305).
* Change connection failed warn log to error and print error message, see [PR-309](https://github.com/apache/pulsar-client-go/pull/309).
* Share buffer pool across all partitions, see [PR-310](https://github.com/apache/pulsar-client-go/pull/310).
* Add rerun feature test command to repo, see [PR-311](https://github.com/apache/pulsar-client-go/pull/311).
* Fix CompressMaxSize() for ZLib provider, see [PR-312](https://github.com/apache/pulsar-client-go/pull/312).
* Reduce the size of the MessageID structs by one word on 64-bit arch, see [PR-316](https://github.com/apache/pulsar-client-go/pull/316).
* Do not allocate MessageIDs on the heap, see [PR-319](https://github.com/apache/pulsar-client-go/pull/319).
* Different MessageID implementations for message Production and Consumption, see [PR-324](https://github.com/apache/pulsar-client-go/pull/324).
* Fix producer block when the producer with the same id, see [PR-326](https://github.com/apache/pulsar-client-go/pull/326).
* Get the last message when LatestMessageID and inclusive, see [PR-329](https://github.com/apache/pulsar-client-go/pull/329).
* Fix go.mod issue with invalid version, see [PR-330](https://github.com/apache/pulsar-client-go/pull/330).
* Fix producer goroutine leak, see [PR-331](https://github.com/apache/pulsar-client-go/pull/331).
* Fix producer state by reconnecting when receiving unexpected receipts, see [PR-336](https://github.com/apache/pulsar-client-go/pull/336).
* Avoid producer deadlock on connection closing, see [PR-337](https://github.com/apache/pulsar-client-go/pull/337).

## Contributors

Our thanks go to the following contributors from the community for helping this release:

- [LvBay](https://github.com/LvBay)
- [cgfork](https://github.com/cgfork)
- [jaysun91](https://github.com/jaysun91)
- [liangyuanpeng](https://github.com/liangyuanpeng)
- [nitishv](https://github.com/nitishv)
- [quintans](https://github.com/quintans)
- [snowcrumble](https://github.com/snowcrumble)
- [shohi](https://github.com/shohi)
- [simonswine](https://github.com/simonswine)
- [dferstay](https://github.com/dferstay)
- [zymap](https://github.com/zymap)


[0.1.1] 2020-06-19

## Improve

- [Fixed batching flag logic](https://github.com/apache/pulsar-client-go/pull/209)
- [Fix data race when accessing partition producer state](https://github.com/apache/pulsar-client-go/pull/215)
- [Fixed tls connection issue](https://github.com/apache/pulsar-client-go/pull/220)
- [Add flag to disable forced topic creation](https://github.com/apache/pulsar-client-go/pull/226)
- [Add Athenz authentication provider](https://github.com/apache/pulsar-client-go/pull/227)
- [Fixed race condition in producer Flush() operation](https://github.com/apache/pulsar-client-go/pull/229)
- [Removed unnecessary flush in sync Send() operation](https://github.com/apache/pulsar-client-go/pull/230)
- [Allow empty payload for nonbatch message](https://github.com/apache/pulsar-client-go/pull/236)
- [Add internal connectionReader readAtLeast error information](https://github.com/apache/pulsar-client-go/pull/237)
- [Fix zstd memory leak of zstdProvider ](https://github.com/apache/pulsar-client-go/pull/245)
- [Expose replicated from filed on message struct](https://github.com/apache/pulsar-client-go/pull/251)
- [Fix send async comments](https://github.com/apache/pulsar-client-go/pull/254)
- [Fix perf-produce cannot be closed](https://github.com/apache/pulsar-client-go/pull/255)
- [Fix perf-producer target](https://github.com/apache/pulsar-client-go/pull/256)
- [Fix fail to add batchbuilder](https://github.com/apache/pulsar-client-go/pull/260)
- [skip debug print out when batch disabled with no messages](https://github.com/apache/pulsar-client-go/pull/261)
- [Add check for max message size](https://github.com/apache/pulsar-client-go/pull/263)
- [Build and test with multiple versions of Go](https://github.com/apache/pulsar-client-go/pull/269)
- [When CGO is enabled, use C version of ZStd](https://github.com/apache/pulsar-client-go/pull/270)
- [Stop partition discovery on Close](https://github.com/apache/pulsar-client-go/pull/272)
- [Microbenchmark for compression](https://github.com/apache/pulsar-client-go/pull/275)
- [Allow to have multiple connections per broker](https://github.com/apache/pulsar-client-go/pull/276)
- [Increase writeRequestsCh channel buffer size](https://github.com/apache/pulsar-client-go/pull/277)

### Contributors

Our thanks go to the following contributors from the community for helping this release:

- [yarthur1](https://github.com/yarthur1)
- [vergnes](https://github.com/vergnes)
- [sijie](https://github.com/sijie)
- [shustsud](https://github.com/shustsud)
- [rueian](https://github.com/rueian)
- [mileschao](https://github.com/mileschao)
- [keithnull](https://github.com/keithnull)
- [abatilo](https://github.com/abatilo)
- [cornelk](https://github.com/cornelk)
- [equanz](https://github.com/equanz)
- [jerrypeng](https://github.com/jerrypeng)
- [jonyhy96](https://github.com/jonyhy96)

[0.1.0] 2020-03-24

## New Feature
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -17,9 +17,9 @@
# under the License.
#

ARG GO_VERSION=golang:1.12
FROM apachepulsar/pulsar:latest as pulsar

FROM golang:1.12 as go
FROM $GO_VERSION as go

RUN apt-get update && apt-get install -y openjdk-11-jre-headless

30 changes: 30 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -103,6 +103,36 @@ fmt.Printf("Received message msgId: %#v -- content: '%s'\n",

```

Create a Reader:

```go
client, err := pulsar.NewClient(pulsar.ClientOptions{URL: "pulsar://localhost:6650"})
if err != nil {
log.Fatal(err)
}

defer client.Close()

reader, err := client.CreateReader(pulsar.ReaderOptions{
Topic: "topic-1",
StartMessageID: pulsar.EarliestMessageID(),
})
if err != nil {
log.Fatal(err)
}
defer reader.Close()

for reader.HasNext() {
msg, err := reader.Next(context.Background())
if err != nil {
log.Fatal(err)
}

fmt.Printf("Received message msgId: %#v -- content: '%s'\n",
msg.ID(), string(msg.Payload()))
}
```

## Contributing

Contributions are welcomed and greatly appreciated. See [CONTRIBUTING.md](CONTRIBUTING.md) for details on submitting patches and the contribution workflow.
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
// This version number refers to the currently released version number
// Please fix the version when release.
v0.1.0
v0.3.0
3 changes: 2 additions & 1 deletion docker-ci.sh
Original file line number Diff line number Diff line change
@@ -25,7 +25,8 @@ cd ${SRC_DIR}

IMAGE_NAME=pulsar-client-go-test:latest

docker build -t ${IMAGE_NAME} .
GO_VERSION=${1:-1.12}
docker build -t ${IMAGE_NAME} --build-arg GO_VERSION="golang:${GO_VERSION}" .

docker run -i -v ${PWD}:/pulsar-client-go ${IMAGE_NAME} \
bash -c "cd /pulsar-client-go && ./run-ci.sh"
Loading