v1.4.0

Major Features 🔥

• Introducing MonoVertex backed by Rust 🦀
• Introducing gRPC bi-directional streaming between numa container and UDF
• Metrics visualiser for MonoVertex
• Flexible scaling with replicasPerScaleUp and replicasPerScaleDown
• Enabling resourceClaims and readyReplicas for Vertex and MonoVertex
• Rolling Update for both Pipeline and MonoVertex

🚨 Breaking Change 🚨

• SDKs have to be updated to the latest version.

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.4.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.4.0/validating-webhook-install.yaml

Other Changes

• docs: updated CHANGELOG.md by @github-actions in #1916
• chore: mono-vertex code review by @vigith in #1917
• feat: add server-info support and versioning to MonoVertex by @kohlisid in #1918
• chore: improve shutdown and health checks for MonoVertex by @yhl25 in #1919
• chore: rename proto name for sourcetransform by @vigith in #1921
• chore: reorganize rust code by @yhl25 in #1922
• Add Lockheed to Users.md by @mdwarne1 in #1934
• chore: add transformer to MonoVertex example by @vigith in #1935
• fix: retry failed messages for MonoVertex sink by @yhl25 in #1933
• chore: implement APIs for mono vertex UI server by @KeranYang in #1931
• chore: update last updated time for mvtx by @yhl25 in #1939
• chore: avoid partial acks by @yhl25 in #1938
• chore: handle drop messages from transformer by @yhl25 in #1942
• feat: autoscaling for MonoVertex by @whynowy in #1927
• fix: default resources mutated when applying templates by @whynowy in #1948
• test: add mono vertex e2e tests by @yhl25 in #1945
• feat: Mono vertex UI by @veds-g in #1941
• chore: add missed tests for java by @KeranYang in #1953
• chore: switch to scratch base image by @BulkBeing in #1955
• feat: enable fallback sink for mvtx by @yhl25 in #1957
• feat: add health for monovertex by @kohlisid in #1954
• fix: replicas derived in UI from mvtx status instead of spec by @veds-g in #1965
• chore: send negative value rateNotAvailable by @kohlisid in #1966
• refactor: re-arrange e2e tests by @KeranYang in #1961
• chore: version update for releasing by @KeranYang in #1944
• docs: updated CHANGELOG.md by @github-actions in #1968
• doc: update roadmap by @vigith in #1970
• chore: fix broken links by @ayildirim21 in #1972
• chore: kustomize transformation config for MonoVertex by @whynowy in #1977
• fix: remove coloring in logs by @vigith in #1975
• fix: minor perf improvements of mvtx fallback sink by @vigith in #1967
• chore: make nightly build pipeline faster by @BulkBeing in #1958
• fix: adding not available for negative processing rates by @veds-g in #1983
• chore: enable current/desired replicas by @whynowy in #1987
• fix: do not pass scale info to MonoVertex by @whynowy in #1990
• chore: metrics and https support for serving by @yhl25 in #1985
• fix: test coverage generation for Rust code by @BulkBeing in #1993
• chore: use numaflow models to create minimum pipeline spec for serving by @yhl25 in #1995
• fix: e2e testing isbsvc deletion timeout issue by @KeranYang in #1997
• chore: fix servesink by @yhl25 in #1999
• chore: disallow MonoVertex replicas less than min or greater than max by @whynowy in #1994
• test: avoid building image twice in e2e testing ci by @whynowy in #2001
• chore(deps): bump micromatch from 4.0.7 to 4.0.8 in /ui by @dependabot in #2002
• feat: more flexible scaling with replicasPerScaleUp and replicasPerScaleDown by @whynowy in #2003
• fix: log format with config load error by @xdevxy in #2000
• chore: mark imageErr as unhealthy by @kohlisid in #2004
• feat: enable resourceClaims for vertex and monovtx by @whynowy in #2009
• chore: refactor metric variable names by @vigith in #2012
• feat: introduce readyReplicas for Vertex and MonoVertex by @whynowy in #2014
• fix: add latency metrics for mvtx by @vigith in #2013
• chore: use RUST_LOG env to control logging in rust binary by @yhl25 in #2017
• chore(deps): bump webpack from 5.93.0 to 5.94.0 in /ui by @dependabot in #2018
• feat: allow configurable retryStrategy by @kohlisid in #2010
• chore: placeholders for rolling update by @whynowy in #2019
• docs: updated CHANGELOG.md by @github-actions in #2021
• chore: added Seekr to USERS.md by @lanedd in #2022
• chore: reduce 5 second wait time every time redis sink check runs in e2e by @KeranYang in #2023
• chore: convert onFailureStrategy to enum in rust by @kohlisid in #2020
• chore: add docs for retry by @kohlisid in #2024
• chore: use kube_codegen.sh instead in codegen. by @xdevxy in #2011
• fix: pause lifecyle changes and add drained status by @kohlisid in #2028
• chore: improve pending logs to be on single line by @vigith in #2037
• feat: rolling update for MonoVertex by @whynowy in #2029
• chore: minor pipeline controller refactor by @whynowy in #2039
• feat: rolling update for Pipeline Vertex by @whynowy in #2040
• chore: pin dependency versions in rust model by @whynowy in #2048
• fix: builtin transformer should keep the keys by @KeranYang in #2047
• fix: Fix numaflow-rs binary location in image by @BulkBeing in #2050
• chore: use readyReplicas to calculate desired replicas by @whynowy in #2052
• chore: remove server info dependencies on go sdk by @KeranYang in #2060
• chore(deps): bump express from 4.19.2 to 4.21.0 in /ui by @dependabot in #2061
• chore: patch instead of update and bugfix by @whynowy in #2059
• fix: skip updating phase for resource check by @kohlisid in #2065
• doc: example for PVC by @vigith in #2067
• feat: allow customization on readyz and livez config by @whynowy in #2068
• chore: suppress some misleading logs from raters by @whynowy in #2075
• fix: support version compatibility check for pre-release versions by @KeranYang in #2069
• Fix: Use Merge patch rather than json patch for pause-timestamp annotation apply by @juliev0 in #2078
• feat: Bidirectional Streaming for User Defined Source by @yhl25 in #2056
• fix: rollback codegen script by @whynowy in #2079
• chore: fix numaflow-rs by @vigith in #2081
• feat: add pause for monovertex by @kohlisid in #2077
• feat: container-type level version compatibility check by @KeranYang in #2087
• feat: Bidirectional Streaming for UDSink by @yhl25 in #2080
• docs: updated CHANGELOG.md by @github-actions in #2094
• chore: fix 2 issues found by code scanning by @whynowy in #2095
• chore(deps): bump rollup from 2.79.1 to 2.79.2 in /ui by @dependabot in #2096
• chore: organize rust code by @vigith in #2099
• chore: merge source_{reader,acker} impl into one by @vigith in #2102
• chore: rename monovertex in Dockerfile by @vigith in #2101
• feat: Use gRPC bidirectional streaming for source transformer by @BulkBeing in #2071
• chore: check for pause timeout after errors by @kohlisid in #2084
• chore(deps): bump tonic from 0.12.2 to 0.12.3 in /rust by @dependabot in #2111
• feat: container-level version compatibility check for monovertex by @KeranYang in #2108
• feat: implement Source trait and use it for user-defined source by @vigith in #2114
• chore: add rust formatting check to CI by @KeranYang in #2117
• Debugging unit test timeout in CI by @BulkBeing in #2118
• chore: separate ack by @vigith in #2119
• chore: Unit tests for metrics.rs by @BulkBeing in #2122
• fix: create buffers and buckets before updating Vertices by @juliev0 in #2112
• chore: k8s lib version upgrade to 1.31 by @whynowy in #2127
• feat: generator based on ticker by @vigith in #2126
• chore: enable python source e2e by @kohlisid in #2116
• refactor: generate static gRPC clients by @whynowy in #2128
• feat: Make Generator Support Leaky Bucket by @yhl25 in #2129
• chore: simplified generator relying purely on Polling by @vigith in #2132
• feat: Use gRPC bidirectional streaming for map by @yhl25 in #2120
• chore: always install specified codegen tools by @whynowy in #2133
• feat: integrate tickgen with monovertex by @yhl25 in #2136
• feat: expose ports for user defined containers by @whynowy in #2135
• fix: pipeline pausing race conditions of draining and terminating source by @kohlisid in #2131
• docs: updated CHANGELOG.md by @github-actions in #2138
• feat: add keys into kafka header while producing by @vigith in #2143
• feat: Unify Batch Map and Unary Map Operations Using a Shared gRPC Protocol by @yhl25 in #2139
• feat: set kafka keys if setKey is set by @vigith in #2146
• feat: actor pattern for forwarder + sink trait by @vigith in #2141
• feat: Unify MapStream and Unary Map Operations Using a Shared gRPC Protocol by @yhl25 in #2149
• feat: Log sink implementation for Monovertex by @BulkBeing in #2150
• feat: support multiple controller with instance config by @whynowy in #2153
• chore: Refactor - Rename Error variants by @BulkBeing in #2154
• fix: create histogram buckets in a range by @kohlisid in #2144
• chore: multi controller awareness on validating webhook by @whynowy in #2156
• chore: do not unwrap by @vigith in #2152
• chore: generate rust protobuf objects by @vigith in #2157
• chore: rename numaflow-grpc to numaflow-pb by @whynowy in #2159
• feat: ISB(jetstream) writer framework by @vigith in #2160
• chore: organize rust imports by @yhl25 in #2164
• feat: check if the buffer is full before writing to ISB by @yhl25 in #2166
• chore: use enums for offset by @yhl25 in #2169
• chore: adding unit tests for offset enum by @yhl25 in #2170
• feat: blackhole sink for Monovertex by @BulkBeing in #2167
• fix: main branch, offset type got updated by @vigith in #2171
• chore: add beumer group to USERS by @th0ger in #2168
• chore: Make TickGen compatible with current builtin by @BulkBeing in #2162
• feat: config management for numaflow rust by @vigith in #2172
• chore: Use sub-registries for transformer/sink/fbsink metrics. by @BulkBeing in #2173
• fix(controller): incorporate instance into lease lock name by @whynowy in #2177
• fix(metrics): fix incorrect metric label and add docs by @whynowy in #2180
• chore: mono vertex validation by @whynowy in #2182
• fix: refine vertex/mvtx pod clean up logic by @whynowy in #2185
• chore(deps): bump http-proxy-middleware from 2.0.6 to 2.0.7 in /ui by @dependabot in #2188
• feat: source and sink implementation in Rust (blocking implementation) by @vigith in #2190
• feat:KafkaSource supports KafkaVersion modification by @qianbeibuzui in #2191
• doc: monovertex by @vigith in #2193
• chore: Batch ack requests and sink responses for better performance #163 by @yhl25 in #2194
• fix(docs): use manifests from main branch in quick-start by @whynowy in #2197
• chore: Load docker image from buildx runner by @BulkBeing in #2198
• chore: minor fixes for rust source and sink by @yhl25 in #2201
• feat: block isbsvc deleting when there is linked pipeline by @whynowy in #2202
• chore: shutdown when we see non retryable udf errors by @yhl25 in #2204
• chore: re-enable e2e tests by @kohlisid in #2210
• feat: metrics visualiser for mono vertex by @veds-g in #2195
• chore: update version compatibility matrix by @KeranYang in #2212

New Contributors ❤️

• @mdwarne1 made their first contribution in #1934
• @lanedd made their first contribution in #2022
• @th0ger made their first contribution in #2168
• @qianbeibuzui made their first contribution in #2191

Full Changelog: v1.3.0...v1.4.0