Merge branch 'ceph:main' into main

.github/CODEOWNERS

@@ -47,18 +47,19 @@ README*                                         @ceph/doc-writers
 *.rst                                           @ceph/doc-writers
 
 # core
-/doc/man/8/ceph-authtool.rst                    @ceph/core
-/doc/man/8/ceph-conf.rst                        @ceph/core
-/doc/man/8/ceph-create-keys.rst                 @ceph/core
-/doc/man/8/ceph-kvstore-tool.rst                @ceph/core
-/doc/man/8/ceph-mon.rst                         @ceph/core
-/doc/man/8/ceph-objectstore-tool.rst            @ceph/core
-/doc/man/8/ceph-osd.rst                         @ceph/core
-/doc/man/8/ceph.rst                             @ceph/core
-/doc/man/8/crushtool.rst                        @ceph/core
-/doc/man/8/monmaptool.rst                       @ceph/core
-/doc/man/8/rados.rst                            @ceph/core
-/doc/rados                                      @ceph/core
+/doc/man/8/ceph-authtool.rst                    @ceph/core @ceph/doc-writers
+/doc/man/8/ceph-conf.rst                        @ceph/core @ceph/doc-writers
+/doc/man/8/ceph-create-keys.rst                 @ceph/core @ceph/doc-writers
+/doc/man/8/ceph-kvstore-tool.rst                @ceph/core @ceph/doc-writers
+/doc/man/8/ceph-mon.rst                         @ceph/core @ceph/doc-writers
+/doc/man/8/ceph-objectstore-tool.rst            @ceph/core @ceph/doc-writers
+/doc/man/8/ceph-osd.rst                         @ceph/core @ceph/doc-writers
+/doc/man/8/ceph.rst                             @ceph/core @ceph/doc-writers
+/doc/man/8/crushtool.rst                        @ceph/core @ceph/doc-writers
+/doc/man/8/monmaptool.rst                       @ceph/core @ceph/doc-writers
+/doc/man/8/rados.rst                            @ceph/core @ceph/doc-writers
+/doc/rados                                      @ceph/core @ceph/doc-writers
+/examples/librados                              @ceph/core
 /qa/standalone                                  @ceph/core
 /qa/suites/rados                                @ceph/core
 /qa/workunits/erasure-code                      @ceph/core
@@ -83,11 +84,13 @@ README*                                         @ceph/doc-writers
 /src/test/osd                                   @ceph/core
 
 # rbd
-/doc/dev/rbd*                                   @ceph/rbd
-/doc/man/8/ceph-rbdnamer.rst                    @ceph/rbd
-/doc/man/8/rbd*                                 @ceph/rbd
-/doc/rbd                                        @ceph/rbd
-/doc/start/quick-rbd.rst                        @ceph/rbd
+/doc/dev/rbd*                                   @ceph/rbd @ceph/doc-writers
+/doc/man/8/ceph-rbdnamer.rst                    @ceph/rbd @ceph/doc-writers
+/doc/man/8/rbd*                                 @ceph/rbd @ceph/doc-writers
+/doc/rbd                                        @ceph/rbd @ceph/doc-writers
+/doc/start/quick-rbd.rst                        @ceph/rbd @ceph/doc-writers
+/examples/librbd                                @ceph/rbd
+/examples/rbd-replay                            @ceph/rbd
 /qa/rbd                                         @ceph/rbd
 /qa/run_xfstests*                               @ceph/rbd
 /qa/suites/krbd                                 @ceph/rbd
@@ -101,9 +104,10 @@ README*                                         @ceph/doc-writers
 /qa/workunits/cls/test_cls_lock.sh              @ceph/rbd
 /qa/workunits/cls/test_cls_rbd.sh               @ceph/rbd
 /qa/workunits/rbd                               @ceph/rbd
+/qa/workunits/windows                           @ceph/rbd
 /src/ceph-rbdnamer                              @ceph/rbd
 /src/cls/journal                                @ceph/rbd
-/src/cls/lock                                   @ceph/rbd
+/src/cls/lock                                   @ceph/rbd @ceph/rgw
 /src/cls/rbd                                    @ceph/rbd
 /src/common/options/rbd*                        @ceph/rbd
 /src/etc-rbdmap                                 @ceph/rbd
@@ -119,7 +123,7 @@ README*                                         @ceph/doc-writers
 /src/test/cli/rbd                               @ceph/rbd
 /src/test/cli-integration/rbd                   @ceph/rbd
 /src/test/cls_journal                           @ceph/rbd
-/src/test/cls_lock                              @ceph/rbd
+/src/test/cls_lock                              @ceph/rbd @ceph/rgw
 /src/test/cls_rbd                               @ceph/rbd
 /src/test/journal                               @ceph/rbd
 /src/test/librbd                                @ceph/rbd
@@ -130,3 +134,45 @@ README*                                         @ceph/doc-writers
 /src/tools/rbd*                                 @ceph/rbd
 /systemd/rbdmap.service.in                      @ceph/rbd
 /udev/50-rbd.rules                              @ceph/rbd
+
+# rgw
+/doc/dev/radosgw                                @ceph/rgw @ceph/doc-writers
+/doc/radosgw                                    @ceph/rgw @ceph/doc-writers
+/examples/rgw                                   @ceph/rgw
+/qa/rgw                                         @ceph/rgw
+/qa/suites/rgw                                  @ceph/rgw
+/qa/tasks/barbican.py                           @ceph/rgw
+/qa/tasks/kafka.py                              @ceph/rgw
+/qa/tasks/keycloak.py                           @ceph/rgw
+/qa/tasks/keystone.py                           @ceph/rgw
+/qa/tasks/pykmip.py                             @ceph/rgw
+/qa/tasks/rabbitmq.py                           @ceph/rgw
+/qa/tasks/rgw*                                  @ceph/rgw
+/qa/tasks/s3*                                   @ceph/rgw
+/qa/tasks/tempest.py                            @ceph/rgw
+/qa/tasks/vault.py                              @ceph/rgw
+/qa/workunits/rgw                               @ceph/rgw
+/src/cls/2pc_queue                              @ceph/rgw
+/src/cls/cmpomap                                @ceph/rgw
+/src/cls/fifo                                   @ceph/rgw
+/src/cls/log                                    @ceph/rgw
+/src/cls/otp                                    @ceph/rgw
+/src/cls/queue                                  @ceph/rgw
+/src/cls/refcount                               @ceph/rgw
+/src/cls/rgw                                    @ceph/rgw
+/src/cls/rgw_gc                                 @ceph/rgw
+/src/cls/user                                   @ceph/rgw
+/src/cls/version                                @ceph/rgw
+/src/rgw                                        @ceph/rgw
+/src/s3select                                   @ceph/rgw
+/src/spawn                                      @ceph/rgw
+/src/test/cls_2pc_queue                         @ceph/rgw
+/src/test/cls_cmpomap                           @ceph/rgw
+/src/test/cls_otp                               @ceph/rgw
+/src/test/cls_queue                             @ceph/rgw
+/src/test/cls_refcount                          @ceph/rgw
+/src/test/cls_rgw                               @ceph/rgw
+/src/test/cls_rgw_gc                            @ceph/rgw
+/src/test/cls_version                           @ceph/rgw
+/src/test/rgw                                   @ceph/rgw
+/src/test/test_rgw*                             @ceph/rgw

.github/labeler.yml

@@ -146,6 +146,7 @@ core:
   - src/tools/monmaptool.cc
   - src/tools/osdmaptool.cc
   - src/tools/rados/**
+  - src/test/librados/**
   - src/test/osd/**
 
 crimson:
@@ -273,4 +274,13 @@ monitoring:
   - src/pybind/mgr/dashboard/ci/check_grafana_dashboards.py
   - src/pybind/mgr/prometheus/**
   - monitoring/**  
-
+
+telemetry:
+  - doc/mgr/telemetry.rst
+  - qa/suites/upgrade/telemetry-upgrade/**
+  - qa/workunits/test_telemetry_pacific.sh
+  - qa/workunits/test_telemetry_pacific_x.sh
+  - qa/workunits/test_telemetry_quincy.sh
+  - qa/workunits/test_telemetry_quincy_x.sh
+  - src/pybind/mgr/telemetry/**
+  - src/telemetry/**

.github/milestone.yml

@@ -4,3 +4,4 @@ base-branch:
   - "(octopus)"
   - "(pacific)"
   - "(quincy)"
+  - "(reef)"

.github/workflows/pr-triage.yml

@@ -14,8 +14,8 @@ jobs:
           sync-labels: ''
           repo-token: "${{ secrets.GITHUB_TOKEN }}"
       - name: Assign to Dashboard project
-        # https://github.com/marketplace/actions/add-to-github-projects?version=v0.3.0
-        uses: actions/add-to-project@a9f041ddd462ed185893ea1024cec954f50dbe42
+        # https://github.com/marketplace/actions/add-to-github-projects?version=v0.4.1
+        uses: actions/add-to-project@4756e6330fe1e0a736690d3cfd9f11c9399c2ed4
         with:
           project-url: https://github.com/orgs/ceph/projects/2
           github-token: ${{ secrets.GITHUB_TOKEN }}

.githubmap

@@ -152,3 +152,6 @@ benhanokh Gabriel Benhanokh <[email protected]>
 kamoltat Kamoltat Sirivadhna <[email protected]>
 anthonyeleven Anthony D Atri <[email protected]>
 petrutlucian94 Lucian Petrut <[email protected]>
+dparmar18 Dhairya Parmar <[email protected]>
+nmshelke Nikhilkumar Shelke <[email protected]>
+neesingh-rh Neeraj Pratap Singh <[email protected]>

.gitmodules

@@ -38,9 +38,6 @@
 [submodule "src/blkin"]
 	path = src/blkin
 	url = https://github.com/ceph/blkin
-[submodule "src/rapidjson"]
-	path = src/rapidjson
-	url = https://github.com/ceph/rapidjson
 [submodule "src/dmclock"]
 	path = src/dmclock
 	url = https://github.com/ceph/dmclock.git

.readthedocs.yml

@@ -1,10 +1,6 @@
 ---
 # Read the Docs configuration file
 # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
-#
-# The pre_build command checks if we're building a named branch (i.e., not a PR).
-# If so, check out doc/releases from the main branch before building so
-# it's always up to date on docs.ceph.com/en/*.
 
 version: 2
 formats: []
@@ -15,9 +11,6 @@ build:
   apt_packages:
     - ditaa
     - graphviz
-  jobs:
-    pre_build:
-      - bash admin/rtd-checkout-main
 python:
   install:
     - requirements: admin/doc-requirements.txt

CMakeLists.txt

@@ -439,6 +439,8 @@ option(WITH_RADOSGW_DBSTORE "DBStore backend for Rados Gateway" ON)
 option(WITH_RADOSGW_MOTR "CORTX-Motr backend for Rados Gateway" OFF)
 option(WITH_RADOSGW_DAOS "DAOS backend for RADOS Gateway" OFF)
 option(WITH_RADOSGW_SELECT_PARQUET "Support for s3 select on parquet objects" ON)
+option(WITH_RADOSGW_ARROW_FLIGHT "Build arrow flight when not using system-provided arrow" OFF)
+option(WITH_RADOSGW_BACKTRACE_LOGGING "Enable backtraces in rgw logs" OFF)
 
 option(WITH_SYSTEM_ARROW "Use system-provided arrow" OFF)
 option(WITH_SYSTEM_UTF8PROC "Use system-provided utf8proc" OFF)

PendingReleaseNotes

@@ -1,5 +1,9 @@
 >=18.0.0
 
+* The RGW policy parser now rejects unknown principals by default. If you are
+  mirroring policies between RGW and AWS, you may wish to set
+  "rgw policy reject invalid principals" to "false". This affects only newly set
+  policies, not policies that are already in place.
 * RGW's default backend for `rgw_enable_ops_log` changed from RADOS to file.
   The default value of `rgw_ops_log_rados` is now false, and `rgw_ops_log_file_path`
   defaults to "/var/log/ceph/ops-log-$cluster-$name.log".
@@ -72,6 +76,69 @@
   namespaces was added to RBD in Nautilus 14.2.0 and it has been possible to
   map and unmap images in namespaces using the `image-spec` syntax since then
   but the corresponding option available in most other commands was missing.
+* RGW: Compression is now supported for objects uploaded with Server-Side Encryption.
+  When both are enabled, compression is applied before encryption.
+* RGW: the "pubsub" functionality for storing bucket notifications inside Ceph
+  is removed. Together with it, the "pubsub" zone should not be used anymore.
+  The REST operations, as well as radosgw-admin commands for manipulating
+  subscriptions, as well as fetching and acking the notifications are removed 
+  as well.
+  In case that the endpoint to which the notifications are sent maybe down or 
+  disconnected, it is recommended to use persistent notifications to guarantee 
+  the delivery of the notifications. In case the system that consumes the 
+  notifications needs to pull them (instead of the notifications be pushed 
+  to it), an external message bus (e.g. rabbitmq, Kafka) should be used for 
+  that purpose.
+* RGW: The serialized format of notification and topics has changed, so that 
+  new/updated topics will be unreadable by old RGWs. We recommend completing 
+  the RGW upgrades before creating or modifying any notification topics.
+* RBD: Trailing newline in passphrase files (`<passphrase-file>` argument in
+  `rbd encryption format` command and `--encryption-passphrase-file` option
+  in other commands) is no longer stripped.
+* RBD: Support for layered client-side encryption is added.  Cloned images
+  can now be encrypted each with its own encryption format and passphrase,
+  potentially different from that of the parent image.  The efficient
+  copy-on-write semantics intrinsic to unformatted (regular) cloned images
+  are retained.
+* CEPHFS: Rename the `mds_max_retries_on_remount_failure` option to
+  `client_max_retries_on_remount_failure` and move it from mds.yaml.in to
+  mds-client.yaml.in because this option was only used by MDS client from its
+  birth.
+* The `perf dump` and `perf schema` commands are deprecated in favor of new
+  `counter dump` and `counter schema` commands. These new commands add support
+  for labeled perf counters and also emit existing unlabeled perf counters. Some
+  unlabeled perf counters became labeled in this release, with more to follow in
+  future releases; such converted perf counters are no longer emitted by the
+  `perf dump` and `perf schema` commands.
+* `ceph mgr dump` command now outputs `last_failure_osd_epoch` and
+  `active_clients` fields at the top level.  Previously, these fields were
+  output under `always_on_modules` field.
+* `ceph mgr dump` command now displays the name of the mgr module that
+  registered a RADOS client in the `name` field added to elements of the
+  `active_clients` array. Previously, only the address of a module's RADOS
+  client was shown in the `active_clients` array.
+* RBD: All rbd-mirror daemon perf counters became labeled and as such are now
+  emitted only by the new `counter dump` and `counter schema` commands.  As part
+  of the conversion, many also got renamed to better disambiguate journal-based
+  and snapshot-based mirroring.
+* RBD: list-watchers C++ API (`Image::list_watchers`) now clears the passed
+  `std::list` before potentially appending to it, aligning with the semantics
+  of the corresponding C API (`rbd_watchers_list`).
+* Telemetry: Users who are opted-in to telemetry can also opt-in to
+  participating in a leaderboard in the telemetry public
+  dashboards (https://telemetry-public.ceph.com/). Users can now also add a
+  description of the cluster to publicly appear in the leaderboard.
+  For more details, see:
+  https://docs.ceph.com/en/latest/mgr/telemetry/#leaderboard
+  See a sample report with `ceph telemetry preview`.
+  Opt-in to telemetry with `ceph telemetry on`.
+  Opt-in to the leaderboard with
+  `ceph config set mgr mgr/telemetry/leaderboard true`.
+  Add leaderboard description with:
+  `ceph config set mgr mgr/telemetry/leaderboard_description ‘Cluster description’`.
+* CEPHFS: After recovering a Ceph File System post following the disaster recovery
+  procedure, the recovered files under `lost+found` directory can now be deleted.
+* core: cache-tiering is now deprecated.
 
 >=17.2.1
 
@@ -119,16 +186,14 @@ of the feature, refer this link on how to perform it:
 https://docs.ceph.com/en/quincy/cephadm/upgrade/#staggered-upgrade
 Relevant tracker: https://tracker.ceph.com/issues/55715
 
-  Relevant tracker: https://tracker.ceph.com/issues/5614
-
-* Cephadm may now be configured to carry out CephFS MDS upgrades without
-reducing ``max_mds`` to 1. Previously, Cephadm would reduce ``max_mds`` to 1 to
-avoid having two active MDS modifying on-disk structures with new versions,
-communicating cross-version-incompatible messages, or other potential
-incompatibilities. This could be disruptive for large-scale CephFS deployments
-because the cluster cannot easily reduce active MDS daemons to 1.
-NOTE: Staggered upgrade of the mons/mgrs may be necessary to take advantage
-of the feature, refer this link on how to perform it:
-https://docs.ceph.com/en/quincy/cephadm/upgrade/#staggered-upgrade
-Relevant tracker: https://tracker.ceph.com/issues/55715
+* Introduced a new file system flag `refuse_client_session` that can be set using the
+`fs set` command. This flag allows blocking any incoming session
+request from client(s). This can be useful during some recovery situations
+where it's desirable to bring MDS up but have no client workload.
+Relevant tracker: https://tracker.ceph.com/issues/57090
 
+* New MDSMap field `max_xattr_size` which can be set using the `fs set` command.
+  This MDSMap field allows to configure the maximum size allowed for the full
+  key/value set for a filesystem extended attributes.  It effectively replaces
+  the old per-MDS `max_xattr_pairs_size` setting, which is now dropped.
+  Relevant tracker: https://tracker.ceph.com/issues/55725

README.md

@@ -29,7 +29,7 @@ You can clone from github with
 
 or, if you are not a github user,
 
-	git clone git://github.com/ceph/ceph
+	git clone https://github.com/ceph/ceph.git
 
 Ceph contains many git submodules that need to be checked out with