Resolve deletion vectors to find inserted and removed rows for CDF #568

OussamaSaoudi-db · 2024-12-06T07:05:41Z

What changes are proposed in this pull request?

This adds a function resolve_scan_file_dv which converts CdfScanFile to ResolvedCdfScanFile by reading deletion vectors, and reconciling paired add/remove deletion vectors.

ResolvedCdfScanFile is a struct that holds a CdfScanFile with its selection vector. It is guaranteed that the scan_type of CdfScanFile will match the _change_type of all the selected rows in its path if it is type add or remove.

treemap_to_bools is split into deletion_treemap_to_bools and selection_treemap_to_bools, which generate selection vectors from either a deletion treemap or selection treemap respectively.

get_selection_vector now depends on a new helper function get_treemap.

How was this change tested?

The following are tested:

adding a deletion vector to a file that did not have one
Removing a deletion vector from a file that had one
restoring a subset of deleted rows
deleting a subset of remaining rows after a deletion already happened
Deleting some rows and restoring others
Remove_dv on type cdc and remove fails
Add, cdc, and rm with no deletion vector return a None selection vector.
ensure that the correct selection vector is generated for selection_treemap_to_bool

codecov · 2024-12-06T07:10:29Z

Codecov Report

Attention: Patch coverage is 98.69452% with 5 lines in your changes missing coverage. Please review.

Project coverage is 82.85%. Comparing base (ed714c5) to head (5767bf1).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
kernel/src/table_changes/resolve_dvs.rs	99.40%	0 Missing and 2 partials ⚠️
kernel/src/actions/deletion_vector.rs	96.42%	1 Missing ⚠️
kernel/src/scan/mod.rs	0.00%	1 Missing ⚠️
kernel/src/scan/state.rs	93.75%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #568      +/-   ##
==========================================
+ Coverage   82.45%   82.85%   +0.39%     
==========================================
  Files          72       73       +1     
  Lines       16054    16426     +372     
  Branches    16054    16426     +372     
==========================================
+ Hits        13238    13609     +371     
+ Misses       2173     2172       -1     
- Partials      643      645       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

scovich · 2024-12-06T21:31:34Z

kernel/src/scan/state.rs

-        let dv_treemap = self
-            .deletion_vector
+    ) -> DeltaResult<Option<RoaringTreemap>> {
+        self.deletion_vector
            .as_ref()
            .map(|dv_descriptor| {
                let fs_client = engine.get_file_system_client();
                dv_descriptor.read(fs_client, table_root)


Just double checking: dv_descriptor.read takes care of slicing the DV when offset and size cover a subset of the DV file?

I don't believe it does. It just handles reading the on-disk file and turning it into a roaring treemap. Something at a higher level needs to slice it up if needed.

scovich · 2024-12-06T21:32:48Z

kernel/src/table_changes/data_read.rs

+///    must be [`CDFScanFileType::Add`]. In this case, both the add and remove deletion vectors are
+///    read if they exist. Then, we find the set of rows in the scan file that have been added, and


How do we find the add DV? I thought a previous PR in the stack only kept the remove DV?

update: the add dv is directly attached to the CDFScanFile

But see also #546 (comment)

Remove_dv is now part of CDFScanFile, and I made the changes to the visitor that you recommended.

scovich · 2024-12-06T21:35:19Z

kernel/src/table_changes/data_read.rs

+                        .into_iter()
+                        .map(Not::not)
+                        .collect();
+                    println!("got dv: {:?} for type {:?}", added_dv, out_type);


TODO: remove or turn to e.g. debug!

kernel/src/table_changes/data_read.rs

scovich · 2024-12-06T22:16:54Z

kernel/src/table_changes/data_read.rs

+        (_, Some(_)) => Err(Error::generic(
+            "Remove DV should only match to an add action!",
+        )),


This is a great reason to embed the remove dv info in the scan file alongside the add dv info...

What you're saying is instead of having UnresolvedCdfScanFile, we move remove_dvs directly into CdfScanFile?

I'll go ahead and implement that, and you lmk

scovich · 2024-12-06T22:33:59Z

kernel/src/table_changes/data_read.rs

+    } = scan_file;
+    let paired_rm_dv = remove_dvs.get(&scan_file.path);
+    match (&scan_file.scan_type, paired_rm_dv) {
+        (CDFScanFileType::Add, Some(rm_dv)) => {


If we embedded the rm_dv in the scan_file itself, then this whole match can go away.
We just need the validation code that asserts:

cdc cannot have a remove

remove cannot have an add

(add can have one or both)

Implemented as part of the simplification.

scovich · 2024-12-06T22:47:42Z

kernel/src/table_changes/data_read.rs

+                    println!("got dv: {:?} for type {:?}", added_dv, out_type);
+                    scan_file.scan_type = out_type;
+
+                    Either::Right(iter::once(ResolvedCDFScanFile {


NOTE: Either<iter::Empty<T>, iter::Once<T>> is just a complicated way to emulate Option<T>...

scovich · 2024-12-06T23:21:04Z

kernel/src/table_changes/data_read.rs

+                if selection_treemap.is_empty() {
+                    // Nothing has been selected, we do not read this data file
+                    Either::Left(iter::empty())


Problem: The DV for a file can be shorter than the file's row count, with missing entries treated as if they were true (= keep). See doc comments for ScanResult::full_mask. My tired Friday pm brain is struggling to grok whether computing a symmetric difference for a remove DV breaks that assumption? But it's definitely not safe to skip file just because its raw physical DV is empty (current code in PR), or even just because all DV entries seem to be false (because every DV has an implied padding of true entries past-end, out to the data length which we won't know until we have an EngineData to work with).

Hmm I may be skipping something, but here's what I know:

This skipping here only happens for add/remove pairs of deletion vectors.

the treemap here represents the selected rows based on the dv. So far in our code we've treated treemaps exclusively as deletion bitmaps. Our dv resolving results in us generating a selection treemap. Hence if the selection treemap no bit set, everything would be false => not selected/skipped

I still think should be safe to skip?

I've added a comment explaining why the treemap is treated as a selection vector .

I think the issue is, let's say your data is:

['a', 'b', 'c'] and your DV is:
[0] // row 0 is deleted

the implied DV is [false, true, true].

This matters when you're just using DVs "as is". However, here you're only look for rows that changed, so if there are a bunch of indices not mentioned in either treemap, they did not change, so should not be returned here.

I think your comment below should work through with the indices first, since that's what the treemap has, and the set difference operations to show the resulting indicies, and then the conversion to boolean selection vectors, to ensure things work as we expect.

Re it being definitely wrong if the vector is empty, that's certainly true for a standard scan, but I think here it's potentially correct if it means that nothing changed between the two vectors.

Maybe to add on. All the important operations here occur where things are just treemaps. Only once we've settled on the final indicies which have changed do we convert to a boolean vector, which is the same as in our scan code. But, because we're using selection vectors here, we don't need to worry about extending, because the implied extra stuff is masked out.

We should probably put a bit // ===== IMPORTANT ===== comment somewhere that calls out that a CDF scan uses a selection vector while a "normal" scan uses a deletion one.

I hopefully explained it more thoroughly this time. Added some notation to be clear at each point what type of treemap we're working with. Also split up selection vector generation into its own section.

scovich · 2024-12-06T23:31:56Z

kernel/src/table_changes/data_read.rs

+            // Helper function to convert a treemap to a [`ResolvedCdfScanFile`]. The `scan_type`
+            // of the [`ResolvedCdfScanFile`] is set to `out_type` This returns an empty iterator
+            // if nothing is selected.
+            fn treemap_to_iter(


We map scan file to iter because we could have 0 (filtered out) 1 (normal) or 2 (adds+removes) results? Which in turn requires two Either so we can capture four possible cases? I wonder if it would be simpler to always produce the iterators and let the caller flatten() them?

Something like:

let add_dv = scan_file.add_dv.dv_info.get_treemap(...)?: let rm_dv = scan_file.rm_dv.dv_info.get_treemap(...)?: let (add_dv, rm_dv) = match (scan_file.add_dv, scan_file.rm_dv, scan_file.scan_type) { (Some(_), _, CDFScanFileType::Remove) => { return Err(... remove cannot have add DV ...); } (_, Some(_), CDFScanFileType::Cdc) => { return Err(... cdc cannot have remove DV ...); } (Some(add_dv), Some(rm_dv), _) => { // Take the symmetric difference so we don't double count rows (Some(&add_dv - &rm_dv), Some(&rm_dv - &add_dv)) } (add_dv, None, CDFScanFileType::Add | CDFScanFileType::Cdc) => { (Some(add_dv.get_or_else(|| Default::default())), None) } (None, rm_dv, CDFScanFileType::Remove) => { (None, Some(rm_dv.get_or_else(|| Default::default()))) } }; let adds = add_dv.map(treemap_to_bools) .map(|sel| ResolvedCDFScanFile { ... }); let removes = rm_dv.map(treemap_to_bools) .map(|sel| sel.into_iter().map(Not::not).collect()) .map(|sel| ResolvedCDFScanFile { ... }); [adds, removes].into_iter().flatten()

... except I think I got the selection vector inversion bit wrong; it should only be inverted if it was a remove DV on an Add action.

Regarding treemap_to_bools: I noticed that it initializes the vec with all true be default, then sets elements to false. I observed that when I initialized the vec with all false and only selected elements to true, it's significantly faster. I found a post explaining why this may be. Zeroed out pages seem to work well with the kernel :)

Given that the selection vector is O(#rows) it may worth avoiding the overhead of iterating over it again with Not::not and take advantage of the zeroed out allocation.

I'm assuming here #rows could be very large.

I came across a weird case 🤔. The way you have it, we seem to have

CdfScanFile { add_dv: DvInfo, rm_dv: DvInfo ... }

where the deletion vector in add_dv/rm_dv could be None.

The problem comes in when differentiating the following cases:

[Remove(path=A, dv=None), Add(path=A, dv=X)]

[Add(path=A, dv=X)].

This has got me thinking whether case 2 would ever happen. Suppose we want to handle it:

I think both of these would fall into this case:

(add_dv, None, CDFScanFileType::Add | CDFScanFileType::Cdc) => { (Some(add_dv.get_or_else(|| Default::default())), None) }

So I'm instead electing to go with something like this:

CdfScanFile { deletion_vector: DvInfo, remove_dv: Option<DvInfo> ... }

so that the cases become clear:

remove_dv = Some(DvInfo( deletion_vector: None))

remove_dv = None

Yes, I can totally imagine zero-initialized is more efficient for many reasons (tho correctness trumps performance, if there is friction between the two).

Your newer approach to handling add vs. rm DV makes a lot of sense, thanks!

nicklan

I think ryan's proposed simplification makes sense. otherwise looks pretty good. just re-request when that's resolved.

… dv resolution

…lection treemap

scovich

Few nits, but looks great

scovich · 2024-12-09T17:05:44Z

kernel/src/scan/state.rs

+        DvInfo {
+            deletion_vector: Some(deletion_vector),
+        }


clippy wrangling nit:

Suggested change

DvInfo {

deletion_vector: Some(deletion_vector),

}

let deletion_vector = Some(deletion_vector);

DvInfo { deletion_vector }

kernel/src/table_changes/resolve_dvs.rs

scovich · 2024-12-09T17:09:11Z

kernel/src/table_changes/resolve_dvs.rs

+            //
+            // The result of this commit is:
+            // - row 0 is restored
+            // - row 1 is unchanged


Suggested change

// - row 1 is unchanged

// - row 1 is unchanged (previously deleted)

kernel/src/table_changes/resolve_dvs.rs

scovich · 2024-12-09T17:49:06Z

kernel/src/table_changes/resolve_dvs.rs

+    let rm_scan_file = CdfScanFile {
+        scan_type: CdfScanFileType::Remove,
+        ..scan_file.clone()


nit: This only gets used if we have Some rm_dv? Maybe rust is smart enough to delay creating it until we actually use it?

Also, the add/rm paths below are the ~same code duplicated. Maybe we could solve both issues in one swoop with a helper method?

let resolve = |dv, scan_file_fn| dv.map(treemap_to_bools).map(|sv| ResolvedCdfScanFile { scan_file: scan_file_fn(), sselection_vector: (!sv.is_emtpy()).then_some(sv), }); let removes = resolve(rm_dv, || CdfScanFile { ... }); let adds = resolve(add_dv, || scan_file);

Hmm rust has a tricky type system. One of them is Fn() -> CdfScanFile while the other one is FnOnce. So I have to do a clone() in the ||scan_file.clone() closure to make it Fn. Also the clone may be optimized by the compiler, but I'm not sure.

I implemented something that I think lazily constructs the rm scan file and helps a bit with the duplication.

If you have tips on viewing optimizations for rust, please let me know! I'm familiar with GodBolt, but I feel like assembly is too low level for the optimizations to be clear.

Oh, you probably just needed to do move || scan_file?

Yeah the problem is that move || scan_file is FnOnce, while the other one is Fn. move | | scan_file is FnOnce because we can only return the captured owned value once. Meanwhile || CdfScanFile { ... } clones its reference to scan_file so it can be called repeatedly. Thus it's Fn

We could make the closure take scan_file_fn: &dyn Fn() -> CdfScanFile, but then we'd have to use | | scan_file.clone() for the add case.

let resolve = |dv: Option<RoaringTreemap>, scan_file_fn: &dyn Fn() -> CdfScanFile| { dv.map(treemap_to_bools).map(|sv| ResolvedCdfScanFile { scan_file: scan_file_fn(), selection_vector: (!sv.is_empty()).then_some(sv), }) }; let removes = resolve(rm_dv, &|| CdfScanFile { scan_type: CdfScanFileType::Remove, ..scan_file.clone() }); let adds = resolve(add_dv, &|| scan_file.clone());

I tried scan_file_fn: &dyn FnOnce() -> CdfScanFile but the compiler isn't happy because you need to consume ownership of the function to call a FnOnce

let resolve = |dv: Option<RoaringTreemap>, scan_file_fn: &dyn FnOnce() -> CdfScanFile| { dv.map(treemap_to_bools).map(|sv| ResolvedCdfScanFile { scan_file: scan_file_fn(), selection_vector: (!sv.is_empty()).then_some(sv), }) }; let removes = resolve(rm_dv, &|| CdfScanFile { scan_type: CdfScanFileType::Remove, ..scan_file.clone() }); let adds = resolve(add_dv, &|| scan_file); Ok([removes, adds].into_iter().flatten())

Moreover generics don't work with closures, so we'd need a proper function fn resolve<T>(dv: ..., scan_file_fn: T),

Perhaps there's a way to specify generic trait impls so that we monomorphize into two resolves: resolve::<impl FnOnce>, and resolve::<impl Fn> but by this point I felt like this added more lines and complexity than it was worth.

nicklan

lgtm! 🥳

nicklan · 2024-12-09T20:59:54Z

kernel/src/table_changes/resolve_dvs.rs

+///    read the deletion vector (if present), and each is converted into a [`ResolvedCdfScanFile`].
+///    No changes are made to the `scan_type`.
+#[allow(unused)]
+fn resolve_scan_file_dv(


it's nice that this actually ended up pretty short :)

zachschuermann

few comments but let's iterate on this!

zachschuermann · 2024-12-09T22:42:26Z

kernel/src/actions/deletion_vector.rs

@@ -402,6 +414,26 @@ mod tests {
        assert_eq!(bools, expected);
    }

+    #[test]
+    fn test_sv_to_bools() {


how expensive is this to run? if cheap, remove #[ignore] from the one above? if expensive should we ignore this one too? or actually if this is relatively expensive but we decide to keep it, maybe we should integrate it with the test above? (just check both apis in the same test?)

I found that it ran just fine on my machine, whereas the other one took a minute.

Giving an algorithmic complexity is hard to say tbh. I go into it in this comment, but the TLDR is that the zero-initialized large sv vector is fast to allocate. Just get a bunch of pages from the OS. The other one is slow because rust has to actually read the pages and set it to true.

I can disable it for consistency.

ohhh right i recall. how about we leave it enabled then and just leave that comment here in the code with it?

The difference is pretty staggering lol. 0.85s vs 40s on my machine!!!

yea let's not keep the slow one. kinda nice we got some coverage through the other path now tho :)

zachschuermann · 2024-12-09T22:56:25Z

kernel/src/table_changes/resolve_dvs.rs

+/// types of `CdfScanFile`s:
+/// 1. The first case is a [`CdfScanFile`] paired with a remove deletion vector. The `scan_type`
+///    must be [`CdfScanFileType::Add`]. In this case, both the add and remove deletion vectors are
+///    read if they exist. Then, we find the set of rows in the that have been added, and  rows that


typo? "we find the set of rows in the that have been added"

zachschuermann · 2024-12-09T23:02:00Z

kernel/src/table_changes/resolve_dvs.rs

+            // vector pairs, we generate selection treemaps.
+            //
+            // We use a motivating example to explain the deletion vector resolution. We read
+            // `rm_dv` and `add_dv`, and they are initialized to the empty map by default.


Suggested change

// `rm_dv` and `add_dv`, and they are initialized to the empty map by default.

// `rm_dv` and `add_dv`, and they are initialized to the empty map by default

zachschuermann · 2024-12-09T23:07:00Z

kernel/src/table_changes/resolve_dvs.rs

+            let adds = &rm_dv - &add_dv;
+            let removes = add_dv - rm_dv;


out of curiosity, do we know the time complexity of these operations?

Let n be size of LHS map, m be the size of RHS map. This is O(m * log(n) + n). This is a set difference, so it iterates over everything in the RHS (O(m)), and for each such element, removes it from the LHS O(log(n)). The adds case has to do a clone of the LHS, so we also add an O(n).

source

zachschuermann · 2024-12-09T23:11:48Z

kernel/src/table_changes/resolve_dvs.rs

+    let treemap_to_bools = if scan_file.remove_dv.is_some() {
+        selection_treemap_to_bools
+    } else {
+        deletion_treemap_to_bools
+    };


can we normalize above (i.e. everything to SV) and avoid this?

Unfortunately not. Treemaps are sets, so constructing the inverse would require that we create a new "full" treemap containing every index (ie the number of rows in the data file). Then we take the full treemap and subtract out the selection treemap.

I think in most cases the set of deleted rows is much smaller than the set of all rows. So constructing such a treemap would be a lot more expensive.

zachschuermann · 2024-12-09T23:16:52Z

kernel/src/table_changes/resolve_dvs.rs

+    let adds = add_dv
+        .map(treemap_to_bools)
+        .map(|sv| resolve(scan_file, sv));
+    Ok([removes, adds].into_iter().flatten())


wonder if the iterator is the right approach? we only ever return one or two items, correct? does the (future) callsite necessitate the iterator?

You're right that we only return one or two items.

The callsite doesn't necessitate it, but it works nicely with flattening:

.map(move |scan_file| { resolve_scan_file_dv(dv_engine_ref.as_ref(), &table_root, scan_file?) }) // Iterator-Result-Iterator

If you know of a better way tho, lmk 👀

zachschuermann · 2024-12-09T23:18:09Z

(and i probably need to take a closer look at tests)

github-actions bot assigned OussamaSaoudi-db Dec 6, 2024

github-actions bot added the breaking-change Change that will require a version bump label Dec 6, 2024

OussamaSaoudi-db requested review from scovich, nicklan and zachschuermann December 6, 2024 07:16

scan file implementation for cdf

bd9585d

scovich mentioned this pull request Dec 6, 2024

ScanFile expression and visitor for CDF #546

Merged

change naming and update documentation for cdf scan files

e2c2d90

OussamaSaoudi-db force-pushed the cdf_dv_resolve branch from 170268b to 5def6bd Compare December 6, 2024 22:51

fixup some naming

3db127f

OussamaSaoudi-db force-pushed the cdf_dv_resolve branch from 5def6bd to 6b133d4 Compare December 6, 2024 22:54

scovich reviewed Dec 6, 2024

View reviewed changes

OussamaSaoudi-db added 2 commits December 6, 2024 16:05

inline dvinfo creation

ec6e629

Change name of expression transform

74eed10

nicklan reviewed Dec 7, 2024

View reviewed changes

OussamaSaoudi-db added 6 commits December 7, 2024 15:16

Add test for null parttion columns

9fe16db

Improve testing for scan_file

8635e38

Change visitor, remove (un)resolved cdf scan file

a17f4b0

Only keep track of remove dv instead of hashmap

5fab00c

Fix remove dv

21553a4

patch rm_dv

08867f8

OussamaSaoudi-db force-pushed the cdf_dv_resolve branch from 22196d7 to 027a73a Compare December 8, 2024 19:27

OussamaSaoudi-db added 7 commits December 8, 2024 11:30

Update comment for CdfScanFileVisitor

1228107

Initial cdf read phase with deletion vector resolution

97a5790

lazily construct empty treemaps

4969e44

Change treemap handling in kernel, use selection vector, and simplify…

dcb17fa

… dv resolution

Rebase onto scan file

ebaf225

update doc comment for dvs

bd49142

fix comment

6287e6e

OussamaSaoudi-db added 5 commits December 8, 2024 12:22

prototype of add/rm dv in cdfscanfile

7fe4f4d

Use default for dv info

d4f95d5

shorten tests, check error cases

8a8f6bf

Rename to physical_to_logical, add comment explaining the deletion/se…

eeaabb0

…lection treemap

Add note about ordinary scans for dv resolution

fa13ade

OussamaSaoudi-db force-pushed the cdf_dv_resolve branch from 720171b to fa13ade Compare December 8, 2024 20:22

Add doc comment to treemap_to_bools_with

4dabdaf

OussamaSaoudi-db requested review from nicklan and scovich December 8, 2024 20:27

OussamaSaoudi-db added 4 commits December 8, 2024 13:29

Update resolve_scan_file_dv docs

7fcb531

Add test and docs

577b424

Add documentation

a970df1

Rename to resolve-dvs

5b70aab

OussamaSaoudi-db mentioned this pull request Dec 9, 2024

Helper methods for CDF Physical to Logical Transformation #579

Merged

Fixup naming and docs

3e0ead6

scovich approved these changes Dec 9, 2024

View reviewed changes

address pr comments

0b207c4

nicklan approved these changes Dec 9, 2024

View reviewed changes

Merge branch 'main' into cdf_dv_resolve

68a4fb1

zachschuermann approved these changes Dec 9, 2024

View reviewed changes

OussamaSaoudi-db added 3 commits December 9, 2024 16:28

Add test comment, ignore sv to bools test

3259769

remove ignore from test

2e9c29e

remove comma

5767bf1

OussamaSaoudi-db merged commit 5816620 into delta-io:main Dec 10, 2024
20 checks passed

OussamaSaoudi-db mentioned this pull request Dec 14, 2024

[tracking issue] Change Data Feed read path #440

Open

21 tasks

zachschuermann removed the breaking-change Change that will require a version bump label Dec 14, 2024

		/// must be [`CDFScanFileType::Add`]. In this case, both the add and remove deletion vectors are
		/// read if they exist. Then, we find the set of rows in the scan file that have been added, and

	// - row 1 is unchanged
	// - row 1 is unchanged (previously deleted)

	// `rm_dv` and `add_dv`, and they are initialized to the empty map by default.
	// `rm_dv` and `add_dv`, and they are initialized to the empty map by default

Resolve deletion vectors to find inserted and removed rows for CDF #568

Resolve deletion vectors to find inserted and removed rows for CDF #568

Conversation

OussamaSaoudi-db commented Dec 6, 2024 • edited Loading

What changes are proposed in this pull request?

How was this change tested?

codecov bot commented Dec 6, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OussamaSaoudi-db Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicklan left a comment

Choose a reason for hiding this comment

scovich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OussamaSaoudi-db Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicklan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zachschuermann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zachschuermann commented Dec 9, 2024

OussamaSaoudi-db commented Dec 6, 2024 •

edited

Loading

codecov bot commented Dec 6, 2024 •

edited

Loading

OussamaSaoudi-db Dec 9, 2024 •

edited

Loading

OussamaSaoudi-db Dec 9, 2024 •

edited

Loading