feat: allow replacement of entire datafile when the schema lines up correctly

[chebbyChefNEQ]

For internal design doc see notion or ping me

What this PR implements

Plan of attack:

• This PR: basic functionality, i.e. when there is no conflict calling this tx should just work
• Next PR: implement more fine-grained conflict resolution
• Potential future PR (when time permits): Allow partial replacement of a datafile. This can be done by "dropping" column indice in a datafile, thereby dropping the column in favor of another

TODO:

• proto definition of the new transaction
• simple rust tests
• test error handling
• PR desc
• python tests
• implement conflict detection

[chebbyChefNEQ]

changed (1), (2), (3) to emoji so the table width align better

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 109 lines in your changes missing coverage. Please review.

Project coverage is 78.77%. Comparing base (a7c5216) to head (c615eb0).

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/transaction.rs	0.00%	109 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3408      +/-   ##
==========================================
- Coverage   78.86%   78.77%   -0.10%     
==========================================
  Files         250      250              
  Lines       91602    91711     +109     
  Branches    91602    91711     +109     
==========================================
  Hits        72244    72244              
- Misses      16395    16503     +108     
- Partials     2963     2964       +1     

Flag	Coverage Δ
unittests	78.77% <0.00%> (-0.10%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[westonpace]

YOLO approval if we can add that warning. Will come back and think about this more later.

[westonpace]

I think we will eventually want some rust-level tests of data replacement but could potentially be done in a future PR.

[chebbyChefNEQ]

Those are in dataset.rs but github is folding the file for some reason :(

[wjones127]

I'd like to see one more test case to verify you tombstone fields correctly:

1. write a, b
2. Column replacement on b
3. Assert that fragment has to data files:
   • file 1 has column id [0, -2] (-2 is the column id tombstone)
   • file 2 has column ids [1]

See this merge insert test for an example of where else this happens:

lance/rust/lance/src/dataset/write/merge_insert.rs

Lines 2046 to 2053 in c73d717

	let has_added_files = |frag: &Fragment| {
	assert_eq!(frag.files.len(), 2);
	let data_files = &frag.files;
	// Updated columns should be only columns in new data files
	// -2 field ids are tombstoned.
	assert_eq!(&data_files[0].fields, &[0, -2, -2]);
	assert_eq!(&data_files[1].fields, &[2, 1]);
	};