Leverage row tracking to reduce memory overhead of log replay

[scovich]

Please describe why this is necessary.

Today, log replay remembers every file it has ever encountered until log replay is complete.

However, if row tracking is enabled, the Delta spec mandates that:

• Writers must set the defaultRowCommitVersion field in new add actions to the version number of the log entry containing the add action.
• Writers must set the defaultRowCommitVersion field in recommitted and checkpointed add actions and remove actions to the defaultRowCommitVersion of the last committed add action with the same path.

The first condition is a base case, and the second condition is a recursive case, and together they guarantee that all file actions correctly identify the commit that first added whatever physical file they reference.

Describe the functionality you are proposing.

We should update log replay to leverage row tracking, when available.

For example, if log replay is processing commit 100, and we encounter add.defaultRowCommitVersion=100, then we know we'll never see that file again and can dedup by calling seen.remove(&key) instead of seen.contains(&key).

Doing so can reduce memory overhead significantly when replaying a log with lots of high-churn commits, because the add+remove pairs that churned can be dropped from the seen set as soon as we see the original add that created the file.

The only gotcha is, I don't think we support row tracking yet?

Additional context

No response