[data lineage] Showing the user what tables & corresponding columns got changed while loading into the database.

[redicane]

DLT will change any column and table name to lower case and will also replace spaces with '_'. As to not confuse the user of the pipeline, it might be helpful to keep track and show these changes after the pipeline has finished loading data into the database. This could be done in load info (showing just how many name changes occurred), logging info every time there is a name change and this information should also be in the schema after the pipeline has finished loading.
Additionally if a column only has NULL values for all rows, it will be dropped. I think this should also be shown to a user. Basically any change of this sort where a user expects thing A to be loaded and thing B (or nothing) gets loaded should be shown to the user.

[rudolfix]

@redicane

Additionally if a column only has NULL values for all rows, it will be dropped
then it is a bug... I will create a ticket
this information should also be in the schema after the pipeline has finished loading
I'm planning to add lineage information to the schema. I'll attach this ticket to it