Maximilian Bucher on 19 Feb 2024 15:49:59
Issue Definition: Error Message: "Delta table 'zsd_ship' has atleast '100' transaction logs, since last checkpoint. For performance reasons, it is recommended to regularly checkpoint the delta table more frequently than every '100' transactions. As a workaround..."
Observation: Going through the notes provided by my previous colleague, I could understand that you have encountered a problem with a Delta table that has not been checkpointed in at least 100 transactions. The error “DeltaTableIsInfrequentlyCheckpointed” has been received, indicating the need for more frequent checkpointing. You have been advised to use SQL or Spark as a workaround to obtain the table schema.
Resolution: - As we were checking on your issue, we tried to reproduce the issue from our end, we could see that for new tables, there is no such error reflecting at our end. You have confirmed that your tables were loading properly after deleting and re running the same. It seems like only "older" tables of 3 months or so are affected (probably the 100 day mark/ load cycle mark).
However, it is not sustainable in the future to delete entire lakehouses regularly in our bronze layer to be able to load data properly. Maybe you can look at the data flow & pipelines again to load it properly.
Best,
Max