Matthias Wong on 09 Oct 2023 11:15:06
Great that we have Delta logs which theoretically allow some sort of time travel on a table. However, the warehouse Delta table does not expose the Delta logs. This means that we have no ability to see what records have been deleted from load to load.
If we have Delta tables, why not expose them in the warehouse through some sort of syntax similar to the for system_time all, or better yet, allow people to access the log as if they were SQL tables, in the same way that SQL Server temporal tables do?
There are multiple use cases. The chief one is incremental processing. In our SQL warehouse, we take advantage of the temporal tables to keep track of what has been inserted/deleted/updated. This allows us to know how to incrementally process down stream tables. many of our stored procedures query the temporal history table for this purpose. If we want to migrate our warehouse to Fabric, we would need to migrate those. But we cannot at the moment because the Fabric warehouse does not support temporal tables and hence no place where we keep track of deletes. This could be fixed if we can expose the Delta logs via normal SQL.
Ambika Jagadish (administrator) on 22 Jan 2024 18:10:20
Thanks for your feedback. We have now prioritized the ability to Time travel from our backlog. If you have any detailed feedback or specific insights about this feature, please don't hesitate to get in touch with me.
- Comments (1)
RE: Support Temporal tables/Expose delta logs in warehouse
We need this! In Databricks there is a good SQL Syntax for time travel with detla logs