Derek Williams on 06 Jan 2024 03:39:33
Lakehouses, and more importantly the data inside them, are not recoverable if the the lakehouse is deleted. There is also no way to recover a prior lakehouse if the data needs to be rolled back to a previous state.
BCDR was recently released but is only for capacity disaster recovery and is too cumbersome as it has an extra cost along with deploying in another region.
We need a way to recover a lakehouse and its history, if it has been deleted or if the data has been corrupted.
Ted Vilutis (administrator) on 05 Feb 2024 23:29:15
We are evaluating this idea and its timeline along with other priorities. Your comments and votes will help us plan and design the feature. Thank you for your feedback!
- Comments (3)
RE: Lakehouse Recovery or Restore Points
In production, this feature is very import to restore the lakehouse at ETL starting point when ETL process fails.Snowflake has one command to restore its warehouse to a time point
RE: Lakehouse Recovery or Restore Points
I have voted for this idea.My thoughts at first glance:I think when someone deletes a Lakehouse, then the Lakehouse (including it's data) should be moved to a "recycle bin" for a retention period (e.g. 30 days) before it gets completely deleted. In this period when the deleted lakehouse is in the "recycle bin", it should be possible to recover the Lakehouse (including it's data) by using an easy out-of-the-box solution.The same should also be possible at Lakehouse table level.Option to adjust settings in admin portal:Preferably, someone like the capacity admin can adjust the retention period in order to balance storage costs vs. recovery possibility time window.Also, someone like the capacity admin should have the option to allow selected users (security group) or all users to delete items from the "recycle bin", in this case the item gets completely deleted and unrecoverable. In order to save storage costs if necessary or delete unwanted data.
RE: Lakehouse Recovery or Restore Points
Is there any update on this feature request?To me, this is a core feature that a production ready system must have.This is currently keeping our company from using Fabric when building solutions.