Synapse
Under ReviewHaving the ability to use a view created in a Lakehouse in a Spark Notebook.
Erwin de Kreuk on 31 Oct 2023 07:18:38
Having the ability to use a view created in a Lakehouse in a Spark Notebook.
Currently this is not possible and you will get an exception, table or view not found.
As an example. you want to combine data from a Silver Layer from different ERP systems in 1 view to create your Gold Layer from there, this is only possible with a Merge in Synapse DataWarehouse and not with a Notebook in Spark currently
Ted Vilutis (administrator)
We are evaluating this idea and its timeline along with other priorities. We are looking to enable Views in Lakehouse Explorer as it's not yet supported. Your comments and votes will help us plan and design the feature. Thank you for your feedback!
- Comments (2)
RE: Having the ability to use a view created in a Lakehouse in a Spark Notebook.
Erwincould you please explain more about the definition of "View created in Lakehouse"? in current notebook, you can use the Spark SQL to create a view like df.createOrReplaceTempView("my_view") on top of a dataframe and run further Spark SQL query on this view object, is this the View object you are referring to or something else?
RE: Having the ability to use a view created in a Lakehouse in a Spark Notebook.
good idea, in addtion, it would be nice to be able to read in a notebook (spark.read.table?) , from views created in warehouses