Christoph Mettler on 08 Jun 2023 11:29:27
Data providers typically deliver data in formats such as parquet, JSON, or CSV. In traditional data warehouse scenarios, these files often land in a persistent staging area. Many organize these staging folders in a structured manner such as year/month/week/day. Currently, these files need to be converted to delta format with an additional workload before they can be consumed in the data warehouse.
The support of openrowset/create external table, as with parquet file globbing, would greatly simplify the load and transform to final delta files in the warehouse workload without the need for additional workloads/datalflow/pipelines complexity.
Administrator
Can you share more details about your scenarios and if Lakehouse Shortcuts would work for you?
- Comments (3)
RE: support openrowset/polybase
Hello Admin, is this being considered?Thanks.
RE: support openrowset/polybase
Hello,Currently we are serving many BI workloads(Power BI Reports, Excel Tables, etc) through Synapse Serverless SQL openrowset with data from datalake in different formats and from different layers (in medallion architecture).We are trying to move to Fabric now, but all our data is in parquet format in ADLS2 and existing pipelines are also ingesting the data into ADLS2 (external data, export setupts, etc).Shortcut is good but it can't be served to BI through fabric without loading it into delta tables. So there needs another ETL to load the ADLS2 data into Delta Tables through spark ETL which is killing mosquito with hammer.Hope to see OPENROWSET in fabric soon.Thanks.
RE: support openrowset/polybase
we use this pattern extensively at work, ADF, azure storage, staging table in sql server, our data engineer dismissed Fabric because of the lack of this feature.