Toby Riley on 05 Jun 2023 10:42:23
Name Column Mapping Mode in SQL Endpoints for Lakehouse is not supported so tables with unsupported columns names (spaces, special chars, capitalization) do not work.
To re-create the behaviour.
1. Create a table in Databricks with column mapping enabled.
%python
spark.conf.set("spark.databricks.delta.properties.defaults.minWriterVersion", 5)
spark.conf.set("spark.databricks.delta.properties.defaults.minReaderVersion", 2)
spark.conf.set("spark.databricks.delta.properties.defaults.columnMapping.mode", "name")
2. Add shortcut tables from Databricks to Synapse Lakehouse. The data shows correctly.
3. Go to the SQL Endpoint. Tables fail to load. with the error. Corrective Action: Recreate the table without column mapping property.
4. This has a knock-on effect with data failing to feed into the datasets and Power BI.
Charles Webb (administrator)
We're working on this functionality along with deletion vectors support. Stay tuned to the blog for updates!
- Comments (3)
RE: Support Name Column Mapping Mode in SQL Endpoints for Lakehouse
Good information, thankssehackear.top
RE: Support Name Column Mapping Mode in SQL Endpoints for Lakehouse
Would this also enable "table name mapping". Apologies I know little about spark, but I'm currently having to build a warehouse on top of a lakehouse to serve as the "presentation layer" to clean up table and column names for use in Semantic Models via direct lake mode. Would be much, MUCH nicer if we could simply use "pretty" names with spaces and special characters directly in the Lakehouse for both columns AND tables.Thanks!Scott
RE: Support Name Column Mapping Mode in SQL Endpoints for Lakehouse
This is on our roadmap!