Skip to main content
Microsoft Idea

    Synapse

    Planned

    Partitioning output to Lakehouse in event stream

    Vote (1) Share
    Ryoma Nagata's profile image

    Ryoma Nagata on 06 Jun 2023 07:56:37

    Partitioning is essential for efficiently using Delta Lake in use cases like IoT.

    Administrator

    The recommended way to stream data into Fabric is to use eventstream, and route the data to a KQL database that:

    • Supports partitioning. (Partitioning policy | Microsoft Learn - Though please partition only after you see a performance issue, usually in the Kusto tech you don't need that)
    • Provides high performance interactive queries over streaming data. (In KQL and T-SQL)
    • Makes the data available in parquet in the OneLake, so you can shortcut to it from a lake house to run Spark jobs.

    Partition support for KQL Database : KQL DB lake presence in Delta is coming soon, it will support partitioning. One logical copy in Real-Time Analytics 

    Partition support for Eventstreams : Collecting more requests to evaluate the plan.