Skip to main content
Microsoft Idea

    Data Factory

    New

    DEAL WITH REST API AS DATASOURCES ON DATAFLOW GEN2 (simplify data citizens life)

    Vote (1) Share
    Bamakus Montreuil's profile image

    Bamakus Montreuil on 18 Apr 2024 10:51:12

    DEAL WITH REST API AS DATASOURCES ON DATAFLOW GEN2 (simplify data citizens life)




    • ADD TEMPLATES & FORMS to automatically create M functions with parameters


    • According to deal with API setups (connections , continuation token , API body (post) & API url parameters (get) )


    • Kind of "black box" to use on new tables columns creations ( by passing parameters)


    • Will facilitate API ingestion builds

    Administrator on 31 May 2024 09:42:43

    Not all REST APIs are exactly the same and how they deal with rate limits, authentication and paginations tends to be quite unique to their own implementation.

    We plan to support custom connectors created with the Power Query SDK Power Query SDK overview - Power Query | Microsoft Learn in Dataflow Gen2 in the near future. This would be the most secure and flexible way for you to setup your own connection with the correct authentication flow to your data source as well as supporting any other mechanisms that you might have on your REST API such as pagination and such. There are some limits, for privacy and secure reasons, on what sort of information can be levaraged for scenarios that rely on the response headers or what sort of POST calls we can support (without any authentication today), so our plan today is to support custom connectors so you have the most flexibility when connecting to your unique REST API.

    However, I understand that the request is to provide something generic enough through the Web REST API connector. I'm changing the status to be "Needs Votes" so the community can vote on this idea.