Data Factory
PlannedEnable to pass variables from pipelines as parameters into a Dataflow Gen2
Miguel Escobar on 01 Jun 2023 17:44:52
Enable to modify the mashup document of a Dataflow Gen2 by passing a variable from a pipeline before an execution of the Dataflow
Administrator on 22 Aug 2024 17:14:52
This is currently in planning. Be sure to check out the release plan for timelines around this feature once they become available:
What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
- Comments (28)
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
I'm trying to move large tables from on-prem SQL server to Fabric. Until the data gateways or mirroring improvements go live, splitting the work into many dataflows each touching a different "slice" of data and appending into a Lakehouse is the only viable solution. But this is currently extremely painful, because:You can't simple "save as" a dataflow as a copy. Which means you have to set up many copies of the same dataflow manually, manually map them to the destination tables, etc.You can't change the parameters in the dataflows via a pipeline, which means you can't have a single job that you can reuse for multiple slices in parallelI also agree with the comments about allowing the data sources and data destinations to be controllable via parameters. In my case, if I'm running 10 copies of a dataflow each dealing with a slice of data, I might want to spread them across different gateway clusters so I don't bog any of them down.Thanks,Scott
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
Looking forward to this feature
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
Would like this feature to be available soon as need to have the flexibility to run Data Flow from Pipeline with parameters
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
Definitely require this feature to make DFG2's effective for ETLing a large number of rows from an app via of REST calls that require an advancing row offset - i.e. many calls of the same DFG2 in which only the offset changes.
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
This would be a great feature to have to increase the value of switching our workflows to Dataflow Gen 2 from TE3 and XMLA!
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
Ability to make a data source use a parameter - it is possible in Powerbi data load but not in data factory. This would enable promotion of data sources from Dev to production without altering code.
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
That would be great, because then I could use the pipelines to control and execute dataflows and, for example, execute the same dataflow on different sources.