Data Factory
PlannedEnable to pass variables from pipelines as parameters into a Dataflow Gen2
Miguel Escobar on 01 Jun 2023 17:44:52
Enable to modify the mashup document of a Dataflow Gen2 by passing a variable from a pipeline before an execution of the Dataflow
Administrator on 22 Aug 2024 17:14:52
This is currently in planning. Be sure to check out the release plan for timelines around this feature once they become available:
What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
- Comments (31)
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
I don't see it mentioned, but getting the parameter into the dataflow is not much use unless you can use it in a data source. Some will want it for working through a list of files. Others will want to inject it into a sql query (whether Direct Query or Import). I'd like both, please. Thx
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
Need this very much!!!!!
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
I would think this is a basic function. How can you not pass parameters into a Gen2 Dataflow, ie to dynamically build the SQL?I'd expect this in January
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
I'm trying to move large tables from on-prem SQL server to Fabric. Until the data gateways or mirroring improvements go live, splitting the work into many dataflows each touching a different "slice" of data and appending into a Lakehouse is the only viable solution. But this is currently extremely painful, because:You can't simple "save as" a dataflow as a copy. Which means you have to set up many copies of the same dataflow manually, manually map them to the destination tables, etc.You can't change the parameters in the dataflows via a pipeline, which means you can't have a single job that you can reuse for multiple slices in parallelI also agree with the comments about allowing the data sources and data destinations to be controllable via parameters. In my case, if I'm running 10 copies of a dataflow each dealing with a slice of data, I might want to spread them across different gateway clusters so I don't bog any of them down.Thanks,Scott
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
Looking forward to this feature
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
Would like this feature to be available soon as need to have the flexibility to run Data Flow from Pipeline with parameters
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
Definitely require this feature to make DFG2's effective for ETLing a large number of rows from an app via of REST calls that require an advancing row offset - i.e. many calls of the same DFG2 in which only the offset changes.
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
This would be a great feature to have to increase the value of switching our workflows to Dataflow Gen 2 from TE3 and XMLA!
RE: Enable to pass variables from pipelines as parameters into a Dataflow Gen2
Ability to make a data source use a parameter - it is possible in Powerbi data load but not in data factory. This would enable promotion of data sources from Dev to production without altering code.