Data Engineering
NewRun On Demand Item Job request body should support environment and session assignment
Jens Vestergaard on 26 Apr 2024 06:09:59
I am are trying to build out an automated setup, where before I run data through notebooks in advance know whether current load will be light, medium or heavy. This knowledge I would like to leverage so that I can (automatically and meta data driven) invoke the right environment on our Fabric capacity. Currently I am running a pessimistic approach on the environment, but this fails as data loads get over a certain size (> 1GB).
In essence, I would like to be able to specify which environment the job should be run on in the request body part.
In addition to this, I would like to be able to specify a running session to hook on to, so that the 4+ min. wait for a new session will be eliminated.