Skip to main content

Data Engineering

New

Run On Demand Item Job request body should support environment and session assignment

Vote (3) Share
Jens Vestergaard's profile image

Jens Vestergaard on 26 Apr 2024 06:09:59

I am are trying to build out an automated setup, where before I run data through notebooks in advance know whether current load will be light, medium or heavy. This knowledge I would like to leverage so that I can (automatically and meta data driven) invoke the right environment on our Fabric capacity. Currently I am running a pessimistic approach on the environment, but this fails as data loads get over a certain size (> 1GB).


In essence, I would like to be able to specify which environment the job should be run on in the request body part.

In addition to this, I would like to be able to specify a running session to hook on to, so that the 4+ min. wait for a new session will be eliminated.