Skip to main content

    Data Factory

    Needs Votes

    Fabric pipeline: allow specifying pool/environment for Notebook activity

    Vote (1) Share
    Martin Bode's profile image

    Martin Bode on 13 Dec 2023 12:16:10

    Pipline activities for Databricks notebooks allow specifying details concerning the cluster it is executed on:

    https://learn.microsoft.com/en-us/fabric/data-factory/media/azure-databricks-activity/databricks-activity-cluster.png

    https://learn.microsoft.com/en-us/fabric/data-factory/azure-databricks-activity#configuring-clusters


    Azure Synapse also has this kind of settings for Synapse Notebooks:

    https://learn.microsoft.com/en-us/azure/data-factory/media/transform-data-synapse-notebook/notebook-activity-settings.png

    https://learn.microsoft.com/en-us/azure/data-factory/transform-data-synapse-notebook#settings-tab


    For Fabric notebooks, users cannot specify any of those details

    https://learn.microsoft.com/en-us/fabric/data-factory/media/notebook-activity/choose-notebook-and-add-parameters.png

    https://learn.microsoft.com/en-us/fabric/data-factory/notebook-activity#notebook-settings


    I'd love to see same/similar configuration options for Fabric notebook pipeline activities:

    • pool / environment
    • node size 
    • number of worker nodes (fixed or range for auto-allocation)