Skip to main content
Microsoft Idea

Data Factory

Planned

Enable a way to set incremental refresh in Dataflows

Vote (208) Share
Miguel Escobar's profile image

Miguel Escobar on 31 May 2023 20:13:25

Enable an experience that allows the end-user to setup incremental refresh for Dataflows Gen2 in Data Factory for Microsoft Fabric

Administrator on 02 Apr 2024 19:40:00

Incremental Refresh support for Dataflows Gen2 is planned for second half of 2023. Stay tuned!

Comments (6)
Miguel Escobar's profile image Profile Picture

P B on 28 Jan 2024 22:42:58

RE: Enable a way to set incremental refresh in Dataflows

Hi. I agree, a support for the incremental refresh is absolutely a "must have"!

In addition to that, let users:

  1. to identify Key column(s) from the source (or read meta data from source)
  2. to identify date(time) columns such as: CreatedOn, ModifiedOn, LastModifiedOn
  3. to choose where to drop the result in the Tables area of Files area


BR,

Predrag

Miguel Escobar's profile image Profile Picture

Marco Rozzano on 02 Dec 2023 04:03:41

RE: Enable a way to set incremental refresh in Dataflows

We're eager to use incremental refresh in dataflows gen 2. Any hint about when we can expect this critical feature ? The second half of 2023 is almost over...

Miguel Escobar's profile image Profile Picture

Christian Koefoed on 27 Nov 2023 15:57:52

RE: Enable a way to set incremental refresh in Dataflows

Any update on this? Is this still planned for second half of 2023?

An extremely essential function

Miguel Escobar's profile image Profile Picture

Shameer Hameed Bava on 26 Nov 2023 12:14:44

RE: Enable a way to set incremental refresh in Dataflows

Fabric is in GA, and we are eagerly awaiting IR for Dataflows Gen2. As we're on the verge of 2023, do we have any updates on the ETA?

Miguel Escobar's profile image Profile Picture

Jason Campbell on 14 Jul 2023 17:41:56

RE: Enable a way to set incremental refresh in Dataflows

Incremental refreshes are sorely needed when getting data from on-prem SQL servers. Not having this ability can be (is) a deal breaker.

Miguel Escobar's profile image Profile Picture

Edgar Walther on 31 May 2023 20:29:51

RE: Enable a way to set incremental refresh in Dataflows

Add a Table Action option for Incremental Load.

  • Let the user identify the Primary Key column(s) from the source (or read meta data from source)
  • Add some auditing columns (is current, is active, is deleted, load date, update date, current key)
  • Add new rows, update existing rows, mark deleted rows.

Or is some of this plumbing already taken care of by the delta tables?