Mimoune Djouallah on 29 May 2022 07:39:00
Currently when you create a dataflow, it give you the option to either create a new dataflow or a new datamart, which is a great option for a new user as you get a full Online experience from Data Prep, Modelling to Final reports, and a good option to start a new standalone report
but for existing users with a bit of experience, we prefer another Workflow, give us an option to load a dataflow to an existing Datamart, we want to use a Datamart as a central Database to store a lot of tables for a specific Business Topic, Currently the only way to do it is to create a separate dataflow and load data from there, the problem is, if you have a Datamart with 100 Tables, and only one table has changed, you need to reload all the 99 tables, that's not a very good usage of PowerBI resources Neither for customers nor Microsoft.
When Loading Data from Dataflow to Datamart, we prefer those option in order of Priority.
- Overwrite
- Incremental refresh
- append
- Upsert using PK etc
This is Microsoft opportunity to redefined ETL for mass market, and blur the line between Enterprise ETL and Self service Data preparation.
- Comments (3)
RE: Dataflow, load to an existing Datamart
@ Steven Fuller
both the idea is to decouple data from datamart for maximum flexibility, now if you want to refresh a table, you have to refresh the whole model, that does not work in any serious scenario
RE: Dataflow, load to an existing Datamart
Mimoune, are you referring to previously existing Dataflows as well? or just new dataflows?
Personally I think what you're suggesting would be great, especially for an existing dataflow that someone wants to use in a datamart. I'm sure I'm not the only one who already has many active dataflows that I would love to be able to work with in a datamart, but don't really want to duplicate all of them just for the datamart capabilities.
RE: Dataflow, load to an existing Datamart
Allowing dataflows to choose their destination simplifies the story for users:
- a dataflow is a way to get data from source systems into a storage destination.
- you can pick either basic datalake storage (CSVs + manifest/typing) or richer relational database storage (Azure SQL DB).