Cole Tramp's Microsoft Insights

As more organizations seek greater flexibility and performance in their data pipelines, Microsoft's Dataflow Gen2, now part of Microsoft Fabric, offers compelling improvements over the legacy Dataflow Gen1 experience. If you’ve been using Gen1 for Power BI transformations, it may be time to consider moving your workloads.

Let’s explore the differences, the advantages of Gen2, and how to migrate efficiently using the import/export method.

Understanding the Evolution: Dataflow Gen1 vs Gen2

Dataflow Gen1 was introduced as part of Power BI to help users perform ETL (Extract, Transform, Load) operations with ease using Power Query. It allowed for cloud-based data prep at scale - but it came with several limitations that began to restrict enterprise use cases as demand grew.

Enter Gen2, which integrates directly with Microsoft Fabric and takes dataflow to the next level with scalability, flexibility, and enhanced orchestration.

Key Differences

Why Clients Are Moving to Gen2

We’re increasingly seeing clients hit throughput limits on Dataflow Gen1, which weren’t designed for the volume and complexity of modern data engineering workloads. This results in long refresh times, job failures, and bottlenecks in reporting pipelines.

Gen2 solves this by leveraging Spark-based compute within Microsoft Fabric, enabling far greater parallel processing and throughput. If you're encountering delays or scale challenges with Gen1, this is your cue to transition to Gen2.

Limitations of Gen1 to Keep in Mind

To further understand the urgency to upgrade, here are some notable limitations of Dataflow Gen1 source:

• Shared capacity limits the scalability of transformations
• Limited transformation types and connector support
• No direct support for pipelines or orchestration
• Output destinations are constrained primarily to Power BI datasets or Azure Data Lake
• No advanced error handling or retry logic

These constraints make Gen1 suitable only for lightweight scenarios. As your data ecosystem grows, Gen1 becomes increasingly insufficient.

Migrating from Gen1 to Gen2: The Recommended Approach

Unfortunately, there is no direct migration button from Gen1 to Gen2. However, Microsoft provides a straightforward export/import path to move your dataflow.

Step-by-Step Migration

1. Export your Gen1 Dataflow
   From Power BI Service, navigate to your existing Gen1 dataflow and choose “Export .json”. This gives you the full M code and metadata describing the transformations.
2. Import into Dataflow Gen2
   In Microsoft Fabric, create a new Dataflow Gen2 under the Data Factory experience, and use the “Import .json” option to bring in the same transformations.
3. Configure your destination
   Gen2 writes to OneLake, which can be shared across multiple workloads. This provides unified storage for your datasets and downstream processes.
4. Schedule or trigger execution
   Unlike Gen1, which relies on refresh cycles, Gen2 flows can be triggered via pipeline orchestration, giving you much finer control.

Final Thoughts

The move from Dataflow Gen1 to Gen2 is not just a version upgrade - it’s a strategic shift toward scalable, enterprise-ready data engineering on Microsoft Fabric. With better performance, broader integration, and orchestration capabilities, Gen2 is well-positioned to support modern analytics and AI-driven workloads. If you're already facing Gen1’s performance ceilings - or want to future-proof your environment - it’s time to consider the switch. The export/import method makes it relatively painless, and the payoff in speed and flexibility is worth the effort.

If you have any questions, feel free to reach out to me on Linkedin!