Alternative Monitoring of Azure Data Factory with Azure Monitor Metrics
Audio Highlights
This component uses custom JavaScript to open and close. Custom attributes and additional custom JavaScript is added to this component to make it accessible.
Video Highlights
This component uses custom JavaScript to open and close. Custom attributes and additional custom JavaScript is added to this component to make it accessible.
Why look beyond ADF Studio?
Azure Data Factory (ADF) is Microsoft’s fully managed data-integration service that orchestrates pipelines for ingesting, transforming and loading data at scale. In day-to-day work we lean on ADF Studio to review pipeline runs, activity runs and detailed error messages. That view is feature-rich, but once you execute dozens of pipelines in parallel, scrolling through paginated grids, expanding nested activities and cross-filtering becomes noisy and slow.
ADF ships real-time metrics to Azure Monitor out of the box. By tapping into those metrics you can:
- plot aggregated trends for many pipelines on one chart
- slice and dice by dimensions such as Pipeline Name or Failure Type
- pin the visuals to shared Azure dashboards
- trigger alerts with just a few clicks
Let's unlock these capabilities step by step in this article.
Built-in monitoring - a baseline
Before we leave ADF Studio, let’s agree on the baseline experience. The screenshot below shows the result of three sample pipelines:
- PipelineA failed on the first activity
- PipelineB succeeded end-to-end
- PipelineC contained a Wait activity and was deliberately cancelled
.png)
Enter Azure Monitor Metrics
Monitoring and analyzing ADF runs is rarely straightforward. ADF Studio surfaces everything you need, but also a lot that you don’t. When executions happen in bulk or in parallel, finding specific insights can be hard - sometimes impossible. Standard metrics, accessible in Azure Portal under Monitoring → Metrics, solve many of these pain-points.
.png)
Important caveats
- Metrics do not store full error details, activity input/output or debug runs - those remain exclusive to ADF Studio.
- A metric is simply a fact at a chosen aggregation (minute granularity at best) and time scope, much like facts and measures in a data warehouse
- Every metric comes with dimensions. By applying splitting, you can break a chart down by Pipeline Name, Activity Name, Failure Type and more.
Check Microsoft’s reference for the full list of metrics, units, dimensions and aggregation windows here.
.png)
Practical examples
Below are four diagrams I use most often.
1. Pipeline run status with splitting
Shows succeeded, failed and cancelled pipeline runs, split by Failure Type and Pipeline Name.
.png)
2. Same data - tabular view
.png)
3. Cancelled activity runs - detailed split
Cancelled activity runs, split by Activity Name, Activity Type, Failure type, and Pipeline Name
.png)
4. Same data - tabular view
.png)
Pinning metrics to Azure Dashboards
Azure Monitor metrics come with another bonus: every example above can be pinned to an Azure Dashboard. A dashboard may also host tiles from other services that underpin your solution - for example Storage Accounts with capacity growth metrics or network throughput charts. Configure the dashboard to sync the time range across all tiles. Hover over one graph and every other tile jumps to the exact same timestamp - a fantastic way to correlate events.
You can share dashboards with colleagues (requires RBAC access both to the dashboard and to the underlying services). For detailed setup instructions, check Azure's official guide.
.png)
Retention and limitations
- Azure Monitor stores metrics for the last 90 days at no extra cost - twice as long as ADF Studio monitoring, which keeps 45 days.
- If you need longer history, enable Diagnostic Logs to push metrics to a Storage Account or Log Analytics. Be aware that charting then becomes your responsibility.
- Metrics expose a minimum 1-minute aggregation. They cannot pinpoint sub-minute events.
- Debug-mode runs are not captured - only published (live-mode) pipelines send metrics.
Final thoughts
Azure Monitor metrics will not replace ADF Studio when you need step-level input, output or the offending row that broke your copy activity. Yet they are a powerful complement for:
- operations teams keeping an eye on health and SLA
- developers tracking performance drift over weeks or months
- automated alerting on filtered events without writing a single line of code
Give them a try in your next project - and happy monitoring!
Meet the authors
