I composed this document in response to a recent incident encountered within the application related to Application Insight where our telemetry data was getting sampled.
Error Message: “Some data in this view may be inaccurate due to sampling…..”
The error looks like below :
Before coming to the solution or further details lets understand the basic terms first.
What is Application Insight?
Application Insight collects telemetry about your app, including web server telemetry, web page telemetry, and performance counters. This data can be used to monitor your app’s performance, health, and usage.
It is a feature of Azure Monitor which excels in Application Performance Management (APM) for live web applications.
Through its dashboard, it provides a summary in the overview pane to allow at-a-glance assessment of your application’s health and performance.
What is Telemetry data?
In general, Telemetry is defined as the automatic process of the data created by your systems being remotely collected using agents and protocols. Telemetry data extends to and includes all logs, metrics, events and traces that are created by your applications.
In Application Insights, it sends telemetry from your web application to the Azure portal so that you can analyze the performance and usage of your application. The telemetry model is standardized, so it’s possible to create platform- and language-independent monitoring.
What is Sampling?
Sampling is a feature in Application Insights. It’s the recommended way to reduce telemetry traffic, data costs, and storage costs, while preserving a statistically correct analysis of application data. Sampling also helps you avoid Application Insights throttling your telemetry.
Types of sampling
There are three different sampling methods:
Adaptive sampling automatically adjusts the volume of telemetry sent from the SDK in your ASP.NET/ASP.NET Core app, and from Azure Functions. It’s the default sampling when you use the ASP.NET or ASP.NET Core SDK. Adaptive sampling is currently only available for ASP.NET/ASP.NET Core server-side telemetry, and for Azure Functions.
Fixed-rate sampling reduces the volume of telemetry sent from both your ASP.NET or ASP.NET Core or Java server and from your users’ browsers. You set the rate. The client and server synchronize their sampling so that, in Search, you can navigate between related page views and requests.
Ingestion sampling happens at the Application Insights service endpoint. It discards some of the telemetry that arrives from your app, at a sampling rate that you set. It doesn’t reduce telemetry traffic sent from your app, but helps you keep within your monthly quota. The main advantage of ingestion sampling is that you can set the sampling rate without redeploying your app. Ingestion sampling works uniformly for all servers and clients, but it doesn’t apply when any other types of sampling are in operation.
Solution to the Issue:
Now as we know something about the basic terms, you should be able to relate to the issue. As you can see from the above provided snapshot, the telemetry data was getting sampled by default. I found that some configurations were left unattended which led to this issue.
We thought not to disable the sampling instead configure it so that the data/messages related to 500 HTTP status code or Exceptions should not get sampled.
Out of many, we have 2 particular Sitecore applications built on Asp.net and Asp.Net Core. The configuration applied to both are different but the property to be set was same.
Our both applications were using Adaptive Sampling, so we are going to share the configuration changes for the same only. You can check out more configuration at this link.
- In Asp.net MVC application, once you have configured the app with Application Insight you will see ApplicationInsight.config file. Search out <ExcludedTypes> property.This property accepts below values:Dependency, Event, Exception, PageView, Request, TraceWe have configured to exclude Event and Exception in our case, so the property looks like below.
- In Asp.net Core Application, the configuration may differ based on Dot Net version, our application was using .Net 6 so below are the changes we did in startup.cs file:
services.Configure<TelemetryConfiguration>(telemetryConfiguration =>
{
var telemetryProcessorChainBuilder = telemetryConfiguration.DefaultTelemetrySink.TelemetryProcessorChainBuilder;
telemetryProcessorChainBuilder.UseAdaptiveSampling(maxTelemetryItemsPerSecond: 5, excludedTypes: “Exception”);
telemetryProcessorChainBuilder.Build();
});
services.AddApplicationInsightsTelemetry(options =>
{
options.ConnectionString = Configuration[“ApplicationInsights:ConnectionString”];
options.EnableAdaptiveSampling = false;
});
You can explore more on this link.
Conclusion:
Sampling is very useful while collecting the telemetry data sent by your application, but it’s more important which type of sampling is best suited and how to configure it correctly. Below are some of the highlighted features of it:
- Application Insights service drops (“throttles”) data points when your app sends a high rate of telemetry in a short time interval. Sampling reduces the likelihood that your application sees throttling occur.
- To keep within the quota of data points for your pricing tier.
- To reduce network traffic from the collection of telemetry.
References:
https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling-classic-api