Statsbeat + smoke test (#1660)

* Statsbeat

* Fix merge conflicts

* Add Statsbeat Helper

* Track instrumentation list and encode it as long

* Track instrumentation names

* Add java.vendor to the feature list

* Add Attach statsbeat

* Add azure metadata service api call

* Put common custom dimensions in the base statsbeat

* Test azure metadata sevice call on both linux and windows

* Add statbest metric names

* Add scheduler to send statsbeat

* Reset after each interval

* Refactor update frequency interval

* Remove unused imports

* Increment network statsbeat counters

* Add tests

* Add tests for encode/decode feature

* Add StatsbeatModule

* Use non-static methods

* Add unit tests for NetworkStatsbeat

* Add tests for AttachStatsbeat

* Add tests for FeatureStatsbeat

* Refactor tests

* Update instrumentations map

* Sort instrumentation list alphabetically

* Fix failed tests

* Send statsbeat to a testing ikey

* Refactor

* Refactor

* Fix nullpointerexception in instrumentation list

* Send network statsbeats when its count is not zero

* Refactor

* Move private instance variables to the top of the class

* Reinit resourceProviderId after reset

* Track request durations

* Remove unnecessary changes

* Fix merge conflicts

* Set feature statsbeat to have daily interval

* Clean up logs

* Reset after send is done

* Add smoke test

* Use the default ingestion endpoint if statsbeat's endpoint is unknown

* Get statsbeat ikey from the config

* Delete StatsbeatTelemetry

* Make smoke test work

* Make smoke test stable

* Revert a line

* Fix unit tests compilation errors

* Make feature interval configurable

* Fix not seeing network and attach statsbeat in smoketest

* Update smoke test config

* Validate more statsbeats in the smoke test

* Enable more smoke test app servers for Statsbeat

* Address comments

* Address comments

* Fix lgtm

* Make MetricTelemetry class final since there is no need for StatsbeatTelemetry

* Update statsbeat iKey to be a workspace based mode

* Fix spotbug, make constants package protected

* No need for lazy init

* User more efficient entrySet instead of keySet

* Fix spotbugs

* Init azure metadata service in AttachStatsbeat

* No need for lazy init

* Fix nullpointerexception in ErrorHandlerTest and ThrottlingHandlerTest

* Fix exception messages

* Fix a nullpointerexception

* Handle race condition between send and reset

* Add unit test for race condition

* Make sense to make metadata service have the same interval as AttachStatsbeat

* Decide precisely when to shutdown AzureMetadataService

* Clean up debug logs

* Avoid reading resources from signed jar - logging configuration

* Avoid reading resources from signed jar - sdk version

* Add ignore matcher for ai agent classes

* Add signed jar access debug option

* Convert appsvc logging configuration also

* CLEAN UP SHADOW JAR

* Update submodule

* Cleanup

* Igore a test for now

* Fix Illegal reflective access by org.junit.contrib.java.lang.system.EnvironmentVariables

* Fix unit tests

* Remove unused instance var

* Fix test failures

* Delete a console output log

* Fix spotbug DC_DOUBLECHECK

* Fix statsbeat smoke test not sending 'Request Duration'

* Fix a compiling error in unit test

* Address feedback

* Make statsbeat config internal and skip parsing the connection string

* Address feedback

* Track total and count for request durations

* Address more feedback

* Address feedback

* Use BitSet and move decode to a test utils class

* Remove transient properties since it's not using failOnUnknown on json adapter

* Refactor AttachStatsbeat

* Refactor AzureMetadataService

* Reduce visibility

* Refactor StatsbeatModule

* Introduce resetForTest

* Make a couple things volatile

* A bit more

* Simplify

* Update singleton naming a bit

* Reduce reliance on singletons for testing

* Reduce reliance on singletons for testing

* Synchronization

* Use long instead of double

* Fix a compiling error

* Remove unused methods

* Remove unused imports

* Fix statsbeat module is not properly initialized

* Use AtomicLong for consistency

* Fix array index out of bound

* Fix ThrottlingTest

* Fix ErrorHandlerTest

* Reuse requestSuccessCount for requestDurationCount

* Throw an exception when init gets called more than once

* Switch expected and actual parameters in tests

* Atomicity

* Can not be null

* Inline

* Warning message

* Remove null check

* Remove null check

* Add name to TODO

* Remove getInterval method

* Extract ResourceProvider into enum

* Extract OperatingSystem into enum

* Extract Feature enum

* Fix ErrorHandlerTest and ThrottleingHandlerTest

* Convert CustomDimensions to use explicit fields

* Move more constants out of Constants

* Move more constants out of Constants

* Move more constants out of Constants

* Move more constants out of Constants

* Remove unused field

* Add todo

* Small renaming

* Remove StatsbeatHelper

* Comments

* Send base 64 encoded string instead of utf-8 string for instrumentation and feature

* Add missing copyrights

* Add a comment

* Spotbugs

* Remove todos

* Fix instrumentation is empty in the payload

* WithoutPadding on Base64 encoder

Co-authored-by: Trask Stalnaker <[email protected]>

agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/AiComponentInstaller.java

@@ -53,8 +53,11 @@
 import com.microsoft.applicationinsights.internal.config.ParamXmlElement;
 import com.microsoft.applicationinsights.internal.config.TelemetryConfigurationFactory;
 import com.microsoft.applicationinsights.internal.config.TelemetryModulesXmlElement;
+import com.microsoft.applicationinsights.internal.config.connection.ConnectionString;
+import com.microsoft.applicationinsights.internal.config.connection.InvalidConnectionStringException;
 import com.microsoft.applicationinsights.internal.profiler.GcEventMonitor;
 import com.microsoft.applicationinsights.internal.profiler.ProfilerServiceInitializer;
+import com.microsoft.applicationinsights.internal.statsbeat.StatsbeatModule;
 import com.microsoft.applicationinsights.internal.system.SystemInformation;
 import com.microsoft.applicationinsights.internal.util.PropertyHelper;
 import com.microsoft.applicationinsights.profiler.config.ServiceProfilerServiceConfig;
@@ -157,6 +160,12 @@ private static void start(Instrumentation instrumentation) {
         configuration.getContextInitializers().add(new SdkVersionContextInitializer());
         configuration.getContextInitializers().add(new ResourceAttributesContextInitializer(config.customDimensions));
 
+        try {
+            ConnectionString.updateStatsbeatConnectionString(config.internal.statsbeat.instrumentationKey, config.internal.statsbeat.endpoint, configuration);
+        } catch (InvalidConnectionStringException ex) {
+            startupLogger.warn("Statsbeat endpoint is invalid. {}", ex.getMessage());
+        }
+
         Global.setSamplingPercentage(config.sampling.percentage);
         final TelemetryClient telemetryClient = new TelemetryClient();
         Global.setTelemetryClient(telemetryClient);
@@ -200,6 +209,9 @@ public void run() {
         if (rpConfiguration != null) {
             RpConfigurationPolling.startPolling(rpConfiguration, config);
         }
+
+        // initialize StatsbeatModule
+        StatsbeatModule.initialize(telemetryClient, config.internal.statsbeat.intervalSeconds, config.internal.statsbeat.featureIntervalSeconds);
     }
 
     private static GcEventMonitor.GcEventMonitorConfiguration formGcEventMonitorConfiguration(Configuration.GcEventConfiguration gcEvents) {

agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/wasbootstrap/configuration/Configuration.java

@@ -24,6 +24,7 @@
 import com.microsoft.applicationinsights.agent.bootstrap.diagnostics.DiagnosticsHelper;
 import com.microsoft.applicationinsights.agent.bootstrap.diagnostics.status.StatusFile;
 import com.microsoft.applicationinsights.customExceptions.FriendlyException;
+import com.microsoft.applicationinsights.internal.config.connection.ConnectionString;
 import com.microsoft.applicationinsights.internal.profiler.GcReportingLevel;
 
 import java.util.ArrayList;
@@ -34,6 +35,7 @@
 import java.util.regex.Pattern;
 import java.util.regex.PatternSyntaxException;
 
+import static java.util.concurrent.TimeUnit.DAYS;
 import static java.util.concurrent.TimeUnit.MINUTES;
 
 // an assumption is made throughout this file that user will not explicitly use `null` value in json file
@@ -50,6 +52,7 @@ public class Configuration {
     public Proxy proxy = new Proxy();
     public SelfDiagnostics selfDiagnostics = new SelfDiagnostics();
     public PreviewConfiguration preview = new PreviewConfiguration();
+    public InternalConfiguration internal = new InternalConfiguration();
 
     // this is just here to detect if using old format in order to give a helpful error message
     public Map<String, Object> instrumentationSettings;
@@ -153,10 +156,16 @@ public static class SpringSchedulingInstrumentation {
     }
 
     public static class Heartbeat {
-
         public long intervalSeconds = MINUTES.toSeconds(15);
     }
 
+    public static class Statsbeat {
+        public String instrumentationKey = "c4a29126-a7cb-47e5-b348-11414998b11e"; //workspace-aistatsbeat
+        public String endpoint = ConnectionString.Defaults.INGESTION_ENDPOINT; // this supports the government cloud
+        public long intervalSeconds = MINUTES.toSeconds(15); // default to 15 minutes
+        public long featureIntervalSeconds = DAYS.toSeconds(1); // default to daily
+    }
+
     public static class Proxy {
 
         public String host;
@@ -186,6 +195,11 @@ public static class PreviewConfiguration {
         public GcEventConfiguration gcEvents = new GcEventConfiguration();
     }
 
+    public static class InternalConfiguration {
+        // This is used for collecting internal stats
+        public Statsbeat statsbeat = new Statsbeat();
+    }
+
     public static class PreviewInstrumentation {
         public DisabledByDefaultInstrumentation azureSdk = new DisabledByDefaultInstrumentation();
         public DisabledByDefaultInstrumentation javaHttpClient = new DisabledByDefaultInstrumentation();

agent/exporter/src/main/java/com/microsoft/applicationinsights/agent/Exporter.java

@@ -34,6 +34,7 @@
 import com.google.common.cache.CacheBuilder;
 import com.microsoft.applicationinsights.TelemetryClient;
 import com.microsoft.applicationinsights.TelemetryConfiguration;
+import com.microsoft.applicationinsights.internal.statsbeat.StatsbeatModule;
 import com.microsoft.applicationinsights.telemetry.Duration;
 import com.microsoft.applicationinsights.telemetry.EventTelemetry;
 import com.microsoft.applicationinsights.telemetry.ExceptionTelemetry;
@@ -173,6 +174,7 @@ public CompletableResultCode shutdown() {
     private void export(SpanData span) {
         SpanKind kind = span.getKind();
         String instrumentationName = span.getInstrumentationLibraryInfo().getName();
+        StatsbeatModule.get().getNetworkStatsbeat().addInstrumentation(instrumentationName);
         Matcher matcher = COMPONENT_PATTERN.matcher(instrumentationName);
         String stdComponent = matcher.matches() ? matcher.group(1) : null;
         if (kind == SpanKind.INTERNAL) {

agent/instrumentation/build.gradle

@@ -73,8 +73,8 @@ dependencies {
     implementation group: 'io.opentelemetry.javaagent.instrumentation', name: 'opentelemetry-javaagent-servlet-3.0', version: versions.opentelemetryInstrumentationAlpha
     implementation group: 'io.opentelemetry.javaagent.instrumentation', name: 'opentelemetry-javaagent-servlet-common', version: versions.opentelemetryInstrumentationAlpha
     implementation group: 'io.opentelemetry.javaagent.instrumentation', name: 'opentelemetry-javaagent-spring-scheduling-3.1', version: versions.opentelemetryInstrumentationAlpha
-    implementation group: 'io.opentelemetry.javaagent.instrumentation', name: 'opentelemetry-javaagent-spring-webmvc-3.1', version: versions.opentelemetryInstrumentationAlpha
     implementation group: 'io.opentelemetry.javaagent.instrumentation', name: 'opentelemetry-javaagent-spring-webflux-5.0', version: versions.opentelemetryInstrumentationAlpha
+    implementation group: 'io.opentelemetry.javaagent.instrumentation', name: 'opentelemetry-javaagent-spring-webmvc-3.1', version: versions.opentelemetryInstrumentationAlpha
     implementation group: 'io.opentelemetry.javaagent.instrumentation', name: 'opentelemetry-javaagent-tomcat-7.0', version: versions.opentelemetryInstrumentationAlpha
 
     // also: jaxrs, spring, struts

core/src/main/java/com/microsoft/applicationinsights/TelemetryConfiguration.java

@@ -50,6 +50,7 @@ public final class TelemetryConfiguration {
     private String connectionString;
     private String roleName;
     private String roleInstance;
+    private String statsbeatInstrumentationKey;
 
     private final EndpointProvider endpointProvider = new EndpointProvider();
 
@@ -166,7 +167,6 @@ public String getInstrumentationKey() {
      * @throws IllegalArgumentException when the new value is null or empty
      */
     public void setInstrumentationKey(String key) {
-
         // A non null, non empty instrumentation key is a must
         if (Strings.isNullOrEmpty(key)) {
             throw new IllegalArgumentException("key");
@@ -175,6 +175,14 @@ public void setInstrumentationKey(String key) {
         instrumentationKey = key;
     }
 
+    public String getStatsbeatInstrumentationKey() {
+        return statsbeatInstrumentationKey;
+    }
+
+    public void setStatsbeatInstrumentationKey(String key) {
+        statsbeatInstrumentationKey = key;
+    }
+
     public String getRoleName() {
         return roleName;
     }

core/src/main/java/com/microsoft/applicationinsights/channel/concrete/TelemetryChannelBase.java

@@ -72,6 +72,9 @@ public abstract class TelemetryChannelBase<T> implements TelemetryChannel {
 
     protected TelemetriesTransmitter<T> telemetriesTransmitter;
     protected TelemetryBuffer<T> telemetryBuffer;
+    protected TelemetriesTransmitter<T> statsbeatTransmitter;
+    protected TelemetryBuffer<T> statsbeatBuffer;
+
 
     private boolean developerMode = false;
 
@@ -135,8 +138,10 @@ protected synchronized void initialize(TelemetryConfiguration configuration, Str
         makeSureEndpointAddressIsValid(endpointAddress);
 
         final ConfiguredTransmitterFactory<T> transmitterFactory = getTransmitterFactory();
-        telemetriesTransmitter = transmitterFactory.create(configuration, maxTransmissionStorageCapacity, throttling, maxInstantRetry);
+        telemetriesTransmitter = transmitterFactory.create(configuration, maxTransmissionStorageCapacity, throttling, maxInstantRetry, false);
         telemetryBuffer = new TelemetryBuffer<>(telemetriesTransmitter, maxTelemetryBufferCapacityEnforcer, sendIntervalInSeconds);
+        statsbeatTransmitter = transmitterFactory.create(configuration, maxTransmissionStorageCapacity, throttling, maxInstantRetry, true);
+        statsbeatBuffer = new TelemetryBuffer<>(statsbeatTransmitter, maxTelemetryBufferCapacityEnforcer, sendIntervalInSeconds);
 
         setDeveloperMode(developerMode);
         isInitailized = true;

core/src/main/java/com/microsoft/applicationinsights/channel/concrete/inprocess/InProcessTelemetryChannel.java

@@ -24,10 +24,13 @@
 import com.microsoft.applicationinsights.TelemetryConfiguration;
 import com.microsoft.applicationinsights.channel.concrete.TelemetryChannelBase;
 import com.microsoft.applicationinsights.internal.channel.ConfiguredTransmitterFactory;
+import com.microsoft.applicationinsights.telemetry.MetricTelemetry;
 import com.microsoft.applicationinsights.telemetry.Telemetry;
 
 import java.util.Map;
 
+import static com.microsoft.applicationinsights.internal.statsbeat.Constants.STATSBEAT_TELEMETRY_NAME;
+
 /**
  * An implementation of {@link com.microsoft.applicationinsights.channel.TelemetryChannel}
  *
@@ -64,13 +67,23 @@ protected boolean doSend(Telemetry telemetry) {
         if (telemetry.previouslyUsed()) {
             throw new IllegalStateException("Telemetry was previously used: " + telemetry);
         }
-        telemetryBuffer.add(telemetry);
+
+        // TODO Prepare for AAD support for Statsbeat iKey
+        if (telemetry instanceof MetricTelemetry) {
+            MetricTelemetry mt = (MetricTelemetry) telemetry;
+            if (STATSBEAT_TELEMETRY_NAME.equalsIgnoreCase(mt.getTelemetryName())) {
+                statsbeatBuffer.add(telemetry);
+            } else {
+                telemetryBuffer.add(telemetry);
+            }
+        } else {
+            telemetryBuffer.add(telemetry);
+        }
         return true;
     }
 
     @Override
     protected ConfiguredTransmitterFactory<Telemetry> createTransmitterFactory() {
         return new InProcessTelemetryTransmitterFactory();
     }
-
 }

core/src/main/java/com/microsoft/applicationinsights/channel/concrete/inprocess/InProcessTelemetryTransmitterFactory.java

@@ -47,14 +47,14 @@
 final class InProcessTelemetryTransmitterFactory implements ConfiguredTransmitterFactory {
 
     @Override
-    public TelemetriesTransmitter create(TelemetryConfiguration configuration, String maxTransmissionStorageCapacity, boolean throttlingIsEnabled, int maxInstantRetries) {
+    public TelemetriesTransmitter create(TelemetryConfiguration configuration, String maxTransmissionStorageCapacity, boolean throttlingIsEnabled, int maxInstantRetries, boolean isStatsbeat) {
         final TransmissionPolicyManager transmissionPolicyManager = new TransmissionPolicyManager(throttlingIsEnabled);
         transmissionPolicyManager.addTransmissionHandler(new ErrorHandler(transmissionPolicyManager));
         transmissionPolicyManager.addTransmissionHandler(new PartialSuccessHandler());
         transmissionPolicyManager.addTransmissionHandler(new ThrottlingHandler(transmissionPolicyManager));
         transmissionPolicyManager.setMaxInstantRetries(maxInstantRetries);
         // An active object with the network sender
-        TransmissionNetworkOutput actualNetworkSender = TransmissionNetworkOutput.create(configuration, transmissionPolicyManager);
+        TransmissionNetworkOutput actualNetworkSender = TransmissionNetworkOutput.create(configuration, transmissionPolicyManager, isStatsbeat);
 
         return finishTransmitterConstruction(maxTransmissionStorageCapacity, transmissionPolicyManager, actualNetworkSender);
     }

core/src/main/java/com/microsoft/applicationinsights/internal/channel/ConfiguredTransmitterFactory.java

@@ -13,5 +13,5 @@ public interface ConfiguredTransmitterFactory<T> {
      * @param maxInstantRetries
      * @return
      */
-    TelemetriesTransmitter<T> create(@Nullable TelemetryConfiguration configuration, String maxTransmissionStorageCapacity, boolean throttlingIsEnabled, int maxInstantRetries);
+    TelemetriesTransmitter<T> create(@Nullable TelemetryConfiguration configuration, String maxTransmissionStorageCapacity, boolean throttlingIsEnabled, int maxInstantRetries, boolean isStatsbeat);
 }

core/src/main/java/com/microsoft/applicationinsights/internal/channel/common/ErrorHandler.java

@@ -2,6 +2,7 @@
 
 import com.microsoft.applicationinsights.internal.channel.TransmissionHandler;
 import com.microsoft.applicationinsights.internal.channel.TransmissionHandlerArgs;
+import com.microsoft.applicationinsights.internal.statsbeat.StatsbeatModule;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -48,6 +49,10 @@ boolean validateTransmissionAndSend(TransmissionHandlerArgs args) {
             case TransmissionSendResult.SERVICE_UNAVAILABLE:
             case TransmissionSendResult.CLIENT_SIDE_EXCEPTION:
                 backoffAndSendTransmission(args);
+                // TODO (heya) remove this null check later
+                if (StatsbeatModule.get() != null) {
+                    StatsbeatModule.get().getNetworkStatsbeat().incrementRetryCount();
+                }
                 return true;
             default:
                 logger.trace("Http response code {} not handled by {}", args.getResponseCode(),

core/src/main/java/com/microsoft/applicationinsights/internal/channel/common/ThrottlingHandler.java

@@ -6,6 +6,7 @@
 import java.util.Date;
 import java.util.TimeZone;
 
+import com.microsoft.applicationinsights.internal.statsbeat.StatsbeatModule;
 import org.apache.http.Header;
 
 import com.google.common.base.Strings;
@@ -62,6 +63,10 @@ boolean validateTransmissionAndSend(TransmissionHandlerArgs args) {
             case TransmissionSendResult.THROTTLED_OVER_EXTENDED_TIME:
                 suspendTransmissions(TransmissionPolicy.BLOCKED_BUT_CAN_BE_PERSISTED, args.getRetryHeader());
                 args.getTransmissionDispatcher().dispatch(args.getTransmission());
+                // TODO (heya) remove this null check later
+                if (StatsbeatModule.get() != null) {
+                    StatsbeatModule.get().getNetworkStatsbeat().incrementThrottlingCount();
+                }
                 return true;
             default:
                 logger.trace("Http response code {} not handled by {}", args.getResponseCode(),