Building observable systems enables development teams at CSE to measure how well the application is behaving.
- Provide holistic view on the health of the application.
- Help measure business performance for the customer.
- Measure operational performance of the system.
- Identify and diagnose failures to get to the problem fast.
- Pillars of Observability
- Recommended Practices
- Pitfalls to Avoid
- Recommended Tools and Approaches
- Tools
- Recipes
- Correlation Id: Include unique identifier at the start of the interaction to tie down aggregated data from various system components and provide a holistic view. Read more guidelines about using correlation id.
- Ensure health of the services are monitored and provide insights into system's performance and behavior.
- Faults, crashes, and failures are logged as discrete events. This helps engineers identify problem area(s) during failures.
- Ensure logging configuration (eg: setting logging to "verbose") can be controlled without code changes.
- Ensure that metrics around latency and duration are collected and can be aggregated.
- Start small and add where there is customer impact.
Read more here to understand what to watch out for while designing and building an observable system.
On-premise Application Insights is a service that is compatible with Azure App Insights, but stores the data in an in-house database like PostgreSQL or object storage like Azurite.
On-premises Application Insights is useful as a drop-in replacement for Azure Application Insights in scenarios where a solution must be cloud deployable but must also support on-premises disconnected deployment scenarios.
On-premises Application Insights is also useful for testing telemetry integration. Issues related to telemetry can be hard to catch since often these integrations are excluded from unit-test or integration test flows due to it being non-trivial to use a live Azure Application Insights resource for testing, e.g. managing the lifetime of the resource, having to ignore old telemetry for assertions, if a new resource is used it can take a while for the telemetry to show up, etc. The On-premise Application Insights service can be used to make it easier to integrate with an Azure Application Insights compatible API endpoint during local development or continuous integration without having to spin up a resource in Azure. Additionally, the service simplifies integration testing of asynchronous workflows such as web workers since integration tests can now be written to assert against telemetry logged to the service, e.g. assert that no exceptions were logged, assert that some number of events of a specific type were logged within a certain time-frame, etc.