Dynamic Sampling Launch

[ale-cota]

Just because you don’t record a problem, that doesn’t mean it didn’t happen.

This week we launched Dynamic Sampling - available to all new customers, and opt-in for existing ones. Dynamic Sampling is rebuilding on our original approach to data volume scale with a focus on maximizing the value out of your performance data - automatically.

What’s new in this approach?

We started a closed beta for Dynamic Sampling back in July with a strong hypothesis that our customers know best what data is most valuable to them. This made us focus on providing fine controls to target specific transactions and configure sampling rules directly in the Sentry product.

Talking to our early adopters, we found that our users loved the idea, but were still facing the same old questions: “what sample rate should I choose?”, “how do I make sure I get the most value?”, and many wanted Sentry to be more opinionated on that. Big shoutout to all of you who provided feedback and helped us understand how to evolve this 🙂

That’s why we’ve decided to remove the micro-management of sampling rules entirely. Instead, we’ll make intelligent decisions to automatically capture the data we believe is most valuable for most customers and the right amount based on your traffic. At the same time, we wanted to keep the ability for customers to influence how data is sampled, so we’re providing a mechanism to express priorities, so they can get the most value out of the transactions we store and filter out the noise.

We’ll be adding more capabilities in the future, but we’re starting out with:

• Prioritising data from latest releases to help detect performance issues early in your roll-out process
• Prioritising data from dev environments to help you find issues before going live
• Deprioritising health checks to filter out noise

Opting In

We’re rolling this out iteratively and today it’s only live for new customers. Existing customers will be getting this functionality soon, but if you’re excited to get started, we might be able to opt you in today — just contact your account rep or [email protected].

We want to hear what you think!

We really want to get your feedback on this:

• what do you think about our new approach to sampling?
• what other kind of data do you think Sentry should retain automatically?

Please post below to let us know. If you have any questions or feedback you want to give privately, you can also email me at [email protected]. Thanks a lot!

[AbhiPrasad]

We collect CI information about all of our GitHub actions in Sentry via https://github.com/getsentry/sentry-github-actions-app - a GitHub application that @armenzg built. With this app, we instrument all of our GitHub actions and send that data to our performance product. A CI job run -> a Sentry transaction. See the project that contains this data: https://sentry.io/organizations/sentry/projects/sentry-github-action-ci-data/?project=5899451.

To get further insights, this transaction is tagged with a variety of tags we use for filtering. In particular, repo and title, which is used to tell the repository that generated ran the GitHub Action and the title of the CI workflow.

With dynamic sampling and the sampling volume at Sentry, there is a significant difference between processed and indexed transactions.

This is problematic for the CI project since we require filtering the data by specific tags to do meaningful analysis of the data. Since the sentry-github-actions-app project gets CI transactions from every single GitHub action in every Sentry repo, we require filtering by repo tag to isolate GitHub actions per repository.

Unfortunately by adding this repo tag (for ex. getsentry/sentry-javascript), we switch from processed -> indexed transactions, which means that our dashboards become much less useful. Things like failure rate, tpm all use indexed transactions when we filter by the repo tag, so are not representative of the state of CI, which makes analysis like identifying what CI jobs to improve very hard.

For example, we have a widget that we use to track the most common CI failures on the master branch. Right now in the past 30 days, only 3 indexed transactions were ingested that have the getsentry/sentry-javascript repo tag on them - that's the only count we have access to if we wanted to analyze this information in Sentry. This means we can't trust Sentry to get the correct idea of how many failing CI jobs we have (which we would use to prioritize fixing flaky CI over new features/bug fixes).

This means we cannot effectively use dashboards with the current state of dynamic sampling on our CI data. CI data is also low volume in nature, so even the indexed transactions are low volume since they get drowned out by higher volume repos (same project, but different sources for each transaction).

To fix this, I would like to either

a) Have the ability to turn off dynamic sampling for a certain project - I am willing to pay extra to get as much fidelity as possible into these transactions
b) Be able to configure what tags are attached to processed transactions, so I can filter by them (I understand this has cardinality limitations, but the discover/dashboards is hard to use without it).