Use LaunchDarkly feature flags

[rhauch]

Goals

1. Decouple feature release from deployment. This allows code to be shipped and deployed regardless of when the feature is ready.
2. More granular control over who we release features to. Feature flags act as a metaphorical gate to limit which and how many users see a feature, and make it possible to coordinate rollouts of a feature or set of features across multiple services or application. Feature flags also make it easy to temporarily or permanently disable features or versions of released applications that customers have installed in their environments.
3. Data driven experimentation (Stretch). Enable the use of A/B tests and experiments in production to drive measurable outcomes.

Non-Goals

1. Use feature flags for every new feature. Features can be implemented and made available in new versions of the extension, without using feature flags, and we will continue to do this for most features. By introducing feature flags, we make it possible to use feature flags where it makes sense.
2. Durable device identifier. Instead, we want an anonymous identifier that is generated each time the sidecar starts up.

Background

Using feature flags gives us some control over the behavior of already-released versions. Feature flags help decouple the enablement of features from deployments (i.e., releasing and installation by users), and we could use them this way. But more importantly it also allows us to turn off certain features in already-released versions if/when those features are discovered to be problematic.

The first step is adding support for using any feature flags to the Confluent for VS Code extension and sidecar. After that, we can choose when, where and how we want to use specific feature flags.

Example uses of feature flags

Here are some examples of how the Confluent for VS Code extension and sidecar might use feature flags:

1. Disable specific versions of Confluent for VS Code; aka, the “kill switch”. This gives us the ability to disable old versions after a designated period of time (e.g., 6 months), or killing a specific version after we discover a severe vulnerability. We don't anticipate having to do either, but using a feature flag gives us the option should we need it. This is especially true since we can't control which versions users have installed, especially since it's possible for users to install old releases that are published on the VS Code Marketplace or GitHub.
2. Disabling specific features in VS Code, across multiple versions. For example, we might choose to disable all versions of the extension from emitting telemetry if we've discovered certain versions are problematic (e.g., emitting far too much telemetry). We're currently using Sentry, and at some point we may want to move off of Sentry and may want to disable the use of Sentry in already published versions. Or in the case of a significant CCloud outage, we might want to temporarily prevent Confluent for VS Code users from logging into CCloud, if we know those attempts will fail and to help reduce the load on CCloud as is it recovered.
3. Sending notices or announcements to users of all/specific versions of Confluent for VS Code. For example, sending notice of a significant outage or event. This may be combined with other use cases (e.g., Fix Semaphore build status badge URL #3 above).
4. Coordinating extension features with CCloud feature launches. Some CCloud features are rolled out by region, by organization, or by other schedules. When we add functionality related to those CCloud features to the Confluent for VS Code extension, we would likely want to expose that functionality to users only if they can also use the CCloud feature.
5. Rolling out new features in a coordinated manner. We may prefer to roll out a new feature gradually, starting with a limited set of users before enabling the feature for all users.

Precedence

The Confluent CLI uses LaunchDarkly feature flags for all the reasons mentioned above.

Limitations

The major limitation for an open source project is that only some Confluent employees have access to the LaunchDarkly projects where the feature flags are defined and evaluated.

Proposal

Requirements

• VS Code extension will evaluate the kill switch flag, before starting up the sidecar.
• Sidecar will evaluate LD feature flags.
• Use new IDE-specific feature flags and CCloud-specific feature flags.
• Include in LD flag evaluation contexts:
  • anonymous device IDs generated by each sidecar process
  • CCloud user ID and organization ID when authenticated
• Use LaunchDarkly client-side ID, which is not a secret and can be included in code.
• Extension uses LaunchDarkly NodeJS client-side SDK or JavaScript client-side SDK
• Sidecar polls and caches HTTP API evaluations (no client-side SDK for Java)
• Confluent for VS Code extension will sent the VS Code version and extension version to the sidecar in the run command via system properties or environment variables, so these are available for LD contexts upon sidecar startup.

User experience

For the most part, feature flags should be transparent to all users. The user experience will simply include the features that are enabled, while users are not aware of features that are disabled.

There are a few exceptions that help move users off of old extension versions.

• Deprecated versions of Confluent for VS Code. When a user is using a version of Confluent for VS Code that has been deprecated, upon extension activation the user should see a notice stating this and recommend they upgrade to the latest version. However, the extension continues to operate as usual.
• Disabled Versions of Confluent for VS Code. When a user is using a version of Confluent for VS Code that has been disabled, upon extension activation the user should see a notice stating this and suggest they upgrade to the latest version. Also, the extension does not really operate, as it is disabled.

Basic design

The sidecar will contain the bulk of the functionality to use LaunchDarkly to evaluate feature flags. The sidecar can internally use this mechanism to base decisions upon the values of feature flags. The sidecar will also expose a single endpoint that the VS Code extension can use to obtain the current cached value for a specific feature flag, as evaluated with the current context:

GET /gateway/v1/feature-flags/{id}/value

returns the current value of the specified feature flag evaluated for the current context, in its JSON representation (one of object, string, number, boolean, array, or null). This method will fail with 404 Not Found if a flag with the specified id is unknown.

Note that the Authorization: Bearer <sessionToken> header with valid session token must be provided by the extension. If the session token is invalid, this request will fail with 401 Unauthorized.

LD Clients

LaunchDarkly has two kinds of SDKs: client-side SDKs and server-side SDKs. The sidecar and VS Code extensions are both client-side applications, meaning they should only use LD client-side SDKs. Never use server-side SDKs in client applications.

LD clients in the VS Code extension

The VS Code extension will need to initialize singleton client for the IDE project and with the project’s client-side ID, the client configuration options, and appropriate flag evaluation context for the project and installation (see above). After it has initialized the client, the extension can safely call variation to access the IDE feature flags, an in particular the ide.global.enable flag that should be evaluated before the extension starts the sidecar.

Although other feature flag evaluations for the IDE project could be issued this same way, the extension will instead use the new REST API of the sidecar to evaluate feature flags. This simplifies the logic related to feature flags in the VS Code extension, and encapsulates the (non-trivial) logic of evaluating and caching flag evaluations for the IDE project and Confluent Cloud projects, and for maintaining default values in case LaunchDarkly is not available.

LD clients for the IDE Sidecar

LaunchDarkly does not have a client-side SDK for Java, and the server-side Java SDK is not to be used in client applications like the sidecar. In these cases, LaunchDarkly recommends using the LaunchDarkly service APIs. Therefore, the sidecar will use a simple HTTP client to poll the flag evaluation results, cache them, and use them whenever the sidecar needs to evaluate a flag internally or when the extension calls the new REST API to get the current value of a feature flag. The sidecar will refresh the flag evaluations every 60 seconds.

The sidecar can use the same HTTP client to poll the flag evaluation results for multiple projects, since the client-side ID and evaluation contexts are included as path parameters in HTTP GET requests of the form:

GET https://confluent.cloud/ldapi/sdk/evalx/{client-id}/contexts/{context}

where {client-id} is the project-specific client-side ID and {context} is the base64-encoded JSON object for the evaluation context. Note this uses Confluent Cloud’s proxy for LaunchDarkly, which helps avoid problems due to ad blockers or network restrictions on user machines.

VS Code version and extension version

The sidecar will need the VS Code version and Confluent for VS Code extension version in order to add that information to the LD context. The Confluent for VS Code extension will pass these to the sidecar at startup via system properties (e.g., -Dvscode.version=20.15.1 -Dvscode.extension.version=v0.17.1) or via environment variables (e.g., VSCODE_VERSION=20.15.1 and VSCODE_EXTENSION_VERSION=v0.17.1). The sidecar will then have this information immediately upon startup. The system property will take precedence over the environment variable, and any v prefix included in the values will be ignored.

Proposed Feature Flags

Flag Name	Flag Type	Description	Default value in the sidecar
ide.global.enable	boolean	Returns whether the IDE extension is enabled, which is enabled by default. If false, the ide.global.notices should be displayed to the user. This exists primarily to allow disabling very old versions or if needed problematic versions.	true
ide.global.notices	array	Returns an object array with zero or more notices, each with message and suggestion string fields, that should be displayed to the user.	[{"message": "Unable to reach LaunchDarkly.com", "suggestion": "Some feature may not work as expected."}]
ide.ccloud.enable	boolean	True by default, but false if Confluent Cloud features should be temporarily disabled due to a significant CCloud outage. Used in conjunction with ide.global.notices.	true
ide.segment.enable	boolean	True if the extension should emit Sentry events, or false otherwise. This will default to true, but can be disabled (perhaps on specific versions) if needed.	true

Handling LaunchDarkly or Network outages

It is possible that the sidecar starts up and is unable to evaluate the feature flags, either because of an ongoing LaunchDarkly.com outage or because the user's local network is not connected. In such cases, the sidecar is expected to start up quietly, and use “safe” default values for the feature flags that ship with that version of the sidecar. (The “safe” defaults for a single feature flag may change over time; initially, the default might be to disable a new feature, but after some releases the default may be the feature should be enabled in some later version of the sidecar.)

In cases of an outage or inability to reach LaunchDarkly.com, the sidecar will return the default values for the feature flags. The sidecar will then continue every 60 seconds to retry evaluating the feature for the current context; only upon success will the feature flag values be cached in the sidecar and used when feature flags are evaluated. If an outage occurs after the sidecar has cached feature flag values, those cached values will continue to be used until the outage ends or the sidecar process terminates.

LD Contexts used for evaluating flags

The LaunchDarkly contexts provide information about the user, device, etc., and are used when evaluating the feature flags. Each LD project is defined with the allowed context kinds, and feature flags in the project can define variations that use the attributes of those contexts.

Device context

An LD context with device context kind and the following attributes will be used for all flag evaluations:

Attribute	Description
key	The context kind and key must uniquely identify the context. We will generate a new v4 UUID for the key of each new context.
sidecar.version	The version of the IDE Sidecar application, such as 0.26.0. This does not have the v prefix used in GitHub tags and versions, so this should allow us to use LD’s semantic version comparisons.
vscode.extension.version	The version of the extension, such as 0.16.1. This does not have the v prefix used in GitHub tags and versions, so this should allow us to use LD’s semantic version comparisons.
vscode.version	The version of VS Code, such as 20.15.1. This does not have the v prefix used in GitHub tags and versions, so this should allow us to use LD’s semantic version comparisons.
os.name	The name of the operating system, such as macOS, Windows, Ubuntu Linux, etc.
os.version	The version of the operating system, such as 13.6.7 (for macOS), 10.0.1904562 (for Windows), 22.0413 (for Ubuntu Linux), etc. This does not have the v prefix used in GitHub tags and versions, so this should allow us to use LD’s semantic version comparisons.
os.type	The type of operating system; one of Windows, MacOS, Linux, or Other.

The device context is the same for the lifetime of the sidecar.

Confluent Cloud context

When a user creates a connection to CCloud, the LD context used to evaluate feature flags will be a multi-context with the device context and a user context with these attributes:

Attribute	Description
key	The context kind and key must uniquely identify the context. The key will be the user resource ID of the authenticated CCloud user.
user.resource_id	The user resource ID of the authenticated CCloud user.
org.resource_id	The organization resource ID of the authenticated CCloud user.
email	The email address of the authenticated CCloud user.

When a user does not have a CCloud connection, the LD context will just consist of the device context.