forked from aws/aws-toolkit-vscode
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
telemetry(amazonq): calculate % of non-generated (user-written) code a…
…ws#5991 ## Problem With the release of many Q features(Inline Suggestion, chat, inline chat, /dev, /test, /doc, /review, /transform), we need to know the % code written by all Q features. This requires calculating and reporting the user written code. The reporting of the code contribution of each Q features was already implemented. ## Solution Calculate and report the user written code for each language by listening to document change events while Q is not making changes to the editor. We add flags to know whether Q is making temporary changes for suggestion rendering or Q suggestion is accepted, by doing so, the document change events are coming from the user. We ignore certain document changes when their length of new characters exceeds 50. Previous data driven research has shown that user tend to copy a huge file from one place to another, making the user written code count skyrocketing but that is actually some existing code not written by the user. We plan to first collect data from IDEs and let it run in the background in shadow mode before we finish the service side aggregation, fix possible bugs and eventually present the AI code written % to the customers. Note: The JB PR aws/aws-toolkit-jetbrains#5215. The JB implementation depends on a reliable JB internal message bus to pass information. Using VSC event listener might mess up the boolean state of Q editing or not.
- Loading branch information
Showing
18 changed files
with
431 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
194 changes: 194 additions & 0 deletions
194
packages/amazonq/test/unit/codewhisperer/tracker/userWrittenCodeTracker.test.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,194 @@ | ||
| /*! | ||
| * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import assert from 'assert' | ||
| import * as sinon from 'sinon' | ||
| import * as vscode from 'vscode' | ||
| import { UserWrittenCodeTracker, TelemetryHelper, AuthUtil } from 'aws-core-vscode/codewhisperer' | ||
| import { createMockDocument, resetCodeWhispererGlobalVariables } from 'aws-core-vscode/test' | ||
|
|
||
| describe('userWrittenCodeTracker', function () { | ||
| describe('isActive()', function () { | ||
| afterEach(async function () { | ||
| await resetCodeWhispererGlobalVariables() | ||
| UserWrittenCodeTracker.instance.reset() | ||
| sinon.restore() | ||
| }) | ||
|
|
||
| it('inactive case: telemetryEnable = true, isConnected = false', function () { | ||
| sinon.stub(TelemetryHelper.instance, 'isTelemetryEnabled').returns(true) | ||
| sinon.stub(AuthUtil.instance, 'isConnected').returns(false) | ||
| assert.strictEqual(UserWrittenCodeTracker.instance.isActive(), false) | ||
| }) | ||
|
|
||
| it('inactive case: telemetryEnabled = false, isConnected = false', function () { | ||
| sinon.stub(TelemetryHelper.instance, 'isTelemetryEnabled').returns(false) | ||
| sinon.stub(AuthUtil.instance, 'isConnected').returns(false) | ||
| assert.strictEqual(UserWrittenCodeTracker.instance.isActive(), false) | ||
| }) | ||
|
|
||
| it('active case: telemetryEnabled = true, isConnected = true', function () { | ||
| sinon.stub(TelemetryHelper.instance, 'isTelemetryEnabled').returns(true) | ||
| sinon.stub(AuthUtil.instance, 'isConnected').returns(true) | ||
| assert.strictEqual(UserWrittenCodeTracker.instance.isActive(), true) | ||
| }) | ||
| }) | ||
|
|
||
| describe('onDocumentChange', function () { | ||
| let tracker: UserWrittenCodeTracker | undefined | ||
|
|
||
| beforeEach(async function () { | ||
| await resetCodeWhispererGlobalVariables() | ||
| tracker = UserWrittenCodeTracker.instance | ||
| if (tracker) { | ||
| sinon.stub(tracker, 'isActive').returns(true) | ||
| } | ||
| }) | ||
|
|
||
| afterEach(function () { | ||
| sinon.restore() | ||
| UserWrittenCodeTracker.instance.reset() | ||
| }) | ||
|
|
||
| it('Should skip when content change size is more than 50', function () { | ||
| if (!tracker) { | ||
| assert.fail() | ||
| } | ||
| tracker.onQFeatureInvoked() | ||
| tracker.onTextDocumentChange({ | ||
| reason: undefined, | ||
| document: createMockDocument(), | ||
| contentChanges: [ | ||
| { | ||
| range: new vscode.Range(0, 0, 0, 600), | ||
| rangeOffset: 0, | ||
| rangeLength: 600, | ||
| text: 'def twoSum(nums, target):\nfor '.repeat(20), | ||
| }, | ||
| ], | ||
| }) | ||
| assert.strictEqual(tracker.getUserWrittenCharacters('python'), 0) | ||
| assert.strictEqual(tracker.getUserWrittenLines('python'), 0) | ||
| }) | ||
|
|
||
| it('Should not skip when content change size is less than 50', function () { | ||
| if (!tracker) { | ||
| assert.fail() | ||
| } | ||
| tracker.onQFeatureInvoked() | ||
| tracker.onTextDocumentChange({ | ||
| reason: undefined, | ||
| document: createMockDocument(), | ||
| contentChanges: [ | ||
| { | ||
| range: new vscode.Range(0, 0, 0, 49), | ||
| rangeOffset: 0, | ||
| rangeLength: 49, | ||
| text: 'a = 123'.repeat(7), | ||
| }, | ||
| ], | ||
| }) | ||
| tracker.onTextDocumentChange({ | ||
| reason: undefined, | ||
| document: createMockDocument('', 'test.java', 'java'), | ||
| contentChanges: [ | ||
| { | ||
| range: new vscode.Range(0, 0, 1, 3), | ||
| rangeOffset: 0, | ||
| rangeLength: 11, | ||
| text: 'a = 123\nbcd', | ||
| }, | ||
| ], | ||
| }) | ||
| assert.strictEqual(tracker.getUserWrittenCharacters('python'), 49) | ||
| assert.strictEqual(tracker.getUserWrittenLines('python'), 0) | ||
| assert.strictEqual(tracker.getUserWrittenCharacters('java'), 11) | ||
| assert.strictEqual(tracker.getUserWrittenLines('java'), 1) | ||
| assert.strictEqual(tracker.getUserWrittenLines('cpp'), 0) | ||
| }) | ||
|
|
||
| it('Should skip when Q is editing', function () { | ||
| if (!tracker) { | ||
| assert.fail() | ||
| } | ||
| tracker.onQFeatureInvoked() | ||
| tracker.onQStartsMakingEdits() | ||
| tracker.onTextDocumentChange({ | ||
| reason: undefined, | ||
| document: createMockDocument(), | ||
| contentChanges: [ | ||
| { | ||
| range: new vscode.Range(0, 0, 0, 30), | ||
| rangeOffset: 0, | ||
| rangeLength: 30, | ||
| text: 'def twoSum(nums, target):\nfor', | ||
| }, | ||
| ], | ||
| }) | ||
| tracker.onQFinishesEdits() | ||
| tracker.onTextDocumentChange({ | ||
| reason: undefined, | ||
| document: createMockDocument(), | ||
| contentChanges: [ | ||
| { | ||
| range: new vscode.Range(0, 0, 0, 2), | ||
| rangeOffset: 0, | ||
| rangeLength: 2, | ||
| text: '\na', | ||
| }, | ||
| ], | ||
| }) | ||
| assert.strictEqual(tracker.getUserWrittenCharacters('python'), 2) | ||
| assert.strictEqual(tracker.getUserWrittenLines('python'), 1) | ||
| }) | ||
|
|
||
| it('Should not reduce tokens when delete', function () { | ||
| if (!tracker) { | ||
| assert.fail() | ||
| } | ||
| const doc = createMockDocument('import math', 'test.py', 'python') | ||
|
|
||
| tracker.onQFeatureInvoked() | ||
| tracker.onTextDocumentChange({ | ||
| reason: undefined, | ||
| document: doc, | ||
| contentChanges: [ | ||
| { | ||
| range: new vscode.Range(0, 0, 0, 1), | ||
| rangeOffset: 0, | ||
| rangeLength: 0, | ||
| text: 'a', | ||
| }, | ||
| ], | ||
| }) | ||
| tracker.onTextDocumentChange({ | ||
| reason: undefined, | ||
| document: doc, | ||
| contentChanges: [ | ||
| { | ||
| range: new vscode.Range(0, 0, 0, 1), | ||
| rangeOffset: 0, | ||
| rangeLength: 0, | ||
| text: 'b', | ||
| }, | ||
| ], | ||
| }) | ||
| assert.strictEqual(tracker.getUserWrittenCharacters('python'), 2) | ||
| tracker.onTextDocumentChange({ | ||
| reason: undefined, | ||
| document: doc, | ||
| contentChanges: [ | ||
| { | ||
| range: new vscode.Range(0, 0, 0, 1), | ||
| rangeOffset: 1, | ||
| rangeLength: 1, | ||
| text: '', | ||
| }, | ||
| ], | ||
| }) | ||
| assert.strictEqual(tracker.getUserWrittenCharacters('python'), 2) | ||
| }) | ||
| }) | ||
| }) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.