Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Application Insights warning on llm model deprecation #2524

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
+55 −0
Open

Application Insights warning on llm model deprecation #2524

Changes from 14 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
8068a6e
Slice 548931: [AI][Public Preview]Copilot Toolkit: Targeted ISV commu…
Oct 31, 2024
07d1124
added current deprecated models to deprecation list
Nov 1, 2024
4caeb89
modified dates on model deprecation list to be deprecated from novemb…
Nov 1, 2024
22f3f66
corrected deprecated models from gpt4o to gpt4
Nov 15, 2024
0f7c2f8
Added EventId to log message
Nov 22, 2024
d3e8d63
Apply suggestions from code review
christian-andersen-msft Nov 27, 2024
46fd9f0
Merge commit 'f9210ad33ddccfdcc713e52e8d4bcf7d58839e27' of https://gi…
Dec 2, 2024
86d84dd
Merge branch 'main' of https://github.com/microsoft/BCApps into isv-t…
Dec 2, 2024
10ba1ee
Added debug telemetry 0000AD0
christian-andersen-msft Dec 6, 2024
a6be283
Added additional telemetry tests
christian-andersen-msft Dec 6, 2024
9388ab9
Merge branch 'main' into isv-telemetry-on-llm-deprecation-rebranch
christian-andersen-msft Dec 9, 2024
bbc44ef
temporarely removed duplicate CustomDimension entry
christian-andersen-msft Dec 10, 2024
6b5d7fb
Cleaned up code for release
christian-andersen-msft Dec 10, 2024
e7cc871
Merge branch 'main' into isv-telemetry-on-llm-deprecation-rebranch
christian-andersen-msft Dec 11, 2024
31bb828
Modified model deprecation log message, added gpt4o deprecation dates…
christian-andersen-msft Jan 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#if not CLEAN25
using System.Environment;
#endif
using System.Telemetry;

codeunit 7769 "AOAI Deployments Impl"
{
Expand All @@ -15,11 +16,15 @@
InherentPermissions = X;

var
Telemetry: Codeunit Telemetry;
UnableToGetDeploymentNameErr: Label 'Unable to get deployment name, if this is a third party capability you must specify your own deployment name. You may need to contact your partner.';
GPT4oLatestLbl: Label 'gpt-4o-latest', Locked = true;
GPT4oPreviewLbl: Label 'gpt-4o-preview', Locked = true;
GPT4oMiniLatestLbl: Label 'gpt-4o-mini-latest', Locked = true;
GPT4oMiniPreviewLbl: Label 'gpt-4o-mini-preview', Locked = true;
DeprecatedDeployments: Dictionary of [Text, Date];
DeprecationDatesInitialized: Boolean;
DeprecationMessageLbl: Label 'Deployment %1 deprecated from %2. Check out codeunit 7768 AOAI Deployments', Comment = 'Telemetry message where %1 is the name of the deployment and %2 is the date of deprecation';
christian-andersen-msft marked this conversation as resolved.
Show resolved Hide resolved
christian-andersen-msft marked this conversation as resolved.
Show resolved Hide resolved
#if not CLEAN25
GPT4LatestLbl: Label 'gpt-4-latest', Locked = true;
GPT4PreviewLbl: Label 'gpt-4-preview', Locked = true;
Expand Down Expand Up @@ -103,12 +108,55 @@
exit(GetDeploymentName(GPT4oMiniLatestLbl, CallerModuleInfo));
end;

// Initializes dictionary of deprecated models
local procedure InitializeDeploymentDeprecationDates()
begin
if DeprecationDatesInitialized then
exit;

// Add deprecated deployments with their deprecation dates here
DeprecatedDeployments.Add(Turbo0301SaasLbl, DMY2Date(1, 11, 2024));

Check failure on line 118 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'Turbo0301SaasLbl' does not exist in the current context.
christian-andersen-msft marked this conversation as resolved.
Show resolved Hide resolved
DeprecatedDeployments.Add(GPT40613SaasLbl, DMY2Date(1, 11, 2024));

Check failure on line 119 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'GPT40613SaasLbl' does not exist in the current context.
DeprecatedDeployments.Add(Turbo0613SaasLbl, DMY2Date(1, 11, 2024));

Check failure on line 120 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'Turbo0613SaasLbl' does not exist in the current context.
DeprecatedDeployments.Add(GPT35TurboLatestLbl, DMY2Date(1, 11, 2024));

Check failure on line 121 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'GPT35TurboLatestLbl' does not exist in the current context.
DeprecatedDeployments.Add(GPT35TurboPreviewLbl, DMY2Date(1, 11, 2024));

Check failure on line 122 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'GPT35TurboPreviewLbl' does not exist in the current context.
DeprecatedDeployments.Add(GPT4PreviewLbl, DMY2Date(1, 11, 2024));

Check failure on line 123 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'GPT4PreviewLbl' does not exist in the current context.
DeprecatedDeployments.Add(GPT4LatestLbl, DMY2Date(1, 11, 2024));

Check failure on line 124 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'GPT4LatestLbl' does not exist in the current context.

DeprecationDatesInitialized := true;
end;

// Application Insights telemetry on deprecated models
local procedure LogDeprecationTelemetry(DeploymentName: Text)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we consider routing this through the CopilotTelemetry codeunit? It's a single instance codeunit so we would only initialize the dict once, and we could also then emit information about the current Copilot capability that is used (so you will be able to see from telemetry which capability is using the deprecated model)

var
CustomDimensions: Dictionary of [Text, Text];
IsDeprecated: Boolean;
DeprecatedDate: Date;
begin
InitializeDeploymentDeprecationDates();
IsDeprecated := DeprecatedDeployments.ContainsKey(DeploymentName);
if IsDeprecated then begin
DeprecatedDate := DeprecatedDeployments.Get(DeploymentName);
CustomDimensions.Add('DeploymentName', DeploymentName);
CustomDimensions.Add('DeprecationDate', Format(DeprecatedDate));
Telemetry.LogMessage('0000AD1',
StrSubstNo(DeprecationMessageLbl, DeploymentName, DeprecatedDate),
Verbosity::Warning,
DataClassification::SystemMetadata,
Enum::"AL Telemetry Scope"::All,
CustomDimensions);
end;
end;

local procedure GetDeploymentName(DeploymentName: Text; CallerModuleInfo: ModuleInfo): Text
var
AzureOpenAiImpl: Codeunit "Azure OpenAI Impl";
CurrentModuleInfo: ModuleInfo;
begin
LogDeprecationTelemetry(DeploymentName);

NavApp.GetCurrentModuleInfo(CurrentModuleInfo);

if (CallerModuleInfo.Publisher <> CurrentModuleInfo.Publisher) and not AzureOpenAiImpl.IsTenantAllowlistedForFirstPartyCopilotCalls() then
Error(UnableToGetDeploymentNameErr);

Expand Down
Loading