Code

Add GenAI Preprocessor logic#2314

Code

Merged

alanpo1 merged 72 commits intomainAzure/azureml-assets:mainfrom alanpoblette/SpanLogPreprocessorAzure/azureml-assets:alanpoblette/SpanLogPreprocessorCopy head branch name to clipboardFeb 20, 2024

+1,051-3

Commits

Commits on Feb 12, 2024

• add sample raw logs to e2e test resources

  alanpo1

  committed
  e9bdc0e

  Copy full SHA for e9bdc0e

• working on genai preprocessor

  alanpo1

  committed
  55db58e

  Copy full SHA for 55db58e

Commits on Feb 13, 2024

• adding logic for genAI preprocessor

  alanpo1

  committed
  0f755de

  Copy full SHA for 0f755de

• adding UT tests

  alanpo1

  committed
  adaf0f5

  Copy full SHA for adaf0f5

• select cols in df to v1 schema

  alanpo1

  committed
  ee05204

  Copy full SHA for ee05204

• UT

  alanpo1

  committed
  d0885a2

  Copy full SHA for d0885a2

• fix UT

  alanpo1

  committed
  ae47cf9

  Copy full SHA for ae47cf9

• fix data

  alanpo1

  committed
  1cf5e06

  Copy full SHA for 1cf5e06

• refactor shared mdc code in genai preprocessor

  alanpo1

  committed
  9905df6

  Copy full SHA for 9905df6

• syntax fixes

  alanpo1

  committed
  2329192

  Copy full SHA for 2329192

• let all extra columns through

  alanpo1

  committed
  ac352f3

  Copy full SHA for ac352f3

• syntax

  alanpo1

  committed
  c5c7d8e

  Copy full SHA for c5c7d8e

• syntax

  alanpo1

  committed
  726c1a0

  Copy full SHA for 726c1a0

• doc syntax

  alanpo1

  committed
  ac748ad

  Copy full SHA for ac748ad

• adding e2e test

  alanpo1

  committed
  174d7a1

  Copy full SHA for 174d7a1

• adding dataref example

  alanpo1

  committed
  5369688

  Copy full SHA for 5369688

• remove unused import

  alanpo1

  committed
  252e1c0

  Copy full SHA for 252e1c0

• add file datalake pacakge to genai preprocessor

  alanpo1

  committed
  8393a10

  Copy full SHA for 8393a10

• no need to bump version. Hasn't released v1 yet

  alanpo1

  committed
  5c5ea4e

  Copy full SHA for 5c5ea4e

Commits on Feb 14, 2024

• add dataref testcase but comment out for now

  alanpo1

  committed
  719293d

  Copy full SHA for 719293d

• fix UT

  alanpo1

  committed
  1fae2fb

  Copy full SHA for 1fae2fb

• merge trace aggregator and genai preprocessor PRs into one (#2323)

  Show description for 5b936cd
  alanpo1

  authored
  5b936cd

  Copy full SHA for 5b936cd

• fix test case for to_dict()

  alanpo1

  committed
  badec08

  Copy full SHA for badec08

• add trace aggregator in preprocessor

  alanpo1

  committed
  3eb8da7

  Copy full SHA for 3eb8da7

• user_id and session_id may not be available for v0

  alanpo1

  committed
  904b044

  Copy full SHA for 904b044

• comment out user id and session id in schema

  alanpo1

  committed
  69328cb

  Copy full SHA for 69328cb

• unused import for now

  alanpo1

  committed
  05edf35

  Copy full SHA for 05edf35

• syntax

  alanpo1

  committed
  dd831ee

  Copy full SHA for dd831ee

• fix UT with less schema

  alanpo1

  committed
  e9e70be

  Copy full SHA for e9e70be

• slim down debugging show()

  alanpo1

  committed
  b93f0a2

  Copy full SHA for b93f0a2

Commits on Feb 15, 2024

• refactor trace aggregator to use more pyspark

  alanpo1

  committed
  4876d5f

  Copy full SHA for 4876d5f

• fix import to be same as others

  alanpo1

  committed
  af3e6d0

  Copy full SHA for af3e6d0

• fix UT and import stmts

  alanpo1

  committed
  00937fc

  Copy full SHA for 00937fc

• modify UT

  alanpo1

  committed
  211ee0d

  Copy full SHA for 211ee0d

• modify UT

  alanpo1

  committed
  7c9fa5c

  Copy full SHA for 7c9fa5c

• Merge branch 'main' into alanpoblette/SpanLogPreprocessor

  alanpo1

  authored
  344331b

  Copy full SHA for 344331b

• Update trace_aggregator.py

  alanpo1

  authored
  22c71d0

  Copy full SHA for 22c71d0

• fix UT

  alanpo1

  committed
  3c828ca

  Copy full SHA for 3c828ca

• fix syntax

  alanpo1

  committed
  644d198

  Copy full SHA for 644d198

• syntax

  alanpo1

  committed
  9f227fe

  Copy full SHA for 9f227fe

• refactor tree construction to include all fields from span logs

  alanpo1

  committed
  d750253

  Copy full SHA for d750253

• syntax

  alanpo1

  committed
  fb489bb

  Copy full SHA for fb489bb

• syntax fix

  alanpo1

  committed
  773a3c1

  Copy full SHA for 773a3c1

• format output of trace aggregator to fit agg trace schema

  alanpo1

  committed
  11130f7

  Copy full SHA for 11130f7

• remove unused variable

  alanpo1

  committed
  2881448

  Copy full SHA for 2881448

• unused imports

  alanpo1

  committed
  e00603e

  Copy full SHA for e00603e

• unused definition

  alanpo1

  committed
  5f3154f

  Copy full SHA for 5f3154f

• set auto for parallel pytests

  alanpo1

  committed
  d12324a

  Copy full SHA for d12324a

• Merge branch 'main' into alanpoblette/SpanLogPreprocessor

  alanpo1

  authored
  42d9b55

  Copy full SHA for 42d9b55

• Revert "set auto for parallel pytests"

  Show description for 6aaa37b
  alanpo1

  committed
  6aaa37b

  Copy full SHA for 6aaa37b

• revert main merge

  alanpo1

  committed
  ebc334a

  Copy full SHA for ebc334a

• Update model-monitoring-ci.yml

  alanpo1

  authored
  221fd01

  Copy full SHA for 221fd01

• revert auto change

  alanpo1

  committed
  8abcd34

  Copy full SHA for 8abcd34

Commits on Feb 16, 2024

• Merge branch 'alanpoblette/SpanLogPreprocessor' of https://github.com/Azure/azureml-assets into alanpoblette/SpanLogPreprocessor

  alanpo1

  committed
  daccb1a

  Copy full SHA for daccb1a

• pass schema to map function, not call it on workers

  alanpo1

  committed
  ef16206

  Copy full SHA for ef16206

• addressing some comments

  alanpo1

  committed
  5de7573

  Copy full SHA for 5de7573

• fix UT

  alanpo1

  committed
  83b507d

  Copy full SHA for 83b507d

• add new flag for calculate trace logs in preprocessor

  alanpo1

  committed
  fa89029

  Copy full SHA for fa89029

• return None column if can't find in raw logs data

  alanpo1

  committed
  6177bf0

  Copy full SHA for 6177bf0

• add UT for None columns if not in raw logs

  alanpo1

  committed
  f70745c

  Copy full SHA for f70745c

• Revert "add new flag for calculate trace logs in preprocessor"

  Show description for 1631d0e
  alanpo1

  committed
  1631d0e

  Copy full SHA for 1631d0e

• syntax

  alanpo1

  committed
  8b66f45

  Copy full SHA for 8b66f45

• syntax

  alanpo1

  committed
  70b3912

  Copy full SHA for 70b3912

Commits on Feb 17, 2024

• Remove problematic test for v1

  alanpo1

  authored
  355da83

  Copy full SHA for 355da83

Commits on Feb 20, 2024

• add trace aggregator UT

  alanpo1

  committed
  14a4d63

  Copy full SHA for 14a4d63

• Merge branch 'alanpoblette/SpanLogPreprocessor' of https://github.com/Azure/azureml-assets into alanpoblette/SpanLogPreprocessor

  alanpo1

  committed
  35141f7

  Copy full SHA for 35141f7

• setup pyspark path for each testcase

  alanpo1

  committed
  22daf2a

  Copy full SHA for 22daf2a

• remove broken UT for v1

  alanpo1

  committed
  836396b

  Copy full SHA for 836396b

• Revert "remove broken UT for v1"

  Show description for 7753121
  alanpo1

  committed
  7753121

  Copy full SHA for 7753121

• Reapply "remove broken UT for v1"

  Show description for a46ac85
  alanpo1

  committed
  a46ac85

  Copy full SHA for a46ac85

• Revert "setup pyspark path for each testcase"

  Show description for 4b618f3
  alanpo1

  committed
  4b618f3

  Copy full SHA for 4b618f3

• Revert "add trace aggregator UT"

  Show description for c3c63ac
  alanpo1

  committed
  c3c63ac

  Copy full SHA for c3c63ac