TruLens as namespace package (#1303)

* decouple core trulens packages * add root poetry files * merge utils with core * move feedback to feedback functions * feedback -> feedbacks * include compiled react components * fmt * fmt * feedbacks -> external for clarity * fix dashboard dependencies * fix test imports * update dev installs for llamaindex, langchain * core cleanup * fix import * move core to trulens.core and resolve import errors * fix import errors from trulens.core * fix imports in examples * fix imports in markdown * fix docstring imports * add test case generation as subpackage * add root imports and detach TruRails * add root imports and langchain dep * add get_feedback_result and fix docstrings * add nemo/trurails subpackage * factor out get_feedback_result * fmt * update test imports * fmt and rebuild * fmt * rename to trulens-testing * add trulens-testing * remove trulens explain * move trulens_eval to root dir * factor out rag_triad to external * add testing poetry lock * root packaging * move app types to trulens.core.apps to fix circular import * fix import paths * refactor core.app.app to core.app.base * add __version__ * update formatting * fmt w ruff * lint * simple lint fixes * move instrumentation framework extensions to ext.instrument * isort rules update matching existing config * split out providers and move instrument packages * fix imports and symlinks * linting and symlink fixes * import fixes and optional dependencies * readme/docs updates * rm old dev deps * update imports * update az pipelines * use system python * disable venv * move testing out of main package * use system python * fix config path * install testing utilities * fmt * remove old makefile * fix imports and pydantic serialization * move out component views to ext * slowly deprecate unused optional imports * add httpx * add httpx as optional * mark group as optional * Docs for TruLens as namespace package (#1306) * get docs to build * site fixes * drop utils * drop double intro * drop accidental copy * more fixes --------- Co-authored-by: Josh Reini <[email protected]> Co-authored-by: corey hu <[email protected]> * remove langchain dependency from feedback prompts * fix broken symlinks and types * factor out utils * restore serial * move testing to benchmark * bundle utils into core * mvoe trulens.testing to trulens.benchmark * rollout trulens.ext namespace to trulens.instrument and trulens.providers * fix symlinks * fmt * fix select imports * update makefile, testing, and mypy * update pipeline pytest commands * add lock creation * testing fixes * add __init__ * add ipywidgets and traitlets to deps * dashboard fmt * remove pinecone from unittests * misc fixes * update pipeline and testing to use makefile * add coverage tools * add poetry install * only install tests group * set langchain version * update langchain dep * allow ipywidgets in static tests * disable nemoguardrails for python3.12 * mod name as unnested * skip nemo on 3.12 * skip if > * use makefile for release pipeline * linting fixes * enable linting hook * pyproject formatter * update lock * add release process makefile targets * misc changes to vsc environment * use import paths for docs * replace docs references to trulens_eval * simplify docs paths where possible * add py.typed to subpackages to indicate inline typing * restore releases * add examples and autogen api reference on docs * update notebook pip installs * update dashboard build and fragment decorator * combine huggingface with huggingfacelocal * add __version__ to all packages * providers package name should match namespace * fix guardrails imports * relax on minor version instead of patch * fix OMP error and add package metadata * bump all subpackages to 1.0.0 * fmt * move optional test dependencies to new group * add snowflake as trulens-core extra * make BaseEmbedding optional * fix importlib failures on py3.9 * fix importlib handling under 3.10 * resolve langchain(core/community/openai) versions * fix docs reading notebook build and examples * report pip and python version in pipeline * fix pip version * add make command to publish to pypi and testpypi * truera -> snowflake references * set version 1.0.1a0 and allow prerelease deps * move psutils * update lockfiles * ignore build venvs and mypy * add migration guide * update dev guide and add toml extension * fix old code in litellm example * remove requirements from packaging * remove remaning trulens_eval references * dogfood dev and build fixes * update notebooks using optional provider * docs fixes * fix links * add meta.yaml * add page on uninstalling * update locks * remove App.select_context() * remove select_context impl * typos * add note about python version * Update README.md * make build path relative from recipe * pr fixes * fmt * fix confirm * clarify sync flag * add build section * templatize ci pipelines * try camelCase * split off optional test template * remove old reference * run precommit check on 3.11 * undo unnecessary quote typing * refresh locked versions of scipy and sklearn for benchmarking * pr fixes * spacing * include score range fix * prompt type to str change compatibility * move react source out of package and build components on every install * adapted tru benchmark experiment class to repackaging changes + updated notebook * fix build path * nodejs instructions * update download instructions * fix package data path --------- Co-authored-by: Josh Reini <[email protected]> Co-authored-by: Josh Reini <[email protected]> Co-authored-by: Daniel <[email protected]>
truera · Aug 9, 2024 · a3ceeae · a3ceeae
1 parent 4cadb05
commit a3ceeae
Show file tree

Hide file tree

Showing 712 changed files with 48,202 additions and 151,218 deletions.
diff --git a/.azure_pipelines/README.md b/.azure_pipelines/README.md
@@ -3,14 +3,12 @@
 Note that the pipeline definitions in this folder are for azure pipelines, not
 github pipelines. There are differences between these systems.
 
-- `ci-eval-pr.yaml` is run for all PRs to _TruLens-Eval_. Success is needed for
+- `ci-eval-pr.yaml` is run for all PRs to _TruLens_. Success is needed for
   merging into `main`.
-- `ci-eval.yaml` for _TruLens-Eval_ releases. This includes database migration
+- `ci-eval.yaml` for _TruLens_ releases. This includes database migration
   tests as well as running notebooks. Success is needed for merging into
   `releases/*`. Also, any branch named `releases/*` needs to pass the pipeline
   tests before merging into `main`.
-- `ci.yaml` is run for all PRs to _TruLens-Explain_. Success is needed for
-  merging into `main`.
 
 ## More information
 

diff --git a/.azure_pipelines/ci-docs-pr.yaml b/.azure_pipelines/ci-docs-pr.yaml
@@ -23,7 +23,8 @@ jobs:
     strategy:
       matrix:
         latest:
-          python-version: "3.12"
+          # Snowflake libraries are not available on >=3.12.
+          python-version: "3.11"
 
     steps:
       - checkout: self

diff --git a/.azure_pipelines/ci-eval-pr.yaml b/.azure_pipelines/ci-eval-pr.yaml
@@ -4,14 +4,9 @@
 trigger: none
 
 pr:
-  # Run this pipeline for any PR that wishes to push to main that changes code
-  # inside trulens_eval folder.
   branches:
     include:
       - main
-  paths:
-    include:
-      - trulens_eval
 
 jobs:
   - job: PRBranchProtect # name seems to be important but cannot figure out why
@@ -23,142 +18,19 @@ jobs:
       matrix:
         default: # name is important
           python-version: 3.12
-          tests-folder: tests/unit
-        py308-static:
-          python-version: 3.8
-          tests-folder: tests/unit/static
         py309-static:
           python-version: 3.9
-          tests-folder: tests/unit/static
         py310-static:
           python-version: 3.10
-          tests-folder: tests/unit/static
         py311-static:
           python-version: 3.11
-          tests-folder: tests/unit/static
 
     steps:
-      - checkout: self
-        clean: true
-
-      - bash: |
-          echo "##vso[task.prependpath]$CONDA/bin"
-
-          set -e
-          conda create -y --quiet \
-            -n $(python-version) \
-            python=$(python-version)
-
-          conda init bash
-
-        displayName: Setup conda
-
-      - bash: |
-          set -e
-          source activate $(python-version)
-
-          pip install -r requirements.dev.txt --verbose
-
-          pre-commit run --all-files --show-diff-on-failure
-        condition: eq(variables['python-version'], 3.12) # only run format check with latest python version
-        displayName: Format diffs
-
-      - bash: |
-          set -e
-          source activate $(python-version)
-
-          cd ./trulens_eval
-          pip install -e . --verbose
-
-        displayName: Install trulens
-
-      - bash: |
-          set -e
-          source activate $(python-version)
-
-          pip install -r requirements.test.txt --verbose
-
-        displayName: Install testing packages
-
-      - bash: |
-          set -e
-          source activate $(python-version)
-
-          python --version
-          pip --version
-          pip list --verbose
-
-        displayName: Describe python env
-
-      - bash: |
-          set -e
-          source activate $(python-version)
-
-          cd ./trulens_eval
-          python -m pytest --test-run-title="Required $(python-version) unit tests" $(tests-folder)
-
-        displayName: Unit tests with required packages
-
-      - bash: |
-          set -e
-          source activate $(python-version)
-
-          cd ./trulens_eval
-
-          # Once we start running notebooks in these tests, might have to do this again:
-          # pip install ipykernel
-          # python -m ipykernel install --user \
-          #  --name $(python-version) \
-          #  --display-name $(python-version)
-
-          pip install --verbose -r trulens_eval/requirements.optional.txt
-
-          python --version
-          pip --version
-          pip list --verbose
-
-        displayName: Install optional packages
-
-        # Models: Snowflake Cortex; currently only supports Python 3.8, 3.9, 3.10, and 3.11.
-        # https://docs.snowflake.com/en/developer-guide/snowpark/python/setup#prerequisites
-      - bash: |
-          set -e
-          source activate $(python-version)
-          echo "Installing snowflake-snowpark-python for python $(python-version)"
-
-          if [[ "$(echo $(python-version) | cut -d'.' -f1)" -eq "3" && "$(echo $(python-version) | cut -d'.' -f2)" -lt "12" ]]; then
-              cd ./trulens_eval
-              pip install --verbose "snowflake-snowpark-python>=1.18.0"
-              pip list --verbose
-          else
-              echo "Python version is 3.12 or higher. Skipping installation."
-          fi
-        displayName: Install snowflake-snowpark-python for python < 3.12
-
-      - bash: |
-          set -e
-          source activate $(python-version)
-
-          cd ./trulens_eval
-
-          python -m pytest  --test-run-title="Optional $(python-version) unit tests" $(tests-folder)
-        env:
-          # enables optional tests, see tests/unit/test.py
-          TEST_OPTIONAL: true
-
-          # tests make use of various APIs:
-          OPENAI_API_KEY: $(OPENAI_API_KEY)
-          HUGGINGFACE_API_KEY: $(HUGGINGFACE_API_KEY)
-          PINECONE_API_KEY: $(PINECONE_API_KEY)
-          PINECONE_ENV: $(PINECONE_ENV)
-          HUGGINGFACEHUB_API_TOKEN: $(HUGGINGFACEHUB_API_TOKEN)
-
-          # some tests log into Snowflake.
-          SNOWFLAKE_ACCOUNT: $(SNOWFLAKE_ACCOUNT)
-          SNOWFLAKE_USER: $(SNOWFLAKE_USER)
-          SNOWFLAKE_USER_PASSWORD: $(SNOWFLAKE_USER_PASSWORD)
-          SNOWFLAKE_DATABASE: $(SNOWFLAKE_DATABASE)
-          SNOWFLAKE_ROLE: $(SNOWFLAKE_ROLE)
-          SNOWFLAKE_WAREHOUSE: $(SNOWFLAKE_WAREHOUSE)
-
-        displayName: Unit tests with optional packages
+      - template: templates/env-setup.yaml
+      - template: templates/run-precommits.yaml
+      - template: templates/run-tests.yaml
+        parameters:
+          testSuite: unit
+      - template: templates/run-optional-tests.yaml
+        parameters:
+          testSuite: unit
diff --git a/.azure_pipelines/ci-eval.yaml b/.azure_pipelines/ci-eval.yaml
@@ -1,22 +1,10 @@
 trigger: none
 
 pr:
-  # Run this pipeline for any PR from a branch that starts with "releases/" that
-  # changed code in trulens_eval folder or to eval notebooks in the docs folder.
+  # Run this pipeline for any PR from a branch that starts with "releases/"
   branches:
     include:
       - releases/*
-  paths:
-    include:
-      - trulens_eval
-      - docs/trulens_eval/*.ipynb
-
-# schedules:
-#  - cron: "0 0 * * *"
-#    displayName: Daily midnight build
-#    branches:
-#      include:
-#        - main
 
 jobs:
   - job: ReleaseTests
@@ -27,92 +15,14 @@ jobs:
     strategy:
       matrix:
         latest:
-          pythonVersion: "3.11"
-          condaEnvFileSuffix: "trulens-llm"
-
+          python-version: 3.11
 
     steps:
-      - checkout: self
-        clean: true
-
-      - bash: echo "##vso[task.prependpath]$CONDA/bin"
-        displayName: Add conda to PATH
-
-      - bash: |
-          set -e
-          conda create  -y --quiet -n $(condaEnvFileSuffix) python=3.11
-          conda init bash
-        displayName: Create Anaconda environment
-
-      - bash: |
-          set -e
-          source activate $(condaEnvFileSuffix)
-          which python
-          python --version
-        displayName: Which Python
-
-      - bash: |
-          set -e
-          source activate $(condaEnvFileSuffix)
-          pip install -r requirements.dev.txt
-
-          cd ./trulens_eval
-          pip install -e .
-          pip install -r tests/docs_notebooks/requirements.txt
-          pip install ipykernel
-          python -m ipykernel install --user --name $(condaEnvFileSuffix) --display-name $(condaEnvFileSuffix)
-        displayName: Install pip package
-
-      - bash: |
-          set -e
-          source activate $(condaEnvFileSuffix)
-          pre-commit run --all-files --show-diff-on-failure
-        displayName: Format Code
-
-      - bash: |
-          source activate $(condaEnvFileSuffix)
-          cd ./docs/trulens_eval
-          pip install jupyter
-        displayName: Install jupyter
-
-      - bash: |
-          source activate $(condaEnvFileSuffix)
-
-          cd ./trulens_eval
-          pip install pytest==7.0.1 pytest-azurepipelines
-          pip install -r trulens_eval/requirements.optional.txt
-        displayName: Install optional deps
-
-      - bash: |
-          source activate $(condaEnvFileSuffix)
-          echo "$(python --version)"
-          echo "$(pip list)"
-        displayName: Pip list
-
-      - bash: |
-          source activate $(condaEnvFileSuffix)
-          cd ./trulens_eval
-
-          python -m pytest tests/e2e
-
-        displayName: Run e2e tests
-        env:
-          OPENAI_API_KEY: $(OPENAI_API_KEY)
-          HUGGINGFACE_API_KEY: $(HUGGINGFACE_API_KEY)
-          PINECONE_API_KEY: $(PINECONE_API_KEY)
-          PINECONE_ENV: $(PINECONE_ENV)
-          HUGGINGFACEHUB_API_TOKEN: $(HUGGINGFACEHUB_API_TOKEN)
-
-      - bash: |
-          source activate $(condaEnvFileSuffix)
-          cd ./trulens_eval
-
-          python -m pytest tests/docs_notebooks
-
-        displayName: Run notebook tests
-        env:
-          OPENAI_API_KEY: $(OPENAI_API_KEY)
-          HUGGINGFACE_API_KEY: $(HUGGINGFACE_API_KEY)
-          PINECONE_API_KEY: $(PINECONE_API_KEY)
-          PINECONE_ENV: $(PINECONE_ENV)
-          HUGGINGFACEHUB_API_TOKEN: $(HUGGINGFACEHUB_API_TOKEN)
+      - template: templates/env-setup.yaml
+      - template: templates/run-precommits.yaml
+      - template: templates/run-optional-tests.yaml
+        parameters:
+          testSuite: e2e
+      - template: templates/run-optional-tests.yaml
+        parameters:
+          testSuite: notebook