TruLens 1.3.0

[sfc-gh-jreini]

Optimizing Feedback Functions

In this release, we add important changes for improving the alignment of their LLM-Judge evals to human evaluations.

Global Improvement of Groundedness Feedback

The first is the global improvement of the groundedness feedback function (benchmarks and methods forthcoming). We invite any users to submit feedback (positive or negative) on the effectiveness of the new groundedness function using GitHub Issues or Discussions.

You can view the addition of new groundedness criteria in the GitHub diff below.

New levers for aligning feedback functions

The second change is that we add new easy-to-use levers for you to change the behavior of feedback functions using few-shot examples and custom criteria. Early customers have seen useful benefit in aligning their feedback functions to their collected expert evaluations using these levers.

Adding custom criteria to a feedback function

custom_criteria = """
A positive sentiment should be expressed with an extremely encouraging and enthusiastic tone.
"""

provider.sentiment(
    "When you're ready to start your business, you'll be amazed at how much you can achieve!",
    criteria=custom_criteria,
)

Adding few-shot examples to guide feedback functions

from trulens.feedback.v2 import feedback

fewshot_relevance_examples_list = [
    (
        {
            "query": "What are the key considerations when starting a small business?",
            "response": "You should focus on building relationships with mentors and industry leaders. Networking can provide insights, open doors to opportunities, and help you avoid common pitfalls.",
        },
        3,
    ),
]

provider.relevance(
    "What are the key considerations when starting a small business?",
    "Find a mentor who can guide you through the early stages and help you navigate common challenges.",
    examples=fewshot_relevance_examples_list,
)

What's Changed

• Feedback customization (including few-shot examples) by @sfc-gh-jreini in Feedback customization (including few-shot examples) #1674
• Custom criteria for feedback by @sfc-gh-jreini in Custom criteria for feedback #1705
• Update groundedness criteria (with more optimized prompt) by @sfc-gh-dhuang in Update groundedness criteria (with more optimized prompt) #1710
• Allow existing tables to be used in ground truth datasets by @sfc-gh-dhuang in [SNOW-1733946] Allow existing tables to be used in ground truth datasets #1698

Bug Fixes

• Allow passthrough of feedback parameters including temperature, groundedness configs in the Feedback class by @sfc-gh-jreini in Feedback customization (including few-shot examples) #1674
• Remove / retire sql instrumentation in Cortex Endpoint by @sfc-gh-dhuang in Remove / retire sql instrumentation in Cortex Endpoint #1715
• Poetry < 2.0.0 by @sfc-gh-jreini in Poetry < 2.0.0 #1709
• Update docs to use postgres + psycopg in order to avoid known issues with psycopg2 by @sfc-gh-gtokernliang in Update docs to use postgres + psycopg in order to avoid known issues with psycopg2 #1701
• Update prpr example notebook to reflect latest Cortex provider API by @sfc-gh-dhuang in Update prpr example notebook to reflect latest Cortex provider API  #1712

Preparations for Open Telemetry compatibility

• Introduce Event table for ORM to prepare for OTEL traces by @sfc-gh-gtokernliang in Introduce Event table for ORM to prepare for OTEL traces #1692
• Prototype OTEL exporter by @sfc-gh-gtokernliang in Prototype OTEL exporter #1694
• Prototype @Instrument with OTEL by @sfc-gh-gtokernliang in Prototype @instrument with OTEL #1693
• Move main_input, main_output, and _extract_content out of app.py by @sfc-gh-gtokernliang in Move main_input, main_output, and _extract_content out of app.py #1706
• Move span-related validation + setting logic out of instrument.py by @sfc-gh-gtokernliang in Move span-related validation + setting logic out of instrument.py #1707

Full Changelog: trulens-1.2.11...trulens-1.3.0

---

Important

TruLens 1.3.0 enhances feedback functions with improved groundedness and customization, fixes bugs, and prepares for Open Telemetry compatibility.

• Feedback Functions:
  • Improved groundedness feedback function globally.
  • Added customization options for feedback functions using few-shot examples and custom criteria.
• Bug Fixes:
  • Allow passthrough of feedback parameters in Feedback class.
  • Remove SQL instrumentation in Cortex Endpoint.
  • Update documentation to use Postgres + psycopg.
• Open Telemetry Preparations:
  • Introduce Event table for ORM.
  • Prototype OTEL exporter and @instrument decorator.
• Miscellaneous:
  • Update version to 1.3.0 in pyproject.toml across multiple components.
  • Minor logging changes in run.py.

^{This description was created by for 3a62061. It will automatically update as commits are pushed.}