feat(llm-observability): llm metrics and feedback #27785
Conversation
…ervability-scores
…ervability-scores
|
Size Change: -24 B (0%) Total Size: 1.16 MB ℹ️ View Unchanged
|
…ervability-scores
skoob13
changed the title
📸 UI snapshots have been updated1 snapshot changes in total. 0 added, 1 modified, 0 deleted:
Triggered by this commit. |
|
The UI changes overlap a bit with my #27841, but we'll figure it out |
| <br /> | ||
| <br /> |
There was a problem hiding this comment.
Why two new lines and not some margin?
| export function MetricTag({ properties }: MetricTagProps): JSX.Element { | ||
| const { $ai_metric_name: metricName, $ai_metric_value: metricValue } = properties | ||
| const strValue = String(metricValue) | ||
| const isValueLong = strValue.length > 10 |
| properties: Record<string, any> | ||
| } | ||
|
|
||
| export function MetricTag({ properties }: MetricTagProps): JSX.Element { |
There was a problem hiding this comment.
For metrics, it'd be useful to say that this is a metric, as it's sort of unusually magical that this tag is pulled in from a completely different event (the magic could also be clearer with Feedback, but there at least the event has feedback in the name, while metrics have arbitrary names)
There was a problem hiding this comment.
Yeah, I agree with you. I'd also add a general overview of a trace with its metrics and feedback, which would be much easier to read. These short tags were supposed to help you if you want to find metrics or feedback quickly, for example, inside a generation.
| posthog.capture('$ai_feedback', { | ||
| $ai_feedback_text: feedback, | ||
| $ai_trace_id: traceId, | ||
| }) |
There was a problem hiding this comment.
We must add a posthog-js method for this – users should never have to type dollar-prefixed properties, because they're the features we offer out of the box. This way we ensure the SDKs and the product's UI all work together neatly, and the user has a smooth time implementing. Let's do this as a follow-up
There was a problem hiding this comment.
That makes sense. If we keep adding functionality, the bundle size will concern me, but the plugins probably have already solved this.
Problem
LLM Observability should provide functionality to gather quantitative and non-quantitative feedback from users. We've agreed on two events:
$ai_metricallows you to create your own custom metrics.$ai_metric_name- optional metric name.$ai_metric_value- required metric value.$ai_feedbackallows you to capture user-written feedback.$ai_feedback_text- required text.Screenshot
Changes
Add metrics and feedback to the traces sceneI've decided to do that in a separate PR.Does this work well for both Cloud and self-hosted?
Yes
How did you test this code?
Unit & manual testing.