From 7a355d2e242d677ed4c42ce1cd099130f952d959 Mon Sep 17 00:00:00 2001 From: Roger Yang <80478925+RogerHYang@users.noreply.github.com> Date: Fri, 13 Oct 2023 09:00:08 -0700 Subject: [PATCH 1/7] clarify types --- trace/spec/semantic_conventions.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/trace/spec/semantic_conventions.md b/trace/spec/semantic_conventions.md index 7d1a0eb..56276a7 100644 --- a/trace/spec/semantic_conventions.md +++ b/trace/spec/semantic_conventions.md @@ -13,20 +13,20 @@ The following attributes are reserved and MUST be supported by all OpenInference | `exception.escaped` | Boolean | `true` | Indicator if the exception has escaped the span's scope | | `exception.stacktrace` | String | `"at app.main(app.java:16)"` | The stack trace of the exception | | `output.value` | String | `"Hello, World!"` | The output value of an operation | -| `output.mime_type` | String | `"text/plain"` | MIME type representing the format of `output.value` | -| `input.value` | String | `{"query": "What is the weather today?"}` | The input value to an operation | -| `input.mime_type` | String | `"application/json"` | MIME type representing the format of `input.value` | +| `output.mime_type` | String | `"text/plain"` or `"application/json"` | MIME type representing the format of `output.value` | +| `input.value` | String | `"{'query': 'What is the weather today?'}"` | The input value to an operation | +| `input.mime_type` | String | `"text/plain"` or `"application/json"` | MIME type representing the format of `input.value` | | `embedding.embeddings` | List of objects | `[{"embeeding.vector": [...], "embedding.text": "hello"}]` | List of embedding objects including text and vector data | | `embedding.model_name` | String | `"BERT-base"` | Name of the embedding model used | | `embedding.text` | String | `"hello world"` | The text represented in the embedding | | `embedding.vector` | List of floats | `[0.123, 0.456, ...]` | The embedding vector consisting of a list of floats | -| `llm.function_call` | String | `{function_name: "add", args: [1, 2]}` | Object recording details of a function call in models or APIs | -| `llm.invocation_parameters` | JSON string | `{model_name: "gpt-3", temperature: 0.7}` | Parameters used during the invocation of an LLM or API | +| `llm.function_call` | JSON String | `"{function_name: 'add', args: [1, 2]}"` | Object recording details of a function call in models or APIs | +| `llm.invocation_parameters` | JSON string | `"{model_name: 'gpt-3', temperature: 0.7}"` | Parameters used during the invocation of an LLM or API | | `llm.input_messages` | List of objects | `[{"message.role": "user", "message.content": "hello"}]` | List of messages sent to the LLM in a chat API request | | `llm.output_messages` | List of objects | `[{"message.role": "user", "message.content": "hello"}]` | List of messages received from the LLM in a chat API request | | `message.role` | String | `"user"` or `"system"` | Role of the entity in a message (e.g., user, system) | | `message.function_call_name` | String | `"multiply"` or `"subtract"` | Function call function name | -| `message.function_call_arguments_json` | String | `"{ 'x': 2 }"` | The arguments to the function call in JSON | +| `message.function_call_arguments_json` | JSON String | `"{ 'x': 2 }"` | The arguments to the function call in JSON | | `message.content` | String | `"What's the weather today?"` | The content of a message in a chat | | `llm.model_name` | String | `"gpt-3.5-turbo"` | The name of the language model being utilized | | `llm.prompt_template.template` | String | `"Weather forecast for {city} on {date}"` | Template used to generate prompts as Python f-strings | @@ -42,4 +42,6 @@ The following attributes are reserved and MUST be supported by all OpenInference | `document.id` | String/Integer | `"1234"` or `1` | Unique identifier for a document | | `document.score` | Float | `0.98` | Score representing the relevance of a document | | `document.content` | String | `"This is a sample document content."` | The content of a retrieved document | -| `document.metadata` | JSON string | `'{"author": "John Doe", "date": "2023-09-09"}'` | Metadata associated with a document represented as a JSON string | +| `document.metadata` | Object | `{"author": "John Doe", "date": "2023-09-09"}` | Metadata associated with a document | + +Note: an `object` type is a set of key-value pairs also known as a `Struct`, `Mapping`, `Dictionary`, etc. \ No newline at end of file From 37a02b2ffbed5092fc2650247d507b33504fb46a Mon Sep 17 00:00:00 2001 From: Roger Yang <80478925+RogerHYang@users.noreply.github.com> Date: Fri, 13 Oct 2023 09:00:38 -0700 Subject: [PATCH 2/7] add semconv for reranker --- trace/spec/semantic_conventions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/trace/spec/semantic_conventions.md b/trace/spec/semantic_conventions.md index 56276a7..63b01bb 100644 --- a/trace/spec/semantic_conventions.md +++ b/trace/spec/semantic_conventions.md @@ -43,5 +43,10 @@ The following attributes are reserved and MUST be supported by all OpenInference | `document.score` | Float | `0.98` | Score representing the relevance of a document | | `document.content` | String | `"This is a sample document content."` | The content of a retrieved document | | `document.metadata` | Object | `{"author": "John Doe", "date": "2023-09-09"}` | Metadata associated with a document | +| `reranker.input_documents` | List of objects | `[{"document.id": "1", "document.score": 0.9, "document.content": "..."}]` | List of documents as input to the reranker | +| `reranker.output_documents` | List of objects | `[{"document.id": "1", "document.score": 0.9, "document.content": "..."}]` | List of documents outputted by the reranker | +| `reranker.query` | String | `"How to format timestamp?"` | Query parameter of the reranker | +| `reranker.query` | String | `"cross-encoder/ms-marco-MiniLM-L-12-v2"` | Model name of the reranker | +| `reranker.top_k` | Float | 3 | Top K parameter of the reranker | Note: an `object` type is a set of key-value pairs also known as a `Struct`, `Mapping`, `Dictionary`, etc. \ No newline at end of file From 9dcab5caf98f3638d4f7b91baaacc281490730c0 Mon Sep 17 00:00:00 2001 From: Roger Yang <80478925+RogerHYang@users.noreply.github.com> Date: Fri, 13 Oct 2023 10:23:45 -0700 Subject: [PATCH 3/7] fix typo --- trace/spec/semantic_conventions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/trace/spec/semantic_conventions.md b/trace/spec/semantic_conventions.md index 63b01bb..01d8b91 100644 --- a/trace/spec/semantic_conventions.md +++ b/trace/spec/semantic_conventions.md @@ -47,6 +47,6 @@ The following attributes are reserved and MUST be supported by all OpenInference | `reranker.output_documents` | List of objects | `[{"document.id": "1", "document.score": 0.9, "document.content": "..."}]` | List of documents outputted by the reranker | | `reranker.query` | String | `"How to format timestamp?"` | Query parameter of the reranker | | `reranker.query` | String | `"cross-encoder/ms-marco-MiniLM-L-12-v2"` | Model name of the reranker | -| `reranker.top_k` | Float | 3 | Top K parameter of the reranker | +| `reranker.top_k` | Integer | 3 | Top K parameter of the reranker | Note: an `object` type is a set of key-value pairs also known as a `Struct`, `Mapping`, `Dictionary`, etc. \ No newline at end of file From 94e1e3a746944dd628f9f884b65d9453202438de Mon Sep 17 00:00:00 2001 From: Roger Yang <80478925+RogerHYang@users.noreply.github.com> Date: Fri, 13 Oct 2023 10:24:50 -0700 Subject: [PATCH 4/7] fix typo --- trace/spec/semantic_conventions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/trace/spec/semantic_conventions.md b/trace/spec/semantic_conventions.md index 01d8b91..62ebb32 100644 --- a/trace/spec/semantic_conventions.md +++ b/trace/spec/semantic_conventions.md @@ -49,4 +49,4 @@ The following attributes are reserved and MUST be supported by all OpenInference | `reranker.query` | String | `"cross-encoder/ms-marco-MiniLM-L-12-v2"` | Model name of the reranker | | `reranker.top_k` | Integer | 3 | Top K parameter of the reranker | -Note: an `object` type is a set of key-value pairs also known as a `Struct`, `Mapping`, `Dictionary`, etc. \ No newline at end of file +Note: the `object` type refers to a set of key-value pairs also known as a `Struct`, `Mapping`, `Dictionary`, etc. \ No newline at end of file From 0636e51b374ea872377ccfa3b1521b2428df82e8 Mon Sep 17 00:00:00 2001 From: Roger Yang <80478925+RogerHYang@users.noreply.github.com> Date: Fri, 13 Oct 2023 10:37:46 -0700 Subject: [PATCH 5/7] add description for reranker span_kind --- trace/spec/traces.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/trace/spec/traces.md b/trace/spec/traces.md index d592547..40ecabf 100644 --- a/trace/spec/traces.md +++ b/trace/spec/traces.md @@ -171,6 +171,10 @@ A Chain is a starting point or a link between different LLM application steps. F A Retriever is a span that represents a data retrieval step. For example, a Retriever span could be used to represent a call to a vector store or a database. +#### Reranker + +A Reranker is a span that represents the reranking of a set of input documents. For example, a cross-encoder may be used to compute the input documents' relevance scores with respect to a user query, and the top K documents with the highest scores are then returned by the Reranker. + #### LLM An LLM is a span that represents a call to an LLM. For example, an LLM span could be used to represent a call to OpenAI or Llama. From e110ef15fa541e77f28d5efcceaa209224656e43 Mon Sep 17 00:00:00 2001 From: Roger Yang <80478925+RogerHYang@users.noreply.github.com> Date: Fri, 13 Oct 2023 10:40:37 -0700 Subject: [PATCH 6/7] fix typos --- trace/spec/semantic_conventions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/trace/spec/semantic_conventions.md b/trace/spec/semantic_conventions.md index 62ebb32..5fb26a6 100644 --- a/trace/spec/semantic_conventions.md +++ b/trace/spec/semantic_conventions.md @@ -49,4 +49,4 @@ The following attributes are reserved and MUST be supported by all OpenInference | `reranker.query` | String | `"cross-encoder/ms-marco-MiniLM-L-12-v2"` | Model name of the reranker | | `reranker.top_k` | Integer | 3 | Top K parameter of the reranker | -Note: the `object` type refers to a set of key-value pairs also known as a `Struct`, `Mapping`, `Dictionary`, etc. \ No newline at end of file +Note: the `object` type refers to a set of key-value pairs also known as a `struct`, `mapping`, `dictionary`, etc. \ No newline at end of file From 928b409a803d19189889a47c735fd65f9f6cc4bc Mon Sep 17 00:00:00 2001 From: Roger Yang <80478925+RogerHYang@users.noreply.github.com> Date: Fri, 13 Oct 2023 10:45:44 -0700 Subject: [PATCH 7/7] Update trace/spec/semantic_conventions.md Co-authored-by: Mikyo King --- trace/spec/semantic_conventions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/trace/spec/semantic_conventions.md b/trace/spec/semantic_conventions.md index 5fb26a6..61441ce 100644 --- a/trace/spec/semantic_conventions.md +++ b/trace/spec/semantic_conventions.md @@ -46,7 +46,7 @@ The following attributes are reserved and MUST be supported by all OpenInference | `reranker.input_documents` | List of objects | `[{"document.id": "1", "document.score": 0.9, "document.content": "..."}]` | List of documents as input to the reranker | | `reranker.output_documents` | List of objects | `[{"document.id": "1", "document.score": 0.9, "document.content": "..."}]` | List of documents outputted by the reranker | | `reranker.query` | String | `"How to format timestamp?"` | Query parameter of the reranker | -| `reranker.query` | String | `"cross-encoder/ms-marco-MiniLM-L-12-v2"` | Model name of the reranker | +| `reranker.model_name` | String | `"cross-encoder/ms-marco-MiniLM-L-12-v2"` | Model name of the reranker | | `reranker.top_k` | Integer | 3 | Top K parameter of the reranker | Note: the `object` type refers to a set of key-value pairs also known as a `struct`, `mapping`, `dictionary`, etc. \ No newline at end of file