Evaluations

[bjwswang]

Overall workflow

TO BE DEFINED

Evaluation Types

RAG Evaluation

@Lanture1064 @bjwswang

Our current RAG solution flow :

1. Dataset/VersionedDataset provides source files
2. Dataprocessing process source files to QA csv with QAGenerationChain
3. Knowledgebase generates embeddings from the QA csv file to vectorstore
4. RetrievalQAChain do similarity search against knowledgebase
5. LLM do chat with the similarity searched content as the context

Based on our research ,we decide to use this evaluation framework https://github.com/explodinggradients/ragas

Subtasks:

• Task1 generate test dataset based on our rag solution #494
• Task2 generate metrics by ragas evaluation with our test dataset(QA) #495

Evaluation Lifecycle management

@0xff-dev @bjwswang

For definitions:

• Define a new CRD RAG under group evaluations.arcadia.kubeagi.k8s.com.cn provide lifecycle management of Evaluation Task #507

For task runner:

• Able to use kubernetes Job as evaluation runner #508
• Able to use Tekton(CI/CD) as evaluation runner [Not planned]
• Able to use volcano(a powerful batching system) as evaluation runner [Not planned]

For apiserver:

• Add evalutions apis: create/update/delete/list #565

Overall Workflow

[nkwangleiGIT]

@Lanture1064 pls do further investigation.

[nkwangleiGIT]

here is another project related to evaluate:
https://github.com/promptfoo/promptfoo

[nkwangleiGIT]

Some other thoughts:

• 支持 Prompt 在不同 LLM 下的评估，生成测试报告
  • RAG 评估、RAG Question Generation
  • 自动生成问题，分析问题质量，过滤掉相似度不高的问题
  • 评估指标：检索评估 - Hit Rate、MRR，回答评估 - 公正性、相关性、一致性等

[bjwswang]

Some other thoughts:

• 支持 Prompt 在不同 LLM 下的评估，生成测试报告

  • RAG 评估、RAG Question Generation
  • 自动生成问题，分析问题质量，过滤掉相似度不高的问题
  • 评估指标：检索评估 - Hit Rate、MRR，回答评估 - 公正性、相关性、一致性等

1. RAG问题生成这里采用数据处理服务提供的能力即可
2. ragas的指标(metrics)比较完善。 一些框架的对比可以在 https://shimo.im/docs/1d3aMzQ1wXHnwV3g

[bjwswang]

This story can be closed. For other features like support tektonci and other llms ,we can track in individual issues.