Allow reranking nodes retrieved from vector DB

[TamiTakamiya]

Description

We (Ansible Lightspeed Chatbot) are adding supplemental documents that will include what are not explicitly found in the product documentation. When those documents appear in DB search results, we want place them at higher ranks over other documents. For that purpose, we would like to allow reranking of Vector DB search results before truncating them within the token limit.

The "reranker" is defined in the "custom" directory so that each product can define its own reranker. The efault implementation does nothing and returns the given search results as they are.

Type of change

• Refactor
• New feature
• Bug fix
• CVE fix
• Optimization
• Documentation Update
• Configuration Update
• Bump-up dependent library
• Bump-up library or tool used for development (does not change the final image)
• CI configuration change
• Konflux configuration change

Related Tickets & Documents

• Related Issue #
• Closes #

Checklist before requesting a review

• I have performed a self-review of my code.
• PR has passed all pre-merge test jobs.
• If it is a core feature, I have added thorough tests.

Testing

• Please provide detailed steps to perform tests related to this code change.
• How were the fix/results from this change verified? Please provide relevant screenshots or results.

[tisnik]

it looks ok. Later we might use a plugin system, the same way as history cache or auth modules.

[TamiTakamiya]

@tisnik Thanks for your review. I have rebased, resolved a conflict and fixed a lint error. Would you merge this into the main branch if it looks ok? Thanks!