fix: Remove tiktoken references #98
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
As discussed think the implementation is all good but not sure if right, ultimately a question of how important are tokens being accurate per model basis and the impact either way, think it's for the team to decide/guide |
larry6point6
approved these changes
Feb 21, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
Tiktoken is making direct calls to openAI which is causing a RateLimitError as we don't use openai anymore we shouldn't even be referencing it (security wise not great either)
Its only for metadata so doesn't impact anything properly but turning it off will reduce the amount of herrings and ensure our independence.
Changes proposed in this pull request
Replace tiktoken references with a custom tokeniser simply used to count token usage across codebase and minorly used for wikipedia/govuk search functions
Guidance to review
Please run it locally, check it hasn't broken any of the routes for you, also check you are happy with the approach or if you have any suggestions on what you'd rather use.
Relevant links
Things to check