Skip to content

Commit

Permalink
⚡️ perf: downset tokenizer threshold to reduce browser memory usage (#…
Browse files Browse the repository at this point in the history
…5000)

* ⚡️ perf: down token account

* Update index.ts
  • Loading branch information
arvinxx authored Dec 12, 2024
1 parent 23fe3df commit ec60974
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions src/utils/tokenizer/index.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
export const encodeAsync = async (str: string): Promise<number> => {
if (str.length === 0) return 0;

// 50_000 is the limit of the client
// if the string is longer than 100_000, we will use the server
if (str.length <= 50_000) {
// 30_000 is the limit of the client
// if the string is longer than 30_000, we will use the server
if (str.length <= 30_000) {
const { clientEncodeAsync } = await import('./client');

return await clientEncodeAsync(str);
Expand Down

0 comments on commit ec60974

Please sign in to comment.