Skip to content

Fix docs

Fix docs #56

GitHub Actions / xUnit Tests failed Nov 30, 2023 in 1s

xUnit Tests ❌

Tests failed

❌ src/FastBertTokenizer.Tests/TestResults/test-results-0.4.26.5664.trx

1582 tests were completed in 46s with 1575 passed, 6 failed and 1 skipped.

Test suite Passed Failed Skipped Time
FastBertTokenizer.Tests.CompareBatched 1✅ 20s
FastBertTokenizer.Tests.CompareToHuggingfaceTokenizer 1572✅ 1⚪ 19s
FastBertTokenizer.Tests.LoadTokenizer 2✅ 6❌ 331ms

❌ FastBertTokenizer.Tests.LoadTokenizer

✅ LoadBaaiBgeTokenizerFromJson(path: "data/baai-bge-small-en/tokenizer.json")
❌ LoadBaaiBgeTokenizerFromJson(path: "data/bert-base-chinese/tokenizer.json")
	System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-chinese/tokenizer.json'.
❌ LoadBaaiBgeTokenizerFromJson(path: "data/bert-base-multilingual-cased/tokenizer.json")
	System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-multilingual-cased/tokenizer.json'.
❌ LoadBaaiBgeTokenizerFromJson(path: "data/bert-base-uncased/tokenizer.json")
	System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-uncased/tokenizer.json'.
✅ LoadTokenizerFromVocab(path: "data/baai-bge-small-en/vocab.txt", lowercase: True)
❌ LoadTokenizerFromVocab(path: "data/bert-base-chinese/vocab.txt", lowercase: True)
	System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-chinese/vocab.txt'.
❌ LoadTokenizerFromVocab(path: "data/bert-base-multilingual-cased/vocab.txt", lowercase: True)
	System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-multilingual-cased/vocab.txt'.
❌ LoadTokenizerFromVocab(path: "data/bert-base-uncased/vocab.txt", lowercase: True)
	System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-uncased/vocab.txt'.

Annotations

Check failure on line 29 in src/FastBertTokenizer/BertTokenizer.LoadTokenizerJson.cs

See this annotation in the file changed.

@github-actions github-actions / xUnit Tests

FastBertTokenizer.Tests.LoadTokenizer ► LoadBaaiBgeTokenizerFromJson(path: "data/bert-base-chinese/tokenizer.json")

Failed test found in:
  src/FastBertTokenizer.Tests/TestResults/test-results-0.4.26.5664.trx
Error:
  System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-chinese/tokenizer.json'.
Raw output
System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-chinese/tokenizer.json'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Func`4 createOpenException)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.File.OpenRead(String path)
   at FastBertTokenizer.BertTokenizer.LoadTokenizerJsonAsync(String tokenizerJsonFilePath) in /_/src/FastBertTokenizer/BertTokenizer.LoadTokenizerJson.cs:line 29
   at FastBertTokenizer.Tests.LoadTokenizer.LoadBaaiBgeTokenizerFromJson(String path) in /home/runner/work/FastBertTokenizer/FastBertTokenizer/src/FastBertTokenizer.Tests/LoadTokenizer.cs:line 16
--- End of stack trace from previous location ---

Check failure on line 29 in src/FastBertTokenizer/BertTokenizer.LoadTokenizerJson.cs

See this annotation in the file changed.

@github-actions github-actions / xUnit Tests

FastBertTokenizer.Tests.LoadTokenizer ► LoadBaaiBgeTokenizerFromJson(path: "data/bert-base-multilingual-cased/tokenizer.json")

Failed test found in:
  src/FastBertTokenizer.Tests/TestResults/test-results-0.4.26.5664.trx
Error:
  System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-multilingual-cased/tokenizer.json'.
Raw output
System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-multilingual-cased/tokenizer.json'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Func`4 createOpenException)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.File.OpenRead(String path)
   at FastBertTokenizer.BertTokenizer.LoadTokenizerJsonAsync(String tokenizerJsonFilePath) in /_/src/FastBertTokenizer/BertTokenizer.LoadTokenizerJson.cs:line 29
   at FastBertTokenizer.Tests.LoadTokenizer.LoadBaaiBgeTokenizerFromJson(String path) in /home/runner/work/FastBertTokenizer/FastBertTokenizer/src/FastBertTokenizer.Tests/LoadTokenizer.cs:line 16
--- End of stack trace from previous location ---

Check failure on line 29 in src/FastBertTokenizer/BertTokenizer.LoadTokenizerJson.cs

See this annotation in the file changed.

@github-actions github-actions / xUnit Tests

FastBertTokenizer.Tests.LoadTokenizer ► LoadBaaiBgeTokenizerFromJson(path: "data/bert-base-uncased/tokenizer.json")

Failed test found in:
  src/FastBertTokenizer.Tests/TestResults/test-results-0.4.26.5664.trx
Error:
  System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-uncased/tokenizer.json'.
Raw output
System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-uncased/tokenizer.json'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Func`4 createOpenException)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.File.OpenRead(String path)
   at FastBertTokenizer.BertTokenizer.LoadTokenizerJsonAsync(String tokenizerJsonFilePath) in /_/src/FastBertTokenizer/BertTokenizer.LoadTokenizerJson.cs:line 29
   at FastBertTokenizer.Tests.LoadTokenizer.LoadBaaiBgeTokenizerFromJson(String path) in /home/runner/work/FastBertTokenizer/FastBertTokenizer/src/FastBertTokenizer.Tests/LoadTokenizer.cs:line 16
--- End of stack trace from previous location ---

Check failure on line 34 in src/FastBertTokenizer/BertTokenizer.LoadVocab.cs

See this annotation in the file changed.

@github-actions github-actions / xUnit Tests

FastBertTokenizer.Tests.LoadTokenizer ► LoadTokenizerFromVocab(path: "data/bert-base-chinese/vocab.txt", lowercase: True)

Failed test found in:
  src/FastBertTokenizer.Tests/TestResults/test-results-0.4.26.5664.trx
Error:
  System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-chinese/vocab.txt'.
Raw output
System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-chinese/vocab.txt'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Func`4 createOpenException)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
   at System.IO.StreamReader..ctor(String path)
   at FastBertTokenizer.BertTokenizer.LoadVocabularyAsync(String vocabTxtFilePath, Boolean convertInputToLowercase, String unknownToken, String clsToken, String sepToken, String padToken, NormalizationForm normalization) in /_/src/FastBertTokenizer/BertTokenizer.LoadVocab.cs:line 34
   at FastBertTokenizer.Tests.LoadTokenizer.LoadTokenizerFromVocab(String path, Boolean lowercase) in /home/runner/work/FastBertTokenizer/FastBertTokenizer/src/FastBertTokenizer.Tests/LoadTokenizer.cs:line 27
--- End of stack trace from previous location ---

Check failure on line 34 in src/FastBertTokenizer/BertTokenizer.LoadVocab.cs

See this annotation in the file changed.

@github-actions github-actions / xUnit Tests

FastBertTokenizer.Tests.LoadTokenizer ► LoadTokenizerFromVocab(path: "data/bert-base-multilingual-cased/vocab.txt", lowercase: True)

Failed test found in:
  src/FastBertTokenizer.Tests/TestResults/test-results-0.4.26.5664.trx
Error:
  System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-multilingual-cased/vocab.txt'.
Raw output
System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-multilingual-cased/vocab.txt'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Func`4 createOpenException)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
   at System.IO.StreamReader..ctor(String path)
   at FastBertTokenizer.BertTokenizer.LoadVocabularyAsync(String vocabTxtFilePath, Boolean convertInputToLowercase, String unknownToken, String clsToken, String sepToken, String padToken, NormalizationForm normalization) in /_/src/FastBertTokenizer/BertTokenizer.LoadVocab.cs:line 34
   at FastBertTokenizer.Tests.LoadTokenizer.LoadTokenizerFromVocab(String path, Boolean lowercase) in /home/runner/work/FastBertTokenizer/FastBertTokenizer/src/FastBertTokenizer.Tests/LoadTokenizer.cs:line 27
--- End of stack trace from previous location ---

Check failure on line 34 in src/FastBertTokenizer/BertTokenizer.LoadVocab.cs

See this annotation in the file changed.

@github-actions github-actions / xUnit Tests

FastBertTokenizer.Tests.LoadTokenizer ► LoadTokenizerFromVocab(path: "data/bert-base-uncased/vocab.txt", lowercase: True)

Failed test found in:
  src/FastBertTokenizer.Tests/TestResults/test-results-0.4.26.5664.trx
Error:
  System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-uncased/vocab.txt'.
Raw output
System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/runner/work/FastBertTokenizer/FastBertTokenizer/bin/FastBertTokenizer.Tests/Release/net7.0/data/bert-base-uncased/vocab.txt'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Func`4 createOpenException)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
   at System.IO.StreamReader..ctor(String path)
   at FastBertTokenizer.BertTokenizer.LoadVocabularyAsync(String vocabTxtFilePath, Boolean convertInputToLowercase, String unknownToken, String clsToken, String sepToken, String padToken, NormalizationForm normalization) in /_/src/FastBertTokenizer/BertTokenizer.LoadVocab.cs:line 34
   at FastBertTokenizer.Tests.LoadTokenizer.LoadTokenizerFromVocab(String path, Boolean lowercase) in /home/runner/work/FastBertTokenizer/FastBertTokenizer/src/FastBertTokenizer.Tests/LoadTokenizer.cs:line 27
--- End of stack trace from previous location ---