Bump FastBertTokenizer from 0.5.18-alpha to 1.0.28

[dependabot]

Bumps FastBertTokenizer from 0.5.18-alpha to 1.0.28.

Release notes

Sourced from FastBertTokenizer's releases.

v1.0.28

Highlights

• The API surface is considered stable now (except the parts explicitly marked as experimental).
• Support for netstandard2.0 was added.
• Native AOT is fully supported.
• FastBertTokenizer is now almost allocation free. That makes single-threaded encoding a bit faster and leads to larger improvements when encoding multi-threaded.

Breaking Changes

• Method signature changed: The Encode overload that returned ReadOnlyMemory returns Memory now instead. The old design made sense as the Memory points to a buffer internal to FastBertTokenizer. onnxruntime requires Memory instead of ReadOnlyMemory though. Writing to the buffer from outside doesn't break FastBertTokenizer, so it's okay to expose the buffer as Memory instead of ReadOnlyMemory to simplify usage with onnxruntime.

  - public (ReadOnlyMemory<long> InputIds, ReadOnlyMemory<long> AttentionMask, ReadOnlyMemory<long> TokenTypeIds) Encode(string input, int maximumTokens = 512, int? padTo = null)
  + public (Memory<long> InputIds, Memory<long> AttentionMask, Memory<long> TokenTypeIds) Encode(string input, int maximumTokens = 512, int? padTo = null)

• Some APIs are marked as experimental now. None were before, so it might be required to add <NoWarn>FBERTTOK001</NoWarn> <!-- Experimental FastBertTokenizer features --> to your csproj if you use them.
• The methods with the name Tokenize that were marked as obsolete before and were just renamed to Encode are removed.

Other

• Fixed #39 Add Decode support for input_id sequences that don't start at a word prefix

Commits

• c8337d8 Set version to '1.0'
• cea1c5c Add benchmark section to readme
• 871fde0 Update BenchRust
• fb2762c Add benchmark OtherLibs.FastBertTokenizer_SameDataAsBertTokenizers
• 6a05411 Update rust package tokenizers 0.14 -> 0.19
• 9f5e1de Write-Host publishOutput
• ac2e043 Use StringSpanOrdinalKey as dictionary key instead of string
• 349e876 Extend readme
• ec25988 Slightly improve test coverage
• 3a80a02 Add support for native aot
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)