Signal-FTS5-Extension is a C ABI library which exposes a
FTS5 tokenizer function named
signal_tokenizer that:
- Segments UTF-8 strings into words according to Unicode standard
- Normalizes and removes diacritics from words
- Converts words to lower case
When used as a custom FTS5 tokenizer this enables application to support CJK symbols in full-text search.
cargo rustc --features extension -- --crate-type=cdylibLoad extension from ./target/release/libsignal_tokenizer.dylib.
CREATE VIRTUAL TABLE
fts
USING fts5(content, tokenize='signal_tokenizer')cbindgen --profile release . -o target/release/fts5-tokenizer.hCopyright 2023 Signal Messenger, LLC.
Licensed under the AGPLv3: http://www.gnu.org/licenses/agpl-3.0.html