Skip to content

signalapp/Signal-FTS5-Extension

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Overview

Signal-FTS5-Extension is a C ABI library which exposes a FTS5 tokenizer function named signal_tokenizer that:

  • Segments UTF-8 strings into words according to Unicode standard
  • Normalizes and removes diacritics from words
  • Converts words to lower case

When used as a custom FTS5 tokenizer this enables application to support CJK symbols in full-text search.

Extension Build/Usage Example

cargo rustc --features extension -- --crate-type=cdylib

Load extension from ./target/release/libsignal_tokenizer.dylib.

CREATE VIRTUAL TABLE
fts
USING fts5(content, tokenize='signal_tokenizer')

Generating headers

cbindgen --profile release . -o target/release/fts5-tokenizer.h

Legal things

License

Copyright 2023 Signal Messenger, LLC.

Licensed under the AGPLv3: http://www.gnu.org/licenses/agpl-3.0.html

About

A FTS5 extension for signal_tokenizer.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages