LoginSign Up
← Back to all sparks
S

Sonic

DEVOPS
Velocity5.0

Lightweight and schema-less search backend

github.com/valeriansaliou/sonic ↗

Sonic is shipping tokenizer control and a dependency diet, two tags at a time.

search-indexrusttokenizerdependency-hygienedual-tagging
◆Current state
Sonic is a lightweight Rust search index that now releases in pairs — a server tag (v1.7.x) and a core-crate tag (v0.1.x) covering the same underlying commits. The recent work splits between tokenizer behavior, where pattern matching became opt-in and special tokens stopped being split apart, and a dependency audit that pinned every crate, disabled default features across the board, and trimmed unused RocksDB functionality. Release cadence is tight, with several tags landing within days.
◆Where it's heading
Two things are happening at once. The tokenizer changes are aimed at correctness for users indexing identifiers and structured strings — the recurring theme of treating more strings as IDs and performing implicit AND on ID queries suggests the index was mangling exactly the inputs developers reach for it with. The dependency work reads as supply-chain and binary-size discipline: pinning everything and disabling default features is what a project does when it wants a small, auditable, reproducible build. The core/server split into separate crates is what makes the paired tags necessary.
◆Prediction
Expect the tokenizer work to continue toward configurable behavior rather than better defaults, given pattern matching was made opt-in and non-breaking in the same release. The paired v1.7.x/v0.1.x tagging will keep producing two changelog entries per actual release.

◆Recent moves

  1. 28d ago

    Opt-in tokenizer pattern matching plus a full dependency pin

    The most substantial release in this window: the tokenizer stops splitting special tokens and gains opt-in, non-breaking pattern matching — directly addressing the identifier-mangling problem the recent fixes keep circling. The other half is dependency hygiene, pinning every server crate, disabling default features, and dropping lazy_static.

    View source ↗
  2. 28d ago

    Core crate picks up the tokenizer changes

    The core-crate half of the same release as v1.7.4, carrying the tokenizer changes and the RocksDB feature trim without the server-side packaging work. Anyone tracking this feed should read the two tags as one release, not two.

    View source ↗
  3. 1mo ago

    More strings treated as IDs; Debian packaging stops chowning

    Two small commits — broadening what counts as an ID and dropping a recursive chown on every Debian update. Minor, but part of the same identifier-handling thread that the next release addresses properly.

    View source ↗
  4. 1mo ago

    Core crate tag for the ID-detection change

    The core-crate mirror of v1.7.3, containing only the broader ID-detection commit. No independent content.

    View source ↗
  5. 1mo ago

    ID queries now perform an implicit AND

    A single-commit fix making ID queries combine terms with AND rather than OR. Small, but it changes result sets for anyone querying identifiers — the kind of correctness gap the tokenizer work later generalizes.

    View source ↗
  6. 1mo ago

    Core crate tag for the implicit-AND fix

    The core-crate mirror of v1.7.2, carrying the same single implicit-AND commit. Another instance of one fix producing two changelog entries.

    View source ↗