LoginSign Up
← Back to all sparks
S

sentencepiece

AI-ASSISTANTS
Velocity0.0

Text Tokenization using Byte Pair Encoding and Unigram Modelling

github.com/bnosac/sentencepiece ↗

The R binding to Google's tokenizer has shipped nothing but compiler fixes since 2021.

rnlptokenizationbindingsmaintenance
◆Current state
sentencepiece wraps Google's subword tokenizer for R, exposing BPE and unigram encoding, model training and the BPEembed interface. Functionally it has been frozen since 0.2, which upgraded the vendored library to sentencepiece v0.1.96 and fixed a wordpiece bug for one-character words. Every release since is toolchain work: UBSAN, snprintf on M1 Macs, dropping C++11, then requiring C++17.
◆Where it's heading
This is a binding whose upstream moved on without it. The releases respond to CRAN's compiler policy rather than to sentencepiece's own development, and the vendored third-party tree is where nearly all the churn lands. Its practical role is as a dependency for the surrounding bnosac NLP packages, which is what keeps it on CRAN at all.
◆Prediction
The next release will most likely be another C++ standard or compiler-warning fix; a bump of the vendored sentencepiece library is the change that would matter, and nothing in the entries indicates one is planned.

◆Recent moves

  1. 6mo ago

    Build switched to C++17

    Switches Makevars to C++17, completing the standard migration begun in 0.2.4. Build configuration only.

    View source ↗
  2. 8mo ago

    C++11 dropped; deprecated std::iterator replaced in protobuf-lite

    Drops C++11 from Makevars and replaces a std::iterator use in the vendored protobuf-lite tree that C++17 deprecates. Compiler conformance in third-party code.

    View source ↗
  3. 3y ago

    R CMD check warning from 0.2.2 cleared

    Clears an R CMD check warning introduced by 0.2.2, in a section of vendored protobuf-lite the package does not even use. Published within a minute of 0.2.2.

    View source ↗
  4. 3y ago

    sprintf replaced with snprintf for M1 Mac checks

    Replaces sprintf with snprintf to satisfy an R CMD check note on M1 Macs. The change that 0.2.3 immediately had to patch.

    View source ↗
  5. 4y ago

    clang-UBSAN fix

    A single clang-UBSAN fix. No user-visible change.

    View source ↗
  6. 4y ago

    Vendored sentencepiece upgraded to v0.1.96; wordpiece fix

    The last functional release: upgrades the vendored library to sentencepiece v0.1.96 and fixes wordpiece handling of one-character words. Its notes also repeat the 0.1.3 changelog, which added BPEembed and subword regularisation via the nbest and alpha options.

    View source ↗