LoginSign Up
← Back to all sparks
Q

quanteda

ANALYTICS
Velocity2.5

R package quanteda by quanteda — release notes from GitHub.

github.com/quanteda/quanteda ↗

Text analysis in R keeps optimising its token internals — and builds a path out to torch

text-analysisnatural-language-processingr-packagetorchperformance
◆Current state
quanteda is a mature framework for quantitative text analysis in R. Since the 4.0 rewrite around external-pointer tokens objects, releases have concentrated on the internals: recompilation control, memory reduction on concatenation, type-table consistency between tokens and dfm objects. The newest release adds tokens_recompile() for explicit ID reassignment, stops query functions from recompiling implicitly, and returns dense rather than sparse tensors from as.tensor() with arguments passed through to torch.
◆Where it's heading
Two threads run in parallel. The dominant one is performance and correctness housekeeping on the tokens_xptr representation introduced in 4.0 — each release closes another case where the external-pointer path diverged from the plain tokens path. The quieter thread points outward: as.matrix() returning a document-by-position integer matrix and as.tensor() handing off to torch::torch_tensor() make the tokenised corpus directly consumable by neural models rather than only by quanteda's own bag-of-words machinery.
◆Prediction
The tensor and matrix export path is the least mature part of the surface and gained arguments in this release rather than settling, so expect further work there before the token internals change again.

◆Recent moves

  1. 12d ago

    Explicit token recompilation and a denser path out to torch

    tokens_recompile() gives explicit control over ID reassignment and removing unused types, while query functions like types() and ntoken() stop recompiling tokens_xptr objects implicitly — subsetting now preserves the complete type table, matching dfm behaviour. as.matrix() for tokens returns a dense document-by-position integer matrix, and as.tensor() switches from sparse COO to dense with torch arguments passed through.

    View source ↗
  2. 1y ago

    Corpus chunking and cheaper token concatenation

    corpus_chunk() splits texts into smaller documents, memory usage for concatenating large tokens objects drops significantly, and tokens_ngrams() gains the apply_if argument already present on compound and lookup operations.

    View source ↗
  3. 1y ago

    Faster concatenation and a dfm_lookup naming fix

    Concatenation on tokens and tokens_xptr objects gets significantly faster, verbose messages are made consistent, and a dfm_lookup() bug producing wrong feature names under exclusive matching is fixed.

    View source ↗
  4. 2y ago

    Minor test and documentation fixes

    Two minor fixes: a failing C++ test around fcm() and token re-indexing, and an undeclared package in Rd cross-references.

    View source ↗
  5. 2y ago

    Platform-specific test and installation fixes

    Fixes for test failures caused by shifting Matrix behaviour on r-devel, an undeclared package in Rd cross-references, and a Fedora installation failure from TBB being searched for in the wrong places.

    View source ↗
  6. 2y ago

    CRAN v4.0

    ⚡ SPARK

    tokens_xptr objects extend tokens with external pointers, and once converted the pointer-based methods are dispatched automatically. word4 becomes the default tokeniser, thread count becomes adjustable mid-pipeline through quanteda_options(), the magrittr pipe is replaced throughout with R's native pipe, and apply_if lets a tokens operation run on a document subset.

    View source ↗