LoginSign Up
← Back to all sparks
T

textrecipes

ANALYTICS
Velocity0.0

Preprocessing recipe steps for turning text into features for R models

github.com/tidymodels/textrecipes ↗

Text features finally stay sparse all the way to the model.

text-processingtidymodelsrecipessparse-datatokenizationconsistency
◆Current state
textrecipes supplies the recipes steps for turning text into model-ready columns: tokenizing, hashing, term frequency, TF-IDF, and word embeddings. Version 1.1.0 added a sparse argument to step_dummy_hash(), step_texthash(), step_tf() and step_tfidf() so they emit sparse vectors. The releases before it are a long consistency pass — keep_original_cols on every step that creates columns, informative errors on name collisions, tunable arguments documented, integer rather than double output where integers are what is meant.
◆Where it's heading
Two forces drive this package. One is memory: text produces wide, mostly-zero matrices, and the sparse work is the direct answer, landing in the same period that workflows learned to fit and predict on dgCMatrix input. The other is upstream churn — the tweets tokenizer was deprecated because tokenizers deprecated it, the politeness feature disappeared when textfeatures left Suggests. The package's own agenda is consistency; its release timing belongs to its dependencies.
◆Prediction
Expect the sparse argument to spread to the remaining column-producing steps, since only four of them have it, and expect more steps to be reworked as recipes' own sparse-data support matures.

◆Recent moves

  1. 1y ago

    Hashing and TF-IDF steps can emit sparse vectors

    ⚡ SPARK

    Adds a sparse argument to the four steps that produce the widest output — hashing, text hashing, term frequency and TF-IDF — so they emit sparse vectors instead of dense columns. This is the piece that makes tidymodels' new sparse fitting path actually usable on text.

    View source ↗
  2. 1y ago

    step_textfeatures() sped up; clean_levels NA bug fixed

    Makes step_textfeatures() faster and more robust, fixes step_clean_levels() producing NAs on character columns, and improves the tidy-method documentation so ?tidy.step_*() routes to the step's own page. Mostly documentation, with one real bug underneath.

    View source ↗
  3. 2y ago

    textfeatures dependency dropped; politeness feature removed

    Drops textfeatures from Suggests and removes the politeness feature from step_textfeatures(). A capability removal forced by a dependency, and one that changes the columns an existing recipe produces.

    View source ↗
  4. 2y ago

    untokenize and normalization return factors

    step_untokenize() and step_normalization() now return factors rather than strings. A type correction consistent with the wider output-type cleanup running through this package.

    View source ↗
  5. 2y ago

    keep_original_cols everywhere; hashing column order fixed

    Completes keep_original_cols coverage so every step that creates columns has it, adds an informative error when step_clean_names() is missing non-standard role columns, and fixes hashing steps that inserted new columns before old ones. The column-ordering bug would have silently changed downstream positional references.

    View source ↗
  6. 3y ago

    Tunable arguments documented; name collisions now error

    Documents which arguments are tunable on each step, makes column name collisions error informatively, and fixes step_tf() not being tunable on its weight argument. Discoverability work — a tunable argument nobody knows about is not tunable in practice.

    View source ↗