LoginSign Up
← Back to all sparks
T

tidytext

ANALYTICS
Velocity0.0

Text mining in R using tidy data principles and the tidyverse toolkit

github.com/juliasilge/tidytext ↗

Finished, widely taught, and shipping roxygen fixes.

text-miningtidyversetopic-modelsquantedastablemaintenance
◆Current state
tidytext is the package that made unnest_tokens() and the tidy-data approach to text analysis standard, and it has reached the point where its releases contain nothing to announce. The last three are roxygen package anchors, alt text on vignette figures, and a single bug fix in one stm tidier. The most recent substantive changes were in 0.4.0 and 0.3.3 — stm tidiers for high FREX and lift words, a labels function for scale_x_reordered(), and support for tidying STM models that use content covariates.
◆Where it's heading
The direction is stability, and the release triggers are external. quanteda releases force updates to the dfm tidiers, a Matrix release forces another, tokenizers deprecating its tweet tokenizer forces removal of the tweet-specific functions here, and CRAN's Rd anchor requirement produces a release of its own. Nothing in the recent stream suggests new capability is planned, and for a package this embedded in teaching material that is a defensible position rather than a problem.
◆Prediction
The entries do not support predicting new features. The likely next release is another compatibility update prompted by quanteda, stm, or a CRAN documentation requirement.

◆Recent moves

  1. 1y ago

    Update roxygen package anchors for CRAN

    A single line updating package anchors in roxygen comments, which is CRAN's cross-reference requirement rather than anything the package wanted. The clearest possible marker of where this package is in its life.

    View source ↗
  2. 2y ago

    Figure alt text and a quanteda v4 vignette update

    Adds alt text to figures in the vignettes and README, and updates a vignette for quanteda::dfm() v4. Accessibility and documentation only.

    View source ↗
  3. 3y ago

    Fix the FREX stm tidier

    One bug fix in the FREX stm tidier added two weeks earlier in 0.4.0. Narrow follow-up.

    View source ↗
  4. 3y ago

    FREX and lift tidiers added; tweet tokenizers removed

    Adds stm tidiers for high FREX and lift words, moves hunspell to a suggested dependency, and removes the tweet-specific tokenizers because of upstream changes. The same tokenizers deprecation removed the tweets option from textrecipes around the same time — one upstream decision propagating through two packages.

    View source ↗
  5. 3y ago

    Update dfm tidier for the upcoming Matrix release

    Updates the tidy method for a quanteda dfm ahead of a Matrix release. A compatibility fix with no change of its own.

    View source ↗
  6. 4y ago

    Reordered scales take a labels function; STM content models tidied

    Changes scale_x_reordered() and scale_y_reordered() to take a function as their labels input, fixes how to_lower is passed for character shingles, and adds support for tidying STM models that use content covariates. The reordered-scale change is the one people notice, since it is used in nearly every tidytext plot.

    View source ↗