contentanalysis vs skylight

contentanalysis parses scientific papers from PDF and analyses their content — citation clustering, reference extraction and matching, word distribution, TF-IDF summaries by section. The most recent release adds a different kind of analysis: sentence-level classification of rhetorical moves, built on Swales' CARS model and extended to literature review and discussion sections, using rules by default with an optional Google Gemini path. PDF handling has been reworked in parallel for multi-column layouts and running header removal.

The arc runs from surface features toward discourse structure. Early releases were about getting references matched correctly and plots readable; the current one asks what function each sentence performs in the argument, which is a categorically harder question and one the package answers with rules first and a language model second. The optional-LLM design is worth noting for what it avoids — the analysis still runs without an API key, and the package has already had to prune retired Gemini model versions once, which is the maintenance cost of depending on a hosted model. Reference parsing is being made format-aware rather than pattern-guessing, with CrossRef enrichment filling in what the PDF omits.

Expect the rhetorical move classification to widen to more section types and the rule-based path to keep being the default, given the package has already been forced to track model deprecations on the optional one.