LoginSign Up
← Back to all sparks
P

pomdp

AI-ASSISTANTS
Velocity0.0

Infrastructure for Partially Observable Markov Decision Processes (POMDP)

github.com/mhahsler/pomdp ↗

A POMDP solver that quietly grew into a full reinforcement-learning toolkit.

reinforcement-learningmdppomdpr-packagesimulationteaching-tools
◆Current state
pomdp is an R interface to the pomdp-solve engine for partially observable Markov decision processes, now carrying its own MDP solvers, gridworld environments and simulation code. The 2024 releases moved the heavy accessor and simulation paths into C++ with sparse-matrix support. Recent activity is maintenance-grade: the latest release only adds source data and a journal citation.
◆Where it's heading
The arc runs from POMDP file parsing toward being a general teaching and research toolkit for sequential decision problems, with Q-learning, Sarsa and expected Sarsa sitting beside the exact solvers. Each cycle has widened the MDP side while normalising the POMDP side into a single model representation. The cadence has slowed markedly since the 1.2.0 push, and the newest entry is documentation rather than code.
◆Prediction
With the R Journal reference now landed, the near-term work is most likely consolidation — more datasets and gridworld environments rather than new solver classes.

◆Recent moves

  1. 1y ago

    Adds source data and R Journal citation

    Adds the source data to GitHub and a citation to the package's R Journal article. No code change — this is the publication milestone being recorded rather than a functional release.

    View source ↗
  2. 1y ago

    DynaMaze dataset, gridworld and policy-graph fixes

    Ships the DynaMaze MDP dataset plus fixes to gridworld start-state recording and policy graphs, and folds in a 1.2.3 memory-violation fix in the matrix accessors. It extends the environment library that the 1.2.0 solver work created.

    View source ↗
  3. 2y ago

    Q-learning, Sarsa and gridworlds turn pomdp into an MDP toolkit

    ⚡ SPARK

    The release where the package stopped being POMDP-only: Q-learning, Sarsa and expected Sarsa solvers, gridworld environments, absorbing/reachable state analysis and MDP-POMDP conversion all land at once. Everything since has extended this surface rather than the original solver bindings.

    View source ↗
  4. 2y ago

    C++ backend, sparse models, and regret and value-function accessors

    Bundles 1.1.1 and 1.1.0: an Rcpp backend with parallel support and sparse matrices for large models, plus new value_function(), regret() and transition_graph() accessors. This is the performance groundwork the later MDP solver work was built on.

    View source ↗
  5. 4y ago

    Policy trees for finite-horizon problems and trajectory sampling

    Policy graphs gain finite-horizon policy trees with a specifiable initial belief, and belief-space sampling gains a trajectories method. Incremental work on inspecting solved policies rather than solving them.

    View source ↗
  6. 4y ago

    Solver split into pomdpSolve; first solve_MDP() lands

    Moves the pomdp-solve engine out to a separate pomdpSolve package, flattens the model object, and adds the first solve_MDP(). The packaging split and that one function set up the MDP direction that 1.2.0 later filled in.

    View source ↗