LoginSign Up
← Back to all sparks
C

ClearML

AI-ASSISTANTS
Velocity5.0

End-to-end MLOps platform for experiment tracking and orchestration

clear.ml ↗

ClearML is hardening the SDK against the artifacts it loads — pickles included.

experiment trackinghyperdatasetsartifact securitystorage manager
◆Current state
Recent releases pair hyperdataset work with a steady security pass over the SDK's own inputs. 2.1.7 added an opt-out that blocks processing of pickled artifacts, via a call argument, a config key or CLEARML_BLOCK_PICKLED_ARTIFACTS, and a path-traversal check when import_offline_session extracts a zip; 2.1.6 added integrity-hash verification for pickled DataFrame artifacts; 2.1.8 added a path-traversal check in dataset merging. Alongside that, hyperdatasets gained tagging, version snapshots, single-call publishing and a DataView get method, and 2.1.11 added in-memory data streaming to the storage manager with a 100 MB cap on registration payloads.
◆Where it's heading
Two things are converging. The hyperdataset API is filling in the lifecycle operations a dataset abstraction needs to be usable — snapshot, tag, publish, retrieve — which is the boring work that decides whether people build on it. Meanwhile the SDK is being treated as something that consumes untrusted input, because in a shared experiment tracker it does: an artifact is a file another user uploaded, and Python's default answer to a pickle is to execute it. Blocking that by configuration rather than by default keeps existing pipelines working while giving security-conscious deployments a switch.
◆Prediction
Pickle blocking is opt-out today, and the notes give no timeline for flipping the default. The clearer near-term thread is Python 2 removal and the f-string migration, both described as work in progress across several releases.

◆Recent moves

  1. 5h ago

    In-memory streaming in the storage manager, DataView retrieval

    In-memory data streaming through the storage manager, a DataView get method for retrieval from the API server, and a 100 MB ceiling on data-entry registration payloads.

    View source ↗
  2. 5h ago

    HPO trial pruning and hashlib usedforsecurity fixes

    Hyperparameter optimisation becomes more resilient by pruning trials, parent_ids is deprecated in favour of parent_id while remaining usable, and hashing calls move to usedforsecurity=False so non-security hashes work under restricted crypto policies.

    View source ↗
  3. 1mo ago

    Hyperdataset version snapshots and a static route validator

    Adds a method to snapshot a hyperdataset version and a static route validator for RouterService, alongside a fix to Dataset.get reported from the community.

    View source ↗
  4. 2mo ago

    Hyperdataset tagging and publishing, plus Azure default credentials

    Hyperdataset tagging and single-call publishing arrive, DefaultAzureCredential is supported, and Dataset._merge_datasets gains a path-traversal check. A fix also unblocks project-restricted users from creating dataviews.

    View source ↗
  5. 2mo ago

    Opt-out blocking for pickled artifacts and zip path traversal

    Adds an opt-out that blocks processing of pickled artifacts — per call, by config, or through CLEARML_BLOCK_PICKLED_ARTIFACTS — and a path-traversal check when import_offline_session extracts a zip. Both address the case where an artifact is a file someone else uploaded.

    View source ↗
  6. 2mo ago

    Pickle integrity hashes and configurable plot upload destinations

    Integrity-hash verification for pickled DataFrame artifacts, a configurable plots upload destination, a default shell binary option for script execution, and routing fixes for queues on autoscaler instances.

    View source ↗