LoginSign Up
← Back to all sparks
O

OpenHouse

ANALYTICS
Velocity5.0

Control plane for managing tables in open-source data lakehouse deployments

www.openhousedb.org ↗

LinkedIn's Iceberg control plane, shipping one pull request per release.

icebergdata-lakehousetable-metadataobservabilitycontrol-plane
◆Current state
OpenHouse is LinkedIn's open-source control plane for Iceberg tables, and it releases per merged pull request — version numbers climb several times a week with a single change each. The current work is concentrated on making the service defensible in production: a fix for CREATE OR REPLACE AS SELECT silently wiping table policies, request-ID correlation and a typed exception hierarchy in the data loader, and targeted scheduler logging for jobs observability.
◆Where it's heading
The theme across these releases is treating table metadata as something that must not be lost by accident, and making failures attributable. Policies now merge rather than being rebuilt from the request. Data loader errors carry a request ID and distinguish authentication from transport failure instead of retrying auth errors as transient. Feature toggles gained self-service table overrides so server-side ramps and table-owner opt-in can coexist.
◆Prediction
The jobs-observability plan explicitly defers OTEL gauges, a heartbeat sampler, and DLQ counters to a later phase, so those are the concrete next steps visible in these entries.

◆Recent moves

  1. 7d ago

    CREATE OR REPLACE AS SELECT no longer silently drops table policies

    Fixes a genuine data-governance hazard: the replace path rebuilt the policies property purely from the incoming request, so a replace that omitted policies wiped retention, sharing, PII column tags, replication and history — while ordinary table properties survived. Policies now merge, with request values overriding and unspecified planes carried forward.

    View source ↗
  2. 8d ago

    Automated iceberg-core dependency bump

    A Renovate-generated bump of com.linkedin.iceberg:iceberg-core, whose own release notes state there are no notable improvements and no referenced pull requests. Pure version tracking.

    View source ↗
  3. 8d ago

    Scheduler log lines for jobs observability, phase 1.5

    About 35 lines of localized logging across three files, adding stable tokens for scheduler startup and eligible-count, and distinguishing a genuine queue timeout from poll-lag on already-terminal jobs. Deliberately scoped below the OTEL instrumentation deferred to a later phase.

    View source ↗
  4. 10d ago

    Request-ID correlation and typed catalog exceptions in the data loader

    Assigns an X-Request-ID to every outbound catalog request and exposes it through a typed exception hierarchy, so DataLoader errors can be tied to the specific Tables Service request. It also stops authentication failures from being represented as generic OSErrors and retried as if they were transient I/O.

    View source ↗
  5. 10d ago

    Self-service table overrides for feature toggles

    Extends TableFeatureToggle so an explicit table property wins over server-side targeting, with activation falling back to the server toggle when the property is absent. It resolves the tension between ramping a feature centrally and letting table owners opt in themselves, and adds trailing-wildcard matching for database and table names.

    View source ↗
  6. 11d ago

    Renovate added to track two parallel Iceberg version lines

    Adds Renovate to watch the LinkedIn Iceberg build, keeping the 1.2.x and 1.5.x series OpenHouse depends on simultaneously each on its own newest version without crossing between them. Build hygiene that explains the automated bumps appearing in later releases.

    View source ↗