Apache Iceberg REST Catalog — Open Standard for Catalog Interoperability

The Smallest Component with the Biggest Lock-In

An Iceberg table is, at its core, a pointer. All the data files, manifests, and snapshots live on object storage — but the one piece of state that makes a table a table is the pointer to its current metadata file. The catalog owns that pointer. Every read starts by asking the catalog "where is the current metadata for analytics.events?", and every write ends by asking it to atomically swap that pointer to a new metadata file. That single compare-and-swap is what gives Iceberg ACID semantics on top of dumb object storage.

Which means the catalog — a service whose entire job is to store one pointer per table — quietly became the most locked-in component of the lakehouse. Hive Metastore speaks Thrift. AWS Glue speaks its own SDK. Nessie had its own REST dialect, JDBC catalogs their own connection conventions. Every engine needed a separate connector for every catalog: with four engines and four catalog types you were maintaining a 4×4 compatibility matrix, and adding a new catalog meant shipping connector code into Spark, Flink, Trino, and Hive separately. Swapping catalogs meant touching every engine's configuration and praying the connector versions lined up.

The Iceberg REST Catalog specification collapses that matrix to a line. It defines one HTTP API that any catalog server can implement and any client can consume. Engines ship exactly one REST connector; catalog vendors implement exactly one API. The catalog becomes what it always should have been: a swappable backend behind a URL. Apache Polaris, Apache Nessie, Gravitino, Unity Catalog, Snowflake, and a growing list of commercial platforms all expose the same spec — and a PyIceberg script pointed at any of them cannot tell the difference.

What the Spec Actually Standardizes

The spec is an OpenAPI document maintained in the apache/iceberg repository — not prose, an executable contract. Everything lives under a versioned /v1 prefix and authenticates with OAuth2 bearer tokens. Four groups of endpoints do the real work:

# The endpoints that matter (abbreviated)

GET    /v1/config                                        # handshake + server defaults
POST   /v1/oauth/tokens                                  # OAuth2 client_credentials

GET    /v1/namespaces                                    # list
POST   /v1/namespaces                                    # create
DELETE /v1/namespaces/{ns}                               # drop

GET    /v1/namespaces/{ns}/tables                        # list tables
POST   /v1/namespaces/{ns}/tables                        # create table
GET    /v1/namespaces/{ns}/tables/{table}                # load: metadata + credentials
POST   /v1/namespaces/{ns}/tables/{table}                # commit: atomic update
DELETE /v1/namespaces/{ns}/tables/{table}                # drop
POST   /v1/tables/rename

GET    /v1/namespaces/{ns}/views                         # views: same lifecycle

The Commit Protocol: Optimistic Concurrency over HTTP

The most interesting part of the spec is the part most articles skip: how a commit actually works. A commit request carries two lists — requirements and updates. Requirements are assertions about the table's current state: "the main branch must still point at snapshot 4632", "the current schema ID must be 2". Updates are the changes to apply if every assertion holds: add a snapshot, set a new current schema, update a property.

# POST /v1/namespaces/analytics/tables/events  (commit, abbreviated)
{
  "requirements": [
    { "type": "assert-ref-snapshot-id",
      "ref": "main",
      "snapshot-id": 4632 }
  ],
  "updates": [
    { "action": "add-snapshot",
      "snapshot": { "snapshot-id": 4633, "...": "..." } },
    { "action": "set-snapshot-ref",
      "ref-name": "main",
      "type": "branch",
      "snapshot-id": 4633 }
  ]
}

# Server response:
#   200 -> requirements held, pointer swapped atomically
#   409 -> someone committed first; client refreshes metadata,
#          rebases its changes, and retries

This is classic optimistic concurrency control, and putting it behind HTTP has a consequence that is easy to miss: the server becomes the arbiter of conflicts, not the client. With file-based or Hive catalogs, every engine reimplements commit-conflict handling against a bare compare-and-swap. A REST server can be smarter — it validates requirements server-side, can retry non-overlapping updates internally, and can reject writes that violate policy before they ever touch storage. Concurrent appends from a Flink streaming job and a nightly Spark batch stop being a race between two clients and become two requests queued at one authority.

Choosing a Server

Because the client side is identical everywhere, choosing a catalog is now purely a server-side decision. The realistic contenders:

The short version: if you are picking fresh and want open source, pick Polaris — donated by Snowflake to the ASF in 2024, it has the most complete security model of the open implementations. Pick Nessie when git-style branching (write-audit-publish, multi-table atomic promotion) is the feature you actually need. And note that Hive Metastore is the only row with three red cells — it is the thing this spec exists to replace.

Running Apache Polaris

Polaris models the world as catalogs (each mapping to a warehouse — an object-storage prefix), principals (human or service identities with OAuth2 client credentials), and roles that connect principals to privileges. Getting a local instance to a usable state takes one container and three API calls:

# 1. Run the server (in-memory persistence — dev only)
docker run -p 8181:8181 \
  -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_REGION \
  apache/polaris:latest
# root principal credentials are printed in the startup log

# 2. Exchange root credentials for a management token
curl -s -X POST http://localhost:8181/api/catalog/v1/oauth/tokens \
  -d "grant_type=client_credentials" \
  -d "client_id=root" -d "client_secret=<from-startup-log>" \
  -d "scope=PRINCIPAL_ROLE:ALL"

# 3. Create a catalog bound to an S3 prefix
curl -s -X POST http://localhost:8181/api/management/v1/catalogs \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{
    "name": "prod",
    "type": "INTERNAL",
    "properties": { "default-base-location": "s3://my-lake/iceberg" },
    "storageConfigInfo": {
      "storageType": "S3",
      "allowedLocations": ["s3://my-lake/iceberg"],
      "roleArn": "arn:aws:iam::123456789012:role/polaris-s3-role"
    }
  }'

# 4. Create a service principal for your engines + grant it a role
curl -s -X POST http://localhost:8181/api/management/v1/principals \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"name": "spark-etl", "type": "SERVICE"}'

For production, swap the in-memory store for PostgreSQL-backed persistence, run at least two replicas behind a load balancer, and give the server — not your compute clusters — the IAM role that can reach the lake. The production checklist at the end covers the details; the architectural point is that Polaris deliberately concentrates storage privilege in one small, auditable service.

Local Testing Without Cloud Credentials

You do not need Polaris to develop against the REST API. The Iceberg project maintains a zero-persistence fixture server that implements the full spec against local filesystem storage — no auth, no cloud, state resets on restart. It is the right tool for CI and exactly the wrong tool for anything else.

# Canonical local test server
docker run -p 8181:8181 tabulario/iceberg-rest

# CI pattern (GitHub Actions): run it as a job service,
# point Spark/PyIceberg integration tests at http://localhost:8181,
# tear down with the job. No credentials, no cleanup, no flakes
# from shared dev catalogs.

One Config, Every Engine

Here is the interoperability payoff in practice. Four engines, four config snippets, one catalog server — every one of them boils down to the same three facts: the URI, the credential, the warehouse. A table created by any of them is immediately visible to all of them.

Spark

# spark-defaults.conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.prod=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.prod.type=rest
spark.sql.catalog.prod.uri=https://polaris.internal:8181/api/catalog
spark.sql.catalog.prod.credential=spark-etl:<client-secret>
spark.sql.catalog.prod.warehouse=s3://my-lake/iceberg
# table-scoped STS credentials instead of cluster-wide keys:
spark.sql.catalog.prod.header.X-Iceberg-Access-Delegation=vended-credentials

-- Spark SQL: normal Iceberg from here on
CREATE NAMESPACE IF NOT EXISTS prod.analytics;

CREATE TABLE IF NOT EXISTS prod.analytics.events (
    event_id   BIGINT,
    event_type STRING,
    user_id    BIGINT,
    ts         TIMESTAMP
)
USING iceberg
PARTITIONED BY (days(ts));

INSERT INTO prod.analytics.events
SELECT id, 'click', id % 1000, current_timestamp() FROM range(10000);

-- time travel works exactly as with any other catalog
SELECT * FROM prod.analytics.events VERSION AS OF 1 LIMIT 5;

Flink — streaming writes land as Iceberg snapshots other engines see on their next catalog load:

CREATE CATALOG iceberg_rest WITH (
    'type'          = 'iceberg',
    'catalog-type'  = 'rest',
    'uri'           = 'https://polaris.internal:8181/api/catalog',
    'credential'    = 'flink-etl:<client-secret>',
    'warehouse'     = 's3://my-lake/iceberg',
    'io-impl'       = 'org.apache.iceberg.aws.s3.S3FileIO'
);

INSERT INTO iceberg_rest.analytics.events
SELECT event_id, event_type, user_id, ts FROM kafka_events_stream;

Trino — native REST support since version 413:

# etc/catalog/lake.properties
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://polaris.internal:8181/api/catalog
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.credential=trino-query:<client-secret>
iceberg.rest-catalog.vended-credentials-enabled=true

PyIceberg — no JVM, no Spark, just Python against the same tables:

# pip install "pyiceberg[s3fs,pyarrow]"
from pyiceberg.catalog import load_catalog

catalog = load_catalog("prod", **{
    "uri": "https://polaris.internal:8181/api/catalog",
    "credential": "pyiceberg-job:<client-secret>",
    "warehouse": "s3://my-lake/iceberg",
})

table = catalog.load_table("analytics.events")

# incremental read into Arrow — this is a lightweight cron job,
# not a Spark cluster
df = (
    table.scan(row_filter="event_type = 'click'",
               selected_fields=("event_id", "user_id", "ts"))
    .to_arrow()
    .to_pandas()
)

# schema evolution through the same commit protocol
from pyiceberg.types import FloatType

with table.update_schema() as update:
    update.add_column("revenue", FloatType(), required=False)

The PyIceberg case deserves emphasis. Before the REST catalog, "small Python job that appends to an Iceberg table" usually meant a Spark cluster bootstrapped for a 30-second task, because catalog access required JVM connector code. With REST, table maintenance, incremental exports, and lightweight ingestion become plain Python processes. Entire categories of Spark jobs quietly stop needing Spark.

Credential Vending: The Real Reason to Migrate

Interoperability gets the headlines, but credential vending is the feature with the security team's name on it. The status quo it replaces is grim: every Spark, Flink, and Trino cluster holds long-lived storage credentials — IAM keys or instance roles — typically scoped to entire buckets, because per-table IAM policies are unmanageable at scale. Any compromised executor can read every table in the lake.

With vending enabled, the flow inverts. When a client loads a table with the X-Iceberg-Access-Delegation: vended-credentials header, the catalog calls sts:AssumeRolewith a session policy scoped to that table's storage prefix and returns temporary credentials in the load response:

# GET /v1/namespaces/analytics/tables/events  (response, abbreviated)
{
  "metadata-location": "s3://my-lake/iceberg/analytics/events/metadata/00005-abc.metadata.json",
  "metadata": { "format-version": 2, "...": "..." },
  "config": {
    "s3.access-key-id":     "ASIA...",          # temporary STS key
    "s3.secret-access-key": "...",
    "s3.session-token":     "...",              # scoped to this table's prefix
    "token.expiration-ms":  "1720483200000"     # ~1h lifetime
  }
}

The compute cluster ends up holding a token that can read one table's prefix and expires within the hour. A compromised job leaks access to the tables it was already reading — not the lake. Blast radius shrinks from "bucket" to "table, for an hour", and every access decision flows through one service that can log it.

Migrating from Hive Metastore Without Moving Data

The migration story is better than most people expect, because of what a catalog is. Since it only stores a pointer, moving a table between catalogs means copying a string — not data. The REST spec exposes this directly as register_table: read the current metadata_location from HMS, register it in the new catalog, done. The Parquet files never notice.

# Per table, the entire migration is:
from pyiceberg.catalog import load_catalog

rest_catalog = load_catalog("prod")

rest_catalog.register_table(
    identifier="analytics.events",
    metadata_location=(
        # read from HMS table properties: metadata_location
        "s3://my-lake/iceberg/analytics/events/metadata/00005-abc.metadata.json"
    ),
)

The dangerous part is not the API call — it is the window. If a writer commits to the HMS-backed table after you read its metadata_location and before engines switch catalogs, the REST catalog holds a stale pointer and that commit is silently orphaned. The safe procedure, per table: quiesce writers → read the pointer → register → verify snapshot count matches → flip engine configs → resume. Keep HMS read-only for a transition week as a rollback target, and only then decommission it. Batch tables by pipeline so each writer is paused exactly once.

Production Checklist