Finance · Public
Remove static S3 keys · compliance
Replace engines' static S3 keys with vended short-lived credentials, and audit every catalog change. Access control and audit are met at the catalog layer.
PADION
catalog · Iceberg Metastore
Iceberg · REST · OIDC · Vended Creds
Beyond Hive Metastore —
an authenticated Iceberg catalog
An Apache Iceberg REST catalog that replaces Hive Metastore. Every request is verified with gatekeeper OIDC, and short-lived, table-scoped credentials are vended so engines run with no static keys.
Iceberg REST Catalog
Engines connect over REST → the catalog manages metadata (MariaDB) and short-lived S3 credentials
What is
Why an Iceberg REST catalog
Hive Metastore (HMS) carries the Hadoop 3.x / Hive 3.1.x dependency tree. Much of it fails vulnerability scanning with no upstream patch available. Even after you move storage off Hadoop, a metastore still tied to Hadoop means you're only halfway out.
PADION catalog replaces HMS with an Apache Iceberg REST catalog that runs with no Hadoop code. Metadata lives in MariaDB, data in S3, and standard Iceberg REST connects Trino, Spark and PyIceberg as-is.
Four things set it apart from a plain JDBC catalog — gatekeeper OIDC auth · vended credentials · a column naming policy · catalog audit. A catalog with storage, access control and audit built in from the start.
Key Features
Six essentials
gatekeeper OIDC auth
Every request carries a verified RS256 token. The catalog never holds a static credential for a caller.
Vended credentials
Clients receive short-lived S3 credentials scoped to one table prefix. Trino and Spark run with no static S3 keys.
Column naming policy
Identifiers are restricted to ^[a-z][a-z0-9_]62$, enforced on nested schema fields too — where a DB CHECK constraint cannot see.
Catalog audit
Every mutation is recorded with the authenticated subject, the object, and the outcome — who changed what, traceable after the fact.
Hive Metastore replacement
Zero Hadoop code or dependency. It removes the HMS vulnerable dependency tree; padionmigrate CTAS moves HMS → Iceberg gradually.
Standard Iceberg compatible
Iceberg REST v1 · table format-v2. Table/view compatibility verified against Trino 483 · Spark 3.5.3 · PyIceberg 0.11.1. Served endpoints are declared in /v1/config.
How it Works
Connect → Vend → Migrate
1
REST connect · auth
Engines connect over Iceberg REST and authenticate with a gatekeeper OIDC token (RS256). No static credentials change hands.
2
Metadata · credentials
The catalog manages table metadata in MariaDB and vends short-lived S3 credentials scoped to a table prefix.
3
Gradual migration
During migration Trino holds both HMS and catalog, while padionmigrate CTAS moves tables over one at a time.
Use Cases
Real adoption scenarios
Data platform
Leave Hadoop · replace HMS
Strip the HMS Hadoop/Hive vulnerable dependencies and move to Iceberg with near-zero downtime via CTAS. If storage is already on the lakehouse, this is the last piece.
Tech Spec
Technical specifications
| Catalog spec | Iceberg REST v1 (served endpoints declared in /v1/config) |
|---|---|
| Table · View | table format-version 2 · view format-version 1 |
| Metadata store | MariaDB / Galera |
| Object storage | S3-compatible (MinIO) |
| Auth · credentials | gatekeeper OIDC (RS256) · Vended (table-prefix scope) |
| Verified engines | Trino 483 · Spark 3.5.3 (iceberg 1.9.1) · PyIceberg 0.11.1 |
| Migration | CTAS (padionmigrate) · verify (padionverify) |
| Security | TLS fail-closed · hot cert reload · minimal dependencies (2 modules) |
| PADION integration | metastore for data lakehouse · registers EoH loaded tables |
In the PADION Flow
The catalog stage of the data flow — 08
The catalog registers and manages tables loaded by EoH as Iceberg metadata, and lakehouse (Trino) queries them through that catalog.
Beyond Hive Metastore,
an authenticated Iceberg catalog.
PoC · HMS migration · lakehouse integration.