Architecture and startup

The coordinator plans and schedules; workers execute validated vectorized fragments and exchange Arrow partitions.

text

Client → coordinator: catalog → SQL → optimizer → stage DAG
Coordinator → workers: authenticated executable fragments
Workers → workers: bounded Arrow IPC exchange
Workers → coordinator: measured root results and task telemetry

Runtime responsibilities

The coordinator owns catalog resolution, planning, stage construction, assignment, retry, cancellation, and root-result collection. Workers advertise routable endpoints, validate versioned executable fragments, scan deterministic partitions, run physical operators, and publish exchange outputs.

Crates

core defines contracts; storage reads lake data; exec runs operators; sql and optim plan; catalog persists metadata; server coordinates workers; cli is the remote client.

Studio and FastAPI do not yet route analytical queries through Engine. Root results are synchronous, metadata is single-coordinator, and end-user Engine auth/TLS is not production-qualified.