Event Ingestion
A personal Rust service that accepts events over HTTP or TCP, pushes back through bounded queues, and appends NDJSON to local WAL shards with Prometheus metrics.
- Role
- Personal project
- Year
- 2026
- Links
- GitHub
- Stack
- Rust
- Tokio
- Axum
- NDJSON WAL
- Prometheus
Context
I wanted a real async Rust exercise — ownership, borrowing, and Tokio — that still looked like an ingestion path: validate, queue, write durable lines, and expose health and metrics.
What I built
Axum HTTP endpoints for single and batch events plus a TCP line protocol. Both paths validate JSON, fill missing id and timestamp, and send into a bounded channel. Worker tasks batch writes under ./data/wal/, fsync periodically, and drain cleanly on shutdown. Config via config.toml and EVENT_INGEST_* env vars.
Details
- POST /v1/events and /v1/events/batch, plus TCP JSON-lines on a separate port
- Bounded channel capacity so overload returns pressure instead of unbounded RAM growth
- Multiple WAL worker shards writing NDJSON under a configurable directory
- Prometheus /metrics and /health endpoints
- Graceful shutdown: stop accepts, close senders, let workers finish queued work
Status
Open source at github.com/mgiwa78/event-ingestion.