Distributed Workflow Engine
A personal Go project for concurrent job execution: a fixed worker pool, channels for handoff, context cancellation, and SQLite so unfinished work survives a process restart.
- Role
- Personal project
- Year
- 2026
- Links
- GitHub
- Stack
- Go
- SQLite
- goroutines
- channels
- context
Context
I built this to get comfortable with Go concurrency the way production systems use it — goroutines, channels, and cancelable work — without pretending it was Temporal or a multi-node cluster.
What I built
A CLI that can serve, submit, list, status, cancel, and wait on jobs. Serve polls pending rows, claims them, pushes work onto a jobs channel, and workers return outcomes on a results channel. Cancel flips the row and cancels that job's context. Crash leftovers marked running are reset to pending on startup.
Details
- Fixed worker pool with jobs and results channels
- Per-job context.WithCancel so cancel stops in-flight work cleanly
- SQLite-backed job store with pending, running, completed, failed, and canceled statuses
- Crash recovery: running jobs reset to pending when the engine starts again
- Task types for echo, sleep, and fail, kept separate from the engine core
Status
Open source at github.com/mgiwa78/distributed-workflow-engine. One process, concurrent workers, shared DB — not a multi-node system.