mach/data-stack
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Stack to support data stack local testing
6 commits 1 branch 0 tags 96 KiB
Find a file
Exact
Ricardo Machado 72bc4a4414 Adjustments to volumes.
2026-04-18 18:42:21 +01:00
config Adjustments to volumes. 2026-04-18 18:42:21 +01:00
screenshots First commit 2026-04-10 17:15:42 +01:00
.gitignore Added minio license 2026-04-13 18:46:06 +01:00
CONFIG.md First commit 2026-04-10 17:15:42 +01:00
docker-compose.yaml Adjustments to volumes. 2026-04-18 18:42:21 +01:00
env Adjustments to volumes. 2026-04-18 18:42:21 +01:00
README.md Added volumes for persistence between restarts. 2026-04-10 20:44:49 +01:00

README.md

Stack Overview

This repository defines a local multi-service stack with Docker Compose. The main orchestration file is docker-compose.yaml, and runtime variable values are provided by the root env file.

Because the environment file is named env instead of .env, start the stack with an explicit env-file argument:

Docker

docker compose --env-file env up -d

Podman

podman compose --env-file env up -d

Software Components

Service/Component Name Generic Description Documentation Link
Zookeeper A centralized service for maintaining cluster state, providing distributed synchronization, and providing a coordinator for other distributed applications. Apache ZooKeeper Documentation
Kafka An event streaming platform used to build real-time data pipelines, streaming applications, and data warehouses. Apache Kafka Documentation
Flink A scalable, fault-tolerant stream processing engine that can be used for real-time data processing and analytics. Apache Flink Documentation
PostgreSQL A powerful, open-source relational database system that supports a wide range of data types and complex queries. PostgreSQL Documentation
InfluxDB A time-series database designed for monitoring, analytics, and visualization of metric data. InfluxDB Documentation
MinIO An open-source object storage server that provides high availability, durability, and performance for storing and accessing data in the cloud. MinIO Documentation

Containers

Container Software component Description
zookeeper Apache ZooKeeper To be used by Kafka cluster.
kafka Apache Kafka Event streaming
flink-jobmanager Apache Flink Control plane by accepting jobs, coordinating execution, scheduling tasks, tracking checkpoints, and exposing the Flink web UI on port 8081.
flink-taskmanager Apache Flink Worker plane by executing the job subtasks assigned by the JobManager and providing the configured task slots used to run parallel work.
postgres PostgreSQL Relational database
influxdb InfluxDB 3 Core Time-series database
influxdb-ui InfluxDB 3 UI Browser-based UI for InfluxDB 3.
minio MinIO AIStor S3-compatible object storage

Environment File

The root env file supplies values used by Compose variable substitution.

PostgreSQL variables

Variable Default value Used by Description
POSTGRES_PASSWORD S3cret postgres Database superuser password passed to the Postgres image.
POSTGRES_USER bt_user postgres Initial database user created by the Postgres image.
POSTGRES_DB bt_db postgres Initial database name created at first startup.

Object storage variables

Variable Default value Used by Description
MINIO_USER minio minio Root username mapped to MINIO_ROOT_USER.
MINIO_PASSWORD pass4minio minio Root password mapped to MINIO_ROOT_PASSWORD.
MINIO_LICENSE ./minio.license minio Full path for the Aistore license file.

Exposed ports summary

Service Host port(s)
zookeeper 2181
kafka 9093
flink-jobmanager 8081
postgres 5432
influxdb 8181
influxdb-ui 8888
minio 9000 (api endpoint), 9001 (web interace)

For config steps after initial bootstrap see CONFIG.md