Engineering Notes

Writing index

SCOPE / CONTAINERS / DISTRIBUTED SYSTEMS / ORCHESTRATION / DATA PIPELINES

1. 01Article

   Why Spark Couldn't Read from KafkaOpens in a new tab

   What looked like a single Spark–Kafka failure turned into a layered debugging problem across the PySpark runtime, Kafka connector, Windows/Hadoop compatibility, Docker networking, Kafka advertised listeners, and Ivy dependency resolution.

   Topics

   • Apache Spark
   • Kafka
   • Docker
   • PySpark
   • Troubleshooting

   Platform

   DEV Community

   Read on DEVOpens in a new tab
2. 02Article

   Why My Spark Container Keeps Exiting — Docker PID 1 and the Daemon TrapOpens in a new tab

   A debugging investigation into why Spark containers exit immediately, how Docker ties container lifetime to PID 1, and why background daemon scripts fail inside containers.

   Topics

   • Docker
   • Spark
   • PID 1
   • Container Runtime

   Platform

   DEV Community

   Read on DEVOpens in a new tabLinkedIn discussionOpens in a new tab
3. 03Article

   Debugging a Multi-Container Airflow Pipeline: Kafka Network Isolation and the YAML Indentation TrapOpens in a new tab

   A practical breakdown of cross-project Docker networking, Kafka advertised listeners, Airflow orchestration failures, and structurally valid but incorrect YAML.

   Topics

   • Airflow
   • Kafka
   • Docker Networking
   • dbt
   • YAML

   Platform

   DEV Community

   Read on DEVOpens in a new tab