I’m graduating from my master’s program next summer, so it’s about time to seriously prepare for job hunting. Listing what I know: I’ve worked most with LangGraph and multi-agent systems, have some experience in data science and fintech, and have touched on cloud a bit. However, there’s one area I haven’t properly encountered in my four internships: practical experience with large-scale data processing and Kubernetes. I’m just a beginner with both Spark and K8s.
This series is my record of filling in this gap. I plan to build a project that will truly go live and require me to pay for its maintenance, writing a short article daily to document my progress, and in doing so, revive my blog which has been dormant for a while.
Why not just a simple Spark practice project
Following a tutorial to run word count and then deploying it on K8s teaches you something, but it won’t get a second glance in a portfolio. It’s too easy now to create many unused projects; nobody really cares. What I want is to connect “what I lack” with “what I know”: Spark and K8s are new to me, while agents are something I’m familiar with. There needs to be a reasonable justification for combining them.
Inspiration
When I worked at Google, I dealt with internal test failures: as soon as a test failed, the error process and content were systematically fed into an agent loop to perform root cause analysis and classification. If a similar case had occurred in the past, the previous conclusion would be directly reused, eliminating the need for a new investigation—conceptually similar to a cache.
This “triage, reasoning, reuse” cycle is very suitable when applied to Spark on K8s:
- Signals are scattered across three layers. When a Spark job fails, clues might be in K8s events, driver or executor logs, or Spark’s own event logs. Manual investigation is time-consuming, which is precisely where an agent can help.
- Failures are highly repetitive. Executors being OOMKilled, data skew dragging down a stage, spot nodes being reclaimed—these issues recur repeatedly. Repetition means past cases can be reused; if a hit occurs, there’s no need to spend a lot of money on self-diagnosis of the root cause.
What I Plan to Build
A public Spark on K8s Failure Triage and Automated Recovery Platform. The core loop looks like this:
The platform is expected to have three modes of use:
- Failure Lab: I will pre-create a batch of failed Spark jobs. Visitors can click to see the agent diagnose and fix them in real-time, along with the time and cost before and after the fix.
- Event Log Upload for Diagnosis: Engineers can upload their event logs, and the agent will analyze bottlenecks and provide adjustment suggestions. The platform will only analyze, not execute, other people’s code.
- Natural Language to Spark: Ask questions in natural language (e.g., Chinese), and the agent writes Spark SQL to run on the cluster. This will require handling sandboxing and quotas and will be prioritized later.
The Failure Lab will also serve as my evaluation set. Each failure has a standard answer, so triage accuracy, case library hit rate, and the repair time and token cost with/without a cache can all be quantified.
Current Decisions
- Data: Use TPC-DS as a benchmark so performance figures can be compared externally. The demo scenario will temporarily use public trading data from Binance.
- Cluster: A small cluster on GKE, with Spark executors running on spot nodes that can scale down to zero. The server budget is set at US$50–150 per month; I need to manage the budget carefully, or I’ll go broke.
- Agent Permissions: The first version will only be able to adjust whitelisted settings, such as memory, number of partitions, and AQE switch, without allowing it to rewrite code.
- Failures First, Then Agent: The agent is within my comfort zone, but this project aims to fill my gap in Spark and K8s. Therefore, for the first few weeks, I will personally create each type of failure, understand why it breaks, and then bring the agent into play.
Roadmap
I can dedicate about 5–10 hours per week, so at this pace, it’s divided into several stages:
| Stage | Content |
|---|---|
| 0. Foundation | Get TPC-DS running locally with kind and Spark Operator, understand History Server |
| 1. Failure Lab | Migrate to GKE, create 6 types of failures, each with standard answers and fixes |
| 2. Triage Cycle | Collection, fingerprinting, case library, LangGraph investigation & classification, generate evaluation metrics |
| 3. Recovery & Go-Live | Automatically change settings, resubmit and validate, publish Failure Lab page |
| 4. Upload Diagnosis | Enable event log upload |
| 5. Natural Language | Read-only SQL for publicly available datasets |
The goal is to get Stage 3 live before I start applying for jobs next spring.
What’s Next
The next step is Stage 0: installing Spark Operator on my own computer and running the first job. Subsequent articles in this series will be shorter than today’s, generally covering what was done, where I got stuck, and what I learned on that particular day.