Day 2: Naming, PRD, TDD, and a Skeleton That Doesn't Move Yet

Today is National Day—Happy Birthday, Taiwan! I grabbed a Starbucks today, and the barista drew a super cute illustration on my cup. Didn’t do much else today; the only important things were three meetings in the evening (AI Club, Digital Nomad, and Steering), plus booking my flights—26,800 NTD to take off.

Yesterday I mentioned that the next step was installing the Spark Operator. But I didn’t follow through today. Instead, I detoured to tackle upstream tasks: naming the project, writing the PRD and TDD, locking in the scope, and wiring up an empty skeleton that has the connections in place but no actual implementation yet.

Naming: Rekindle

The project is named Rekindle. Spark represents a flame; when a job crashes, the fire goes out. What this platform does is reignite it. At the same time, this series reignites my abandoned blog, so one name serves two purposes. AI helped me brainstorm it, and I’m very happy with it—plus, there’s a neat intertextual resonance between “spark” and “kindle.”

Writing the PRD and TDD First

Before getting my hands dirty, I usually write two documents (learned this from Andrew Ng—kudos to him!):

• PRD (Product Requirements Document): Answers “what to build and who it’s for.”
• TDD (Technical Design Document): Answers “how we plan to build it.”

The most useful section in the PRD is the user personas. I listed three: interviewers, data engineers running Spark, and myself. Once the interviewer persona was defined, certain design decisions practically made themselves. An interviewer’s attention span is only about two minutes, but scaling up spot nodes from scratch takes several minutes. Therefore, the fault injection lab defaults to replaying pre-recorded real runs, while live execution requires a separate quota.

The TDD follows the structure from my previous projects: the tech stack I want to build, design choices, data flow, evaluation, and failure modes. The “Failure Modes” section is the most valuable to write because it forces me to think about how the system could break before writing code. For example, the biggest risk for the knowledge base is a false positive—applying a fix for Fault A to Fault B. Thus, the false-positive hit rate is tracked as a formal metric, and cases that fail post-fix validation are demoted so they won’t be applied automatically again.

How to Lock in Scope

• Make non-goals more concrete than goals. At this stage, we won’t rewrite user code (only diagnose pain points), won’t build a general-purpose platform, and won’t include a natural language interface in v1.
• Have only one hard milestone. Phase 3 (public launch of the fault lab) must be completed by next February. All other phases can slide.
• Define “Done” for every phase. For example, Phase 0 is: “A single command can rebuild the environment and run a query, and I can explain on the History Server where the time went for a given stage.”
• Keep an explicit checklist of what the agent can do. Fixes can only be picked from a whitelist—currently 11 Spark configurations. Root-cause attribution can only be chosen from 7 fixed categories, one of which is “Cannot be fixed via config; requires human intervention.”
• Leave unknowns undefined for now. For success metrics, I only defined what to measure. Target values will wait until we establish a baseline. Throwing in an “80%” right now would just be guessing.
• Plan what to cut in advance. If time runs short, cut the Agno control group first, then cut the natural language interface.

Skeleton: Real Wiring, Empty Content

Once the documents were done, I moved on to scaffolding the skeleton. The principle is: get the structure in place first, leave the content empty for now. I had AI do the heavy lifting here, but I still reviewed everything carefully:

• Directories, database tables, API routes, and triage loop nodes/edges are all established.
• Each node and service only has function signatures and docstrings; calling them directly raises a NotImplementedError. This is a practice I picked up at Cathay: outline the functions you intend to implement first.
• Unneeded packages are omitted for now. The Gemini SDK and DB connections haven’t been added; we’ll connect them when the time comes.
• Placeholder numbers (embedding dimensions, max retry counts) are noted in the TDD’s unresolved items list.

The benefit of doing this is that discussions can center around something concrete. A triage loop diagram is just an image; once written as a compilable LangGraph workflow, questions like “Which step do we roll back to if validation fails?” can no longer be hand-waved away. Skeletons are cheap, too—refactoring the whole thing right now won’t hurt.

The iteration process was a back-and-forth between Claude Code and me: I provided direction and constraints, it drafted documents and the skeleton, and I made the decisions. Three decisions were finalized today: Gemini for the LLM, Gemini for embeddings, and leaving success metric targets open for now.

Three folders were intentionally left with only READMEs: infra/, spark-jobs/, and eval/. Spark and K8s are the core skills this project aims to build up, so I want to implement those by hand.

What the Tests Test Right Now

The nodes are still empty, but there are already 9 tests. They test the design, not functionality:

• Routes: /healthz works; three unimplemented routes return 501.
• Tables: Names of the 7 tables match the TDD.
• Loop Wiring: The graph compiles, and all 10 nodes are present.
• Branching Rules: Skip investigation on a knowledge base hit; do not proceed to remediation if the root cause is “requires human intervention”; retry invalid fixes up to the limit then give up; re-investigate on validation failure until the retry quota is exhausted.

The branching rules are pure functions—they don’t need Spark, K8s, or an LLM to be tested. These rules were originally just sentences in the TDD; once converted into tests, the spec and code won’t drift apart. Later, when filling in the node logic, as long as these tests still pass, I know the core workflow hasn’t been broken.

cd backend && uv run pytest

Two types of tests will be added later. One tests deterministic logic using fixed fixture files: fingerprint normalization, whitelist validation, event log parsing, and masking rules. The other is the evaluation set: every fault has a ground truth, serving as end-to-end tests for the entire system.

Current Stack

The directory layout looks like this:

backend/      FastAPI, data models, services, LangGraph triage loop
frontend/     React + Vite
faults/       Fault definitions and ground truth
spark-jobs/   TPC-DS and Binance PySpark jobs
infra/        kind, GKE, Spark Operator configurations
eval/         Evaluation scripts and results
docs/         PRD, TDD

What’s Next

The next step is truly Phase 0: setting up the Spark Operator locally with kind and running our first job. The first decisions to make will be which Spark version to use and how to generate the TPC-DS data.