Lately, my life has been incredibly hectic. While desperately trying to supplement data for my thesis, a practical collaboration project on the other side also presented a very real and extremely tormenting engineering challenge.
This article won’t delve into too much specific business logic; instead, it will simply discuss a huge performance pitfall we stumbled upon in our system architecture, and what absurd stress-relief behaviors an engineer might resort to when stuck debugging.
Has the Magic of Parallel Computing Failed?
The underlying architecture of this project is a system where Python handles front-end strategy processing, then hands it over to a Java core engine for high-speed computation.
The original operational model was a very well-behaved “serialization.” In other words, the system had to wait several minutes for one day’s massive data to finish processing before it could proceed to the next day. Seeing the server’s exquisite multi-core CPU sitting idle, as an engineer who pursues performance, I naturally couldn’t stand it. So, I decided to leverage parallel programming techniques to squeeze every drop of hardware performance.
However, something unexpected happened. I had thought that adding Parallel would make it automatically take off like magic, but I just checked the logs and found that even when I very conservatively only spun up 2 workers, parallel computation still failed spectacularly, and the entire system crashed and disconnected. (I tried 6, then 4, then 2, all failed)
The Finicky Java Engine and Slowpoke Python
Staring at the screen full of error messages, my initial judgment was: “This Java core engine is way too finicky!”
The error code displayed SocketTimeoutException. My immediate hypothesis was: this Java engine has extremely strict limits on “communication latency.” When the server simultaneously ran two Java engines and Python scripts to process massive full-day data, instantaneous CPU contention caused delays. Python couldn’t transmit thousands of data records within the specified “microsecond-level” time limit, so the Java engine ruthlessly severed the connection.
Sounds reasonable, right? But as someone who holds a skeptical view of technology, I always felt something was off. Would underlying resource contention really slow down communication enough to trigger a timeout?
So I continued to dig deeper and thoroughly went through the task.log from the parallel version’s disconnection. The result was an epic plot twist:
The Java engine was innocent, the true culprit was actually Python!
An Epic JSON Performance Catastrophe and Its Fix
The truth was revealed. Although SocketTimeoutException was thrown by Java, the root cause was that: Python’s built-in json.loads() processing speed was simply too slow.
During peak data periods (e.g., extreme days with traffic surges), a huge wave of state JSON would flood the system every 5 minutes. When 6 Java engines simultaneously bombarded Python with massive JSON payloads, Python’s native parser simply couldn’t keep up. This led to the socket becoming completely congested and blocked, and eventually, the Java engine, desperately waiting for a response from the other end, had no choice but to forcefully cut the connection.
After finding the root cause, I immediately implemented a global, epic acceleration fix.
I intervened decisively and replaced all JSON parsing tools within the system with orjson, a blazing-fast library rewritten in C/Rust at its core (its parsing speed directly crushes native Python by 10 to 20 times or more). I deeply modified the communication server’s source code and made all cross-language communication bridges orjson-driven.
After the changes, the system finally stopped timing out, data flowed incredibly smoothly, and parallel computation officially took off. It proved that in real-world engineering, there’s no such magic as “one-click acceleration”; all performance bottlenecks are hidden in the most inconspicuous I/O and low-level parsing.
The Performance Art After a Debugging Breakdown: The LinkedIn Easter Egg Project
Alright, the technical story is done. Let’s talk about what I did during the frustrating period of being stuck, before finding orjson, this savior.
When people encounter bottlenecks in problem-solving, they often do unrelated things to escape reality. So, in a fit of rage, I opened LinkedIn and spent several hours thoroughly revamping and organizing my complete resume and experience.
Since I was going to revamp it, I didn’t want to include cold, hollow, canned text. To commemorate these bittersweet university and master’s lives, I also buried many real and absurd “little easter eggs” deep within the profile (especially in the education section).
If you scroll through my LinkedIn and see a mysterious +16 photo stack under the National Taiwan University education entry, don’t hesitate, just click on it. There are no boring certificates inside, only things like “losing a bet and handing out 20 lunchboxes on the street,” “a hard-boiled egg Jellycat taking my graduation photos for me,” and various love-hate relationships between graduate students and GPUs.
If you happen to pass by my profile page, are willing to click and take a look, and successfully discover these easter eggs, feel free to screenshot or private message me to share! As a puzzle-solving reward, I can later grab some tasty snacks from the Google office to reward you 🍪☕️
May everyone’s parallel computations never timeout, and all JSON be parsed in a flash. See you in the next article!