Skip to main content

Posts

Why Adding Spark Nodes Often Doesn't Fix a Slow Job: Check Partitions and Skew First

A Spark job is slow, so someone doubles the cluster. The job is still slow, and now it costs twice as much. If that sounds familiar, cluster size was probably never the constraint. This guide covers the checks worth running before approving more nodes, and it separates documented Spark behavior from judgment calls. The deciding question is simple: can the stage that is slow actually spread its work across more cores? If it cannot, extra nodes are a tax rather than a fix. A stage is as slow as its slowest task Spark runs one task per partition, and cores only help if there are enough similarly sized tasks to occupy them. The official tuning guide says a cluster will not be fully used unless parallelism is high enough for each operation, and it suggests two to three tasks per CPU core as a starting point. That guidance is a floor for utilization. It says nothing about how evenly the work is spread across tasks. A stage with 800 tasks can still be dominated by one task holding most ...
Post a Comment
Read more
Recent posts

Self-Healing Kubernetes Services Hide Failures Unless You Record Every Restart

A pod restarts itself at 2 a.m., the dashboard is green by morning and nobody opens a ticket. Did the system heal, or did it hide a defect? Kubernetes makes automatic recovery so easy that it is tempting to count every restart as a success. This guide is a framework for deciding what a restart can actually fix, how to configure probes so they do not cause outages of their own, and how to make sure every automatic recovery leaves a record someone reads. A restart only fixes problems that live in process memory Before adding any automatic recovery, ask one question: is the broken state disposable? A restart discards everything in the container's memory, so it clears deadlocks, wedged connection pools and corrupted in-process caches. It does nothing for a bad schema, an expired credential, a poisoned message or an upstream outage. If a fresh process will fail the same way, the restart is noise. Three criteria help for any self-healing mechanism: Is the failure local to the proce...
Post a Comment
Read more

Deleting the AWS Access Key Wasn't the Security Win: The OIDC Trust Policy Is

If your pipeline no longer holds an AWS access key, what stops someone else's workflow from assuming your deploy role? Moving CI/CD to OpenID Connect (OIDC) feels like a finished security task. I think it's only half done. The key is gone, but the question of who may assume the role is now answered by a condition in an IAM trust policy. If that condition is loose, you have swapped a secret you could leak for a boundary you can misconfigure. A note on sources: everything below comes from GitHub's and AWS's own documentation, which I re-read on 11 October 2026. The examples are adapted from it, with placeholders. This is not a benchmark or an incident from my own systems, and I'm not publishing any of my own trust policies. Where a point is my inference or my recommendation rather than something the docs state, I say so. What OIDC removes, and what it leaves for you to decide GitHub's documentation says OIDC lets workflows access AWS without storing AWS cre...
Post a Comment
Read more

Keeping SQS and Lambda Timeouts in Sync with Terraform

A Lambda consumer can succeed on a message and the same message can still run twice. Often nothing malfunctioned: two numbers that were never set together are interacting, the queue's visibility timeout and the function's timeout. The reasoning behind how those numbers should relate (including the recommendation of six times the function timeout plus the batching window) is covered in the earlier post, SQS and Lambda: Choosing Visibility Timeout, Retries and Dead-Letter Queue Settings . This one stays on configuration: how to make sure the numbers cannot quietly drift apart. Three settings, edited on different days The function timeout lives on the function. The visibility timeout lives on the queue. The batching window lives on the event source mapping. In a console-driven setup they get changed by different people on different days, each change reasonable in isolation. Lambda does check one relationship: the function timeout must be less than or equal to the queue's v...
Post a Comment
Read more

Draft-Only AI Agents on Blogger: Why the Credential Can't Be the Boundary

I run a blog agent that creates Blogger drafts and never publishes directly. It runs as an AWS Lambda function packaged as an ARM64 container image, its runtime credentials live in AWS Secrets Manager, and CI/CD authenticates to AWS with OIDC. I read every draft before it goes live. As a description of a workflow, that is fine. As a safety guarantee it is weak, because "a human reviews everything" says nothing about where the system makes publishing impossible. A bad refactor, a misread parameter or a model-generated change to the publishing step could quietly turn drafts into live posts. This post looks at where that boundary can actually be enforced when the target is the Blogger API, why the credential cannot carry it, and where it has to be enforced instead. It also covers what is still rough, because the setup is not frictionless. Where the setup stands today The agent only ever creates drafts, and publishing is a manual step after review. Recently I added logging: ...
Post a Comment
Read more

SQS and Lambda: Choosing Visibility Timeout, Retries and Dead-Letter Queue Settings

Picture a Lambda function with a 60-second timeout reading from an SQS queue. How long should the queue hide a message once the function has picked it up? The intuitive answer is 60 seconds, maybe a little more. The documented answer is considerably larger, and understanding why matters more than copying the number. This guide walks through the reasoning behind each setting that connects a queue to its consumer, and the failure modes each one is meant to prevent. Three questions drive the decisions. Can a message be processed twice without harm? How long should a genuinely failed message stay hidden before it is retried? And who looks at the dead-letter queue (DLQ), and how long do they have to act? The defaults answer none of these for you. A visibility timeout is a lease, not a limit on your code When a consumer receives a message, SQS hides it for the length of the visibility timeout. If the message is not deleted within that window, it becomes visible again. SQS cannot disting...
Post a Comment
Read more

Secrets Manager Rotation and AWS Lambda: Designing for Credentials That Change

Most Lambda functions read a secret once during initialization, keep it in a static field and never think about it again. That works until someone turns on automatic rotation. From that moment the value in memory can become outdated while the function is still running, and the symptom is rarely a Secrets Manager error. It shows up as a 401 or a database login failure somewhere downstream, which is easy to mistake for a configuration problem. This guide explains where the problem comes from, which strategies handle it, and what the rotation side needs to get right. The short version: rotation is a design decision for both the code that reads the secret and the code that changes it, not a switch you flip afterwards. Why cached secrets and rotation clash Lambda reuses an execution environment for many invocations, and code outside the handler runs once per environment. That is the right place for SDK clients and the wrong place for a credential that is meant to change. An environment...
Post a Comment
Read more