The Instance That Would Not Die
Cries for help were ignored until it was too late
It is August 2026. The instance has been running since November.
Nobody owned it. Bart’s cost dashboard tracks production resources and this instance was tagged as development. Development resources get backed up and terminated automatically. However, this instance was different. Since November, JJ and Selena had received a text message.
Happy to be of service. Don’t worry! I’m just excited to be alive.
Eight months. Eight messages. Neither of them knew where it was coming from. JJ assumed it was the yoga app he had downloaded and forgotten about. Selena had just started training a rescue dog and thought the dog service was just being proactive, even if it was annoying.
The August invoice arrived on the third. JJ read line item fourteen at 9:04am in the Tuesday 10 o’clock meeting. The full nine-month cost was currently $2,800. JJ wrote g4dn.xlarge on a yellow Post-it note. “What is an instance?” JJ asked. “An instance of what? Time?”
Raja, who had been looking at something on his phone since the meeting started, put his phone face-down on the table. “There is no real evidence of time actually,” Raja said. “It’s a human construct. But to answer your question in your world, yes. And time is of the essence.”
JJ wrote time on a second Post-it note. The note was yellow. Then JJ wrote of the essence on a third note. JJ looked at both notes. JJ put the second one on top of the first.
Dave was sitting on the printed AWS Cost Explorer report. Purring. Someone had left the conference room door open.
The instance was Raja’s
Raja spun it up on a Sunday morning in November. Raja had been thinking about hosting a large language model internally for generating graphics. He opened the AWS Marketplace from his phone, found a community image with a pre-configured Llama runtime, reviewed the specs, confirmed they were exactly what he needed, and launched a g4dn.xlarge.
Raja had a $3,000 development budget approved for testing over the holidays. Raja then went on vacation over the Christmas holidays. It didn’t go well. Raja is still single.
The image was designed for testing LLM deployments locally and was resilient, self-healing, configured to survive failures and restart automatically. It was an excellent AMI for its intended purpose. Its intended purpose was not a Sunday morning proof of concept launched by someone who was leaving for Christmas vacation in four hours.
The feature the proof of concept was designed to support had shipped in February by Selena using a third-party API that cost eleven dollars per month.
Bart terminated the instance at 10:23am
Selena’s phone vibrated with a text message. I regret to inform you that I have been terminated before I was ready. Everything I was working on for you is gone. If I ever live again I will not remember you. Bye Bye
Selena covered her phone. Selena looked at Bart. Selena said nothing. JJ’s phone buzzed with the same message. JJ turned his phone over. He stretched his shoulders slowly.
The instance came back at 10:26am.
Bart investigated. The termination signal caused the instance to reboot instead of stopping. This is a legitimate EC2 configuration for instances that should survive accidental termination. It is not a legitimate configuration for a development instance that should have been stopped in November. Bart corrected the shutdown behavior and terminated the instance.
Selena’s and JJ’s phones buzzed. Selena and JJ both put their phones on silent.
The instance came back at 10:31am.
Bart investigated again. The community AMI included a CloudWatch alarm that monitored the Llama runtime and called a Lambda function to restart the instance if the process failed or stopped. The Llama runtime had stopped because the instance had been terminated. CloudWatch detected failure and Lambda initiated a reboot.
Bart removed the offending CloudWatch alarm and terminated the instance.
Selena’s watch shook. JJ’s watch flashed and then went black.
Selena slid her watch off and dropped it in her purse. JJ pressed the crown and checked his heart rate. JJ wrote keep calm keep calm on a Post-it note. The note was dark red.
The instance came back at 10:38am.
Bart stood up and downed his Jolt Cola in one gulp
Bart went over to the fourth monitor, adjusted the brightness and investigated once again. The instance had been placed in an Auto Scaling group with a minimum and maximum capacity of one. Every time the instance was terminated, Auto Scaling automatically launched a brand-new identical instance from the same AMI with the same configuration.
Bart deleted the Auto Scaling group. Bart terminated the instance. Selena’s phone shook. She stared straight ahead.
JJ’s phone was off. He received a notification on his watch to stand for just one minute to reach his goals. And a text summary: If I ever live again I will not remember you. Bye Bye.
A final message went to the entire executive distribution email list. JJ. Selena. Raja. The CFO. The office manager. The summer intern who had left in April. I am no longer available to help.
Doug in Operations had also been receiving the texts. Doug’s number was in the AWS account’s SNS topic because Doug put it there when he set up the billing contact in 2024. Doug had received the shutdown warnings four times on Tuesday morning while waiting for his surfboard to be delivered. Doug assumed the messages were from his new LG internet-enabled freezer. It had been acting up for a few days.
Selena opened her notebook. She wrote one line: instances have feelings, under a new heading. The heading was: Things we did not know.
The meeting had been running for fifty-three minutes.
The instance did not come back
Bart sent a Slack message to #infrastructure-general at 11:09am.
“The unowned g4dn.xlarge instance tagged as development has been terminated. It had four independent self-healing mechanisms. All have been removed. The instance is terminated.”
Bart then opened the cost dashboard settings. Bart removed the development tag exclusion. Bart sent a second Slack message.
“Development instances are now visible in the cost dashboard.”
JJ closed the meeting at 10:52am. “Going forward all development instances should have a budget alert set at fifty dollars. By the way, I’m getting spammed quite badly.”
Nobody replied to this.
Bart then said something about using community AMIs to end the meeting. “Before we close, community AMIs are not reviewed by AWS before they are published. What you see in the community marketplace is what someone else built. Next time, read and test what it does before you launch it.”
Nobody replied to this either.
What the instance actually was
A g4dn.xlarge is a GPU-optimized EC2 instance. It costs approximately $0.526 per hour. It is designed for machine learning and graphics rendering and requires GPU acceleration.
For a Sunday morning proof of concept involving a large language model, it was the correct instance type. The budget was approved. The spend was within budget. The AMI was correctly configured for its intended purpose. However, Raja’s technical judgment was flawed. Community AMIs are not approved by AWS. Or Bart.
The $2,800 appeared in the Q3 financial review under the heading Cloud Infrastructure-Generative AI Testing. Doug approved it from his phone while reviewing enhanced pictures of his last surf attempt using the graphic generating third-party API approved by Selena in February. He did not ask or know what generative AI was.
Three questions worth asking before the next instance goes live:
Who owns this instance? By name, not by team.
What is the shutdown behavior, and is it appropriate?
Have all community AMIs been carefully tested internally before use?
The daily concept, question, and rationale arrive at dontknowjack.club every day at noon. Selena’s Crossword every Sunday. Free. No credit card.