AI Agents: The Case for Scripts in the Skills Spec

In January 1974 the US economy was falling off a cliff. Unemployment was about to run from 4.9% to 9%. The Sahm rule, the cleanest recession tripwire we have, reads 1.70 on that window. More than three times the level that says "this is a recession and it started months ago."

I handed Haiku 4.5 the same two years of monthly unemployment data and the same instructions, written as prose in a markdown skill file. It gave me twelve tidy three-month averages, a confident minimum, immaculate heading structure, and an answer of 0.03.

No signal. No recession. Nothing to see here, in the middle of the worst downturn since the war.

It didn't balk — it produced the format of correct work with none of the substance.

The claim

A skill, per the Agent Skills specification, is a folder of packaged expertise an agent loads on demand: a SKILL.md with a name, a description, and instructions, plus whatever else the job needs. The spec is deliberately permissive here. A valid skill can be a single markdown file, and a large share of what's on skills.sh is exactly that.

The spec also says deterministic procedures ship as executable scripts in scripts/, not as prose instructions. People read that as a style note. It isn't.

There are exactly two ways to disagree with it, and I've heard both a dozen times:

1. The model can just follow the instructions.
2. Fine, deterministic code. That's what a tool call is for. That's what MCP is for.

I ran an experiment against each one. Both objections lose.

First, what the Sahm rule actually is

Take the unemployment rate. Compute its three-month moving average. Compare that to the lowest three-month average from the previous twelve months. If the current average sits 0.50 percentage points or more above that twelve-month low, a recession has already begun. Thirteen averages, one minimum, one subtraction. The Fed watches it, and it has flagged every US recession since 1970. (It also fired in July 2024 with no recession following. Interesting argument, irrelevant here. I'm testing whether the model can compute the number, not whether the number is right.)

That makes it the test I wanted. It's multi-step, so it separates models that can chain arithmetic from models that can only look like they did. And its entire meaning lives at one threshold, so "close" is worth nothing. A miss of 0.15pp on a 0.50pp trigger is the difference between a recession call and silence. I scored exact match, not tolerance, for that reason.

Objection one: the model can just follow the instructions

Three computations from the macro-stats skill in my fed-briefing project, escalating in difficulty:

Task	Arithmetic
Yield-curve spread	one subtraction (10y − 2y)
CPI YoY inflation	one division
Sahm rule	thirteen averages, a rolling min, a subtraction

250 test cases sampled from real FRED history back to 1948. Ground truth from the skill's ~120-line stdlib script, cross-checked end to end.

Two arms. Prose skill: hand the model the instructions verbatim from a markdown-only skill file, plus the raw monthly values, ask for the number. Opus 5 with adaptive thinking on, and Haiku 4.5, because Haiku is the tier every "build a cheap agent" tutorial actually reaches for. Spec skill: run the script. Every trial logged the parsed answer, error against truth, latency, tokens, cost, and stop reason. Raw results.jsonl is in the repo.

Task	Arm	n	Exact	|err| mean	|err| max
Spread	script	500	100%	0	0
	Opus 5	65	100%	0.000	0.00
	Haiku 4.5	65	96.9%	0.000	0.00
CPI YoY	script	500	100%	0	0
	Opus 5	65	98.5%	0.003	0.18
	Haiku 4.5	65	87.7%	0.009	0.22
Sahm rule	script	500	100%	0	0
	Opus 5	120	98.3%	0.001	0.04
	Haiku 4.5	120	26.7%	0.153	1.67

The gradient is the whole story. One subtraction and everybody's fine. One division and the cheap model starts leaking. Thirteen chained averages and a minimum, and Haiku lands the exact answer 27% of the time, with a mean miss of 0.15pp against a 0.50pp threshold.

Read the last column again. Worst case error 1.67. That's not a wobble in the final decimal. That's the entire signal.

Unexpected failures

145 of 250 Opus 5 requests came back refused mid-computation. They only completed because I had a server-side fallback configured, which quietly rerouted them to Opus 4.8. My first pilot, before I set that up, failed 9 of 12 trials outright and I assumed I'd broken my own harness.

My read is that the reasoning-extraction classifier doesn't love long show-your-work digit chains. I can't prove that from outside; all I have is a correlation between "requires thirteen averages in prose" and "58% refusal rate." The practical point survives either way. If your agent does math in prose, its reliability isn't the model's arithmetic, it's the model's arithmetic times whatever the guardrail stack does that day.

The money

Arm	Total cost	Cost / correct	p50 latency	p95 latency
script	$0	$0	21 ms	42 ms
Opus 5	$5.36	$0.0217	7.9 s	19.1 s
Haiku 4.5	$0.73	$0.0048	3.7 s	6.6 s

Yes, Opus 5 with thinking basically solves it. 98–100%. If your answer is "just use the frontier model," you're paying two cents and eight seconds per number, roughly 375× the script's latency, on every invocation, forever. And you're still not at 100%.

The script's marginal cost of being right is zero. It was zero on run 1 and it'll be zero on run 100,000.

Objection two: that's what tool calls are for

This is the sharper objection and it deserves a real answer. Put the deterministic core behind an API, expose it as a tool, let the model call it. Same determinism, no local execution required.

Two problems. Take the boring one first.

A remote tool call is the wire. Encode, checksum, chunk, bearer token, gateway, job queue, execution service, poll, download, verify, re-expand. That's not a strawman enterprise setup. That's what an HTTP tool call does when you write down every step the agent is responsible for.

I built the same 62-line reconciliation core twice (skill-vs-wire): once as a local skill script, once behind that path, with seeded network latencies and a 3% transient-failure retry rate. Byte-identical output. Everything measured is transport.

Up front: I picked those latencies. This arm is a simulation, not a production trace, so read the timing numbers as illustrative. The two numbers that don't depend on my constants are step count and failure-mode count, and those are the ones that matter.

	Local script	Over the wire
Agent steps	2	9
Failure modes	2	11
Spec lines in context per turn	13	76
Median latency	24.6 ms	328 ms
Latency stdev	0.37 ms	104 ms (×277)

The local path is "run the script, read stdout." The wire path is nine steps, each one a place for the agent to lose the thread, mishandle a transient failure, or retry into a half-completed job.

The variance is the finding that should worry you. ×277 jitter means the agent can't form a reliable expectation of how long its own tool takes. Two runs in sixty took 700+ ms. Not failures, retries. An agent watching a slow response has no way to tell "normal tail" from "stuck," and that ambiguity is exactly what produces duplicate submissions against a system of record.

Now the interesting problem: the tool call usually smuggles objection one back in.

Tools get built as primitives. If your MCP server exposes get_unemployment_series, and the model computes the three-month averages from what comes back, you have not replaced the arithmetic. You've paid the transport cost and handed the model the calculation anyway. You're now in objection one and objection two simultaneously, at 27%, over the network.

A script returns 1.70. A tool call frequently returns twenty-four monthly values and a hopeful expectation. The spec's scripts/ rule is what keeps the whole deterministic procedure on the agent's side and makes it hand back a finished answer.

"But my MCP server runs locally"

Then my transport numbers don't apply to you, and you should ignore them. Three things still do.

Context tax. Tool definitions load on every turn. 76 lines against 13, in my case. Skills disclose progressively; tool schemas don't. Same finding from a different angle in monolith-vs-skill: the "put the policy manual in the system prompt" version of a loan-servicing skill runs 22,669 characters, 82% of it overhead re-read on every query. The spec version is a 17-line SKILL.md and a 73-line script with thirteen tests covering the edge cases.

Granularity. A local server still exposes tools, and tools get built as primitives. A script is a whole procedure by construction. The smuggling problem above comes from how the two get designed, not from the network.

Lifecycle. scripts/analyze.py lives in the same repo as the skill that calls it, versioned together, tested together, reviewed in one diff. An MCP server is a separate deployable, and "which version answered this question in March" becomes a question you have to be able to answer.

"The models are getting better"

They are, and it doesn't help. Accuracy is the one axis where waiting works. The 375× latency gap, the two cents a call, and the 58% guardrail refusal rate don't improve with model quality. Neither does the fact that a wrong answer arrives silently formatted. You can't buy determinism with capability. It's a property of running code.

Why the spec is written this way

Here's the part that decides it inside a large company, and it has nothing to do with tokens.

A deterministic script is regular software. It has a diff, unit tests, a version history, and the property auditors actually ask about: same input, same output, every time, with a log to prove it. When the Sahm computation lives in scripts/analyze.py, the answer to "how do you know that number is right" is thirteen unit tests and a git blame. When it lives in prose, the answer is "we asked the model nicely and it was right somewhere between 27% and 98% of the time depending on the model, the day, and the safety classifier."

One of those survives a risk review.

And the second-order effect is bigger than the first. Move the deterministic work into scripts and the model's remaining job is routing and narration. That's the part that genuinely needs AI-specific oversight, and it's small. Everything else is code review, CI, and change management your organization has been doing for twenty years. Boring is what compliance signs off on.

Write the script

The markdown skill tells the model to be a calculator. The tool call tells it to go ask one across the network, nine steps away, and hope the answer comes back the same. The spec skill hands it a calculator and puts the calculator under source control.

A crash is a bug report. A beautifully formatted 0.03 in a window where the truth is 1.70 is a briefing document that says the economy is fine.

---

Everything is reproducible. The arithmetic experiment is in fed-briefing/experiments/sahm-500, the transport experiment is skill-vs-wire, the prompt structural study is monolith-vs-skill. Total API spend: $6.09.