Class 02 · Movement 01
Software Engineering
Fundamentals
Fundamentals
Session 02 · AI Engineering
Seventy years of making it easier to tell a machine what to do.
1952
1957
1959
1972
1983
1995
2012
2025
1952A-0 compiler
Hid
Doing the translation by hand
Hopper's compiler turns readable notation into machine instructions.
New programmers
The same people — multiplied
The population didn't widen yet. Each programmer did the work of several.
Grace Hopper at the UNIVAC console · CC BY 2.0
The abstraction stack
Every layer up is easier — and can build less.
Low-code / no-code
Excel · Airtable · BubbleHigh-level & object-oriented
COBOL · C · Java · PythonAssembly
MOV AL, 61hMachine code
10110000 01100001Width = what you can still express · Height = how little you must learn
Not a rung
Agents
Claude Code · Cursor — you describe it in English.
Agents don’t sit on top of the pyramid. They write into the middle of it.No-code has a ceiling because the platform decides what you can express. An agent writes the same code a person would — so it doesn’t.
The proof
You are already looking at it.
Slide 03 · this deck, screenshotted
PowerPoint
Claude
OpenAI
KimiThe stakes
The market has already repriced per-seat software.
Prices: Yahoo Finance monthly closes, indexed to Jan 2025 = 100 · Market caps: stockanalysis.com · August 2026
On 3 February 2026 the S&P 500 Software Index fell 13% in a day — its worst on record. Almost none of the damage was revenue. It was the multiple.
The adoption gap · MIT Project NANDA · 2025
Everyone uses AI. Almost no business has changed.
100%
90%
of employees use personal AI tools at work
83%
of generic chatbot pilots reach implementation
5%
of custom, workflow-embedded GenAI pilots show measurable P&L impact within six months.
The other 95%: no measurable effect.
Small business is no different — 76% of SMB CEOs personally use gen AI (Vistage, Q4 2025).
Live pollSign in during class to answer
MIT Project NANDA, “The GenAI Divide” 2025 · Vistage Q4 2025
The role · live listings · captured 18 Aug 2026
Salesforce
Google
OpenAI
Screenshots of the companies’ own public job listings · sources in the course repo
Skill Map · LLM foundations · 1 of 3 · previews Session 06
Tokens in. Tokens out. That is the whole machine.
You type
write me a line of code to say hello world
It sees
Writemealineofcodetosayhelloworld
→
A function from tokens to a distribution
one next token, sampled
→
It emits
System.out.println("Hello,World!");
You read
System.out.println("Hello, World!");
It is not retrieving an answer and it is not reasoning in words. It is scoring every possible next token and picking one — about thirteen times to produce that line. Everything it gets right and everything it gets wrong comes out of that loop.
Compendium: “LLM Foundations: Tokens In, Tokens Out” · /research/#skill-llm-foundations
Skill Map · LLM foundations · 2 of 3 · where it leaks
Ask it to count the r s.
what you see
strawberry
what the model processes
strawberry
“How many r s?” asks about a unit that is not the unit being processed.
and the same sentence, in four languages
English8 tokens
Spanish11 tokens
Hindi19 tokens
Thai24 tokens
Tokens are the billing unit and the context unit, so the penalty lands three times: cost, latency, and how much fits.
But it is not only tokenization. Models recognise the letters and still cannot count them, and word frequency makes no difference (Fu et al., 2024). Counting needs reasoning depth that grows with the count — and transformers are constant-depth (Zhang, Cao & You, 2024). Two causes. The practical rule survives either way: unreliable below the token — so give it a tool.
arXiv 2412.18626 · arXiv 2410.19730 · arXiv 2505.14178 · language token counts are illustrative
Skill Map · LLM foundations · 3 of 3 · the window
A bigger window is not a filing cabinet.
The context window is everything the model can attend to in one pass — working memory, not knowledge. Fill it and the model does not simply hold more. It holds it worse.
a short promptthe whole handbook
accuracy
shape is illustrative — every one of 18 frontier models degraded; the onset differs by model
Same information. Two orderings.
a logically coherent document
the same sentences, shuffled
Models did better on the shuffled version — across all eighteen. Coherent prose appears to pull attention along its own logic as context grows.
Which is why “paste the whole handbook into the prompt” is not an architecture. You have to choose what goes in front of it — and that choice is the next three slides.
Chroma Research · Hong, Troynikov & Huber · 14 July 2025 · 18 models
Skill Map · Grounding · 1 of 3
The model has never seen a single one of their files.
What it learnedthe public internet, up to a date
no overlap
What they knowthe price list · the service manual · three years of tickets
oncea build step
1The business’s files→
2Split into chunks→
3Turn each into a vector→
4Store in an index
every questionwhere cost and latency live
1A question→
2Turn it into a vector→
3Find the nearest chunks→
4Paste them in the prompt
Only now does the model see their facts — as text you pasted, not as something it knows.Which means every failure has two possible homes: the wrong chunks came back, or the right chunks came back and the answer still went wrong. You have to measure them separately.
Compendium: “Grounding” · /research/#skill-grounding
Skill Map · Grounding · 2 of 3
A model only knows what you put in front of it.
Every partner business already has the material — the price list, the service manual, three years of tickets. None of it is in the model. So you go and get it. The interesting question is how good that retrieval has to be.
BM25 retrieval, one shotthe obvious pipeline
1.96%
ORACLE retrieval, one shotthe correct files, handed over
4.8%
An agent with toolslooks, reads, looks again
12.5%
SWE-bench — share of real GitHub issues actually resolved
Perfect retrieval only reached 4.8%. The ceiling was never how good the search was — it was that the model got one look. Doubling your retriever buys a couple of points; letting the model open a file, read it, and open another buys an order of magnitude.
Compendium: “Grounding” · SWE-bench arXiv 2310.06770 · SWE-agent arXiv 2405.15793
Skill Map · Grounding · 3 of 3
Neither retriever is good enough alone.
Keyword · BM25
finds the exact stringERR-4021 · PN-7734 · “Invoice 88213”
× misses “the unit won’t power on”
Vectors · embeddings
finds the meaning“machine won’t start” ≈ “unit fails to power on”
× smooths ERR-4021 into its neighbours
↘↙
Reciprocal Rank FusionYou cannot average the two scores — BM25 is unbounded, cosine lives in [−1, 1]. So RRF throws the scores away and fuses on ranks.
~100 candidatescheap and broad
→cross-encoder rerankexpensive and precise
→top 10 into the promptall the window can afford
Evaluate on their corpus, not on a public benchmark. Dense retrievers trained on one domain frequently lose to plain BM25 when tested zero-shot on another (BEIR, 2021) — and a partner business, with its own part numbers and its own words for things, is the out-of-domain case.
Compendium: “Grounding” · BEIR, Thakur et al., NeurIPS 2021
Skill Map · Agentic systems · 1 of 3 · previews Session 07
An agent is a loop with something to check against.
Plandecide the next move
Actcall a tool
Observeread what came back
Repeator stop
→ Deliver
The condition
A loop only converges if the world answers back. Tests that fail, a compiler that complains, a tool that errors, a screenshot you can look at.
Otherwise: don’t loop
Summarise, translate, classify — nothing observable comes back. One call is the correct design. Most shipped “agents” are workflows wearing the word.
Compendium: “Building Agentic Systems” · /research/#skill-agentic-systems
Skill Map · Agentic systems · 2 of 3
When one call is not enough: three layers.
The model contributes exactly one thing — a next-token distribution. Everything else is engineering. That is why the same model behaves like a different product in two harnesses.
01
The agentic loop
what it does on its own, between your ask and the answer
PlanActObserveRepeatDeliver
02
Context & knowledge
what it holds while it works, and what it goes and looks up
Context windowWorking memoryRetrievalThe business’s documentsCompaction
03
Connectors
how it reaches the real world — the tools it is allowed to call
MCPCLISandboxAPIsFiles
Every tool costs context on every turn, used or not. One MCP server has been measured at ~26,000 tokens across 35 tools; tool-selection accuracy has been reported falling from 43% to under 14% as the count grows. Cursor caps tools at 40 by default. Measured in the field, not in a controlled study — but a shipping product enforcing a limit is a company paying for the answer.This connector exposes 12, scoped per request.
Compendium: “Building Agentic Systems” · /research/#skill-agentic-systems
Skill Map · Agentic systems · 3 of 3 · the connector tax
Every tool you connect is paid for on every turn.
a 200,000-token context window
one MCP server~26,000 tokens · 35 tools
four moreup to ~30% gone
what is left for the actual work
Spent whether or not a single one of those tools is called.
and it picks the right tool less often
5 tools43%
2034%
4024%
60+14%
tool-selection accuracy, as reported — a threefold decline caused only by having more to choose from
More capability is not more capable. Cursor caps tools at 40 by default, from production telemetry — a shipping product enforcing a limit is a company that already paid for this answer. A tool surface is curated and cut, never accumulated.Measured in the field, not in a controlled study · this connector exposes 12, scoped per request
Compendium: “Building Agentic Systems” · practitioner measurement, 2026
Skill Map · Machine learning · 1 of 3
97% accurate. And worth nothing.
100 customers, one month
3actually churn
The model
“Nobody churns.”
97% accurate
Right about all 97 grey dots. Wrong about every single one that mattered.
Every problem a business actually cares about — fraud, churn, defects, no-shows, safety incidents — is imbalanced, because the interesting thing is the rare thing. Which is why “accuracy” is the first number to stop trusting, and the next slide is what replaces it.
Compendium: “Machine Learning Foundations” · /research/#skill-ml-foundations
Skill Map · Machine learning · 2 of 3
Two numbers, and they pull against each other.
model flagged it
model let it pass
it really happened
caughttrue positive
missedfalse negative
it did not
false alarmfalse positive
correctly ignoredtrue negative
Precision — read down the flagged column
Of everything the model flagged, how much was right?
Recall — read across the it-happened row
Of everything that really happened, how much did it catch?
A retention team with a fixed offer budgetwants precision — every false alarm is a discount thrown at someone who was never leaving.
A fraud or safety teamwants recall — every miss is a loss that has already happened.
Which error is worse is not a hyperparameter. It is a question for the owner. Asking it is the difference between a technician and an advisor — and it is the same question, exactly, that you will ask about an AI feature two movements from now.
Compendium: “Machine Learning Foundations” · /research/#skill-ml-foundations
Skill Map · Machine learning · 3 of 3
Ask how the test set was separated.
what it trained on
ageplantenureticketsaccount_closed_date
held out
what it was scored on
ageplantenureticketsaccount_closed_date
That last field only gets filled in when the account closes. It is the answer, wearing the costume of a feature. The model looks extraordinary in evaluation and is worthless in production — because at the moment you need a prediction, the field is empty.
294papers found with leakage
17scientific disciplines
erasedin one case study, correcting for leakage wiped out machine learning’s apparent advantage over older statistical methods entirely
If peer review misses this across seventeen fields, a six-week vendor pilot is not more careful. The whole defence is one question, and you can ask it in the first meeting: what was your held-out set, and how was it separated?
Kapoor & Narayanan, Patterns, August 2023 · /research/#skill-ml-foundations
Skill Map · Evaluation · 1 of 3
Two loops — and one calibrates the other.
Run the cases
A person reads it
Judgement & notes
Change one thing
Humanin the loop
Slow and expensive — and the only thing here with any authority.
→calibrates
Run the cases
A model grades it
Score against a rubric
Change one thing
Agentevaluations
Cheap enough to run on every change — once the human loop has calibrated it.
Same shape, same four moves. The only differences are who reads the output and what it costs — and the direction of trust between them, which only runs one way.
Compendium: “Evaluating Agents” · /research/#skill-eval-driven
Skill Map · Evaluation · 2 of 3
Somebody audited the cheap loop. It was at chance.
14,750 agent runs, graded by machine and by hand, then compared. Agreement is measured as Cohen’s kappa: 0 is chance, 1 is perfect.
00.250.50.7510.049substring matchingthe cheap check most teams ship0.432three-model ensemblemore graders0.567one small judge modelthe best of them0.835two humansthe benchmark being aimed at
0.049 is not a weak metric. It is a random number generator with a dashboard. The judge was agreeing with human graders no more often than chance would — on a scale where the two humans reached 0.835.
And the ensemble lost to a single small model, which should sound familiar: more graders, like more tools, is not better. One preprint, so treat it as a strong signal rather than a settled result — but before this, nobody had checked at all.
arXiv 2604.16706 — 14,750 traces · 13 agents · single-author preprint
Your first client · 1 of 4 · the process
Where the hours go.
The business you just heard from — The Gold Trader — runs every order through this loop in QuickBooks, and it takes long enough that they want it gone. Four boxes. That is the whole problem.
thegoldtraderstore.com ↗01Take it in
The jewelry and the order arrive. Every piece, price and client detail has to be collected and entered before anything else can move.
→
02Create the invoice
Line items, prices, client details — assembled into an invoice in QuickBooks, for every single order.
→
03Send it
The invoice goes out to the client, and the waiting starts.
→
04Confirm it
Follow up until the client confirms and the books can close. Then the next order starts the loop again.
Your job is not to admire this diagram. Somewhere in these four boxes there are hours to give back — finding exactly where, and what software collapses it, is the engagement.their words in the room beat anything on this slide — take notes
Your first client · 2 of 4 · the teams
Teams of three.
The class splits into small teams for this — three people each, self-selected. Nobody is assigned: you pick your own two.
team · 3
team · 3
team · 3
team · 3
team · 3
team · 3
team · 3
team · 3
Tomorrow morning, sign-ups go live. Log into Connect.AI, look at the open teams on your dashboard, and take a seat in one. The two people in the box with you are your team for the pitch.connect-ai.world · groups are already scheduled — they appear when they open
Your first client · 3 of 4 · the roles
One trainer. One scoper. One builder.
Why three? Because a forward-deployed team has three different jobs to do at once — and each one needs a single owner.
the trainerOwns the people.
Sits with the business and learns how the work is actually done — then, when the build lands, teaches the client to run it themselves. If the business cannot use it without you standing there, it is not shipped.
the technical scoperOwns the problem.
Turns what the business says into something buildable: maps the workflow, decides what is in and out of version one, and writes down what “working” means before anyone builds anything.
the builderOwns the build.
Takes the scope and makes it real — builds with agents, demos early and often, and cuts what does not fit the week. The pitch stands on what the builder can show.
“Generally” is the honest word. On a team of three, everyone does some of everything — but each of these jobs has exactly one name on it, and next Tuesday all three get tested at once.the framework we follow for the whole engagement
Your first client · 4 of 4 · the pitch
Next Tuesday, you pitch.
One week after the visit, the teams come back with answers.
5minutesYour team pitches
Next Tuesday, every team presents its solution to the client’s problem — the problem as you scoped it, and the thing you would build. Five minutes, all three of you.
→
1voteThe room decides
An in-class vote picks the winning pitch. Convince the room the way you would convince the client.
→
1teamThe winner builds it
The winning team gets the follow-up call with the business — and then actually builds the solution for them. Real client, real build.
This is the course in one week. Hear a business, scope its problem, pitch the fix — and if you win the room, build it for real.vote format may flex on the day — the winner still builds
pop quiz · 1 of 3 · part one · the argument
Every step up the abstraction ladder made programming easier. What did each step trade away?
ASpeed — higher-level code runs slower
BExpressiveness — the ceiling on what you can still build
CReliability — more abstraction means more bugs
DMoney — the better tools cost more
lock in your answer — then click
Width on the pyramid is what you can still express, and every rung up narrows it. Agents skip the trade — they write ordinary code, so they inherit no ceiling.
pop quiz · 2 of 3 · part two · the skill map
Connecting more tools makes an agent more capable. Why not connect everything?
ATools can conflict with each other and crash the agent
BMost tools sit behind paid subscriptions
CEvery tool is paid for in context on every turn — and picking the right one gets harder
DEach extra tool slows the agent's responses down
lock in your answer — then click
Tool definitions cost context whether or not they are ever called, eating the window the real work needs — and selection accuracy falls as the count grows.
pop quiz · 3 of 3 · part three · the engagement
One team, three seats. What are the three roles?
ADesigner · developer · tester
BManager · analyst · engineer
CPresenter · researcher · coder
DTrainer · technical scoper · builder
lock in your answer — then click
The trainer owns the people, the scoper owns the problem, the builder owns the build — one owner each, and everyone does some of everything.
session 02 · software engineering fundamentals
Thank you.
Tomorrow morning — sign-ups open on Connect.AI. Pick your team of three.
Next Tuesday — five minutes. Your pitch.
connect-ai.world
Connect.AIClass 02 · Software Engineering Fundamentals01 / 29