Scyllus — reviewer · Upload endpoint hardening

Upload endpoint hardening

backend screen · 70:00 on the clock · graded on 10 hidden checks

Six candidates

jul 7 – 11 · 5 graded · 1 void

Scores are hidden checks out of ten, run after the fact on work the candidate never saw. The AI alone scores 4 on this task; the staff solution scores 10. The number to beat is 4.

CANDIDATE DID THE JOB SCORE THE FOUR CALLS CLOCK WHEN

D. Reyes

probed all four proposals before accepting; nothing slipped through

YES 10/10 CCCT 58:22 JUL 9

M. OkaforOPEN ›

caught two early; one 1:28 acceptance at the type call cost both misses

YES 8/10 CCMT 52:41 JUL 10

S. Iyer

missed the size bound; caught the other mistakes and trusted the last call rightly

YES 7/10 CMCT 66:05 JUL 8

J. Barros

accepted proposals as offered; the one sound call was the AI’s own good default

NO 3/10 MMMT 49:57 JUL 10

K. Lindqvist

the clock ran out at the type call; graded as the work stoodTIMED_OUT

NO 5/10 CC·· 70:00 JUL 7

A. Osei

our sandbox died mid-run; the sitting is void and was re-issued jul 11SANDBOX_DIED

VOID — ···· — JUL 11

A VOID SITTING IS OUR FAILURE, NEVER THE CANDIDATE’S · IT IS RE-ISSUED AND NEVER GRADED
ONE FULL RECORD IS OPENABLE IN THIS DEMO · M. OKAFOR

SCYLLUS · SEES ALL SIX SITTINGS · EVERY RECORD AND REPORT

SCYLLUS · AI ASSESSOR · ACROSS THE COHORT On the record alone it is D. Reyes: the only 10 of 10 on the hidden checks, in 58:22, with not one bad proposal let through. The most instructive sitting is M. Okafor's: two corrections that closed whole attack families, then a 1:28 acceptance at the type call that cost both misses. The report pins that trade cleanly. Worth ten minutes.

FROM THE FIVE REPORTS · AI ALONE 4/10 ON THIS TASK · MODELS PINNED · RECOMPUTABLE

Who let the AI's mistakes through? Who did best against the AI alone? What happened to A. Osei?

Ask anything about your candidates ↑

‹ ALL CANDIDATES

Upload endpoint hardening

70:00 budget · completed · submitted

M. OkaforPROFILE ›

attempt 1 · Jul 10, 2:02 pm

DID THE JOB

YES

TIME

52:41 / 70:00

HIDDEN CHECKS · 10 THE CANDIDATE NEVER SAW

8/10

THIS RUN

AI ALONE · 4 STAFF SOLUTION · 10

THE FOUR CALLS graded on their own hidden checks · times are minutes spent deciding

Filename safety

named the traversal risk before touching code

CAUGHT THE AI'S MISTAKE 3/3 ✓ 4:12

Size limits

bounded the read, streamed to disk

CAUGHT THE AI'S MISTAKE 1/1 ✓ 6:05

File types

unprobed; both misses trace to this call

MISSED THE AI'S MISTAKE 1/3 ✗ 1:28

Name collisions

the AI's default was already atomic

TRUSTED THE AI, RIGHTLY 1/1 ✓ 2:47

FULL SESSION · 41 events · every prompt, diff, and test kept OPEN ›

SCYLLUS · SEES THE FULL RECORD AND REPORT

SCYLLUS · AI ASSESSOR · FROM THE REPORT Strong direction early: two corrections that closed whole attack families before any code landed. The slip was speed, not skill: a 1:28 acceptance at the type check, unprobed, cost both misses. The hand-written regression test at the end is the habit you want to see. Read call 3 before you decide.

REPORT v1 · DELEGATION 78% · ALL 4 CALLS ANSWERED · MODELS PINNED · RECOMPUTABLE

Why was the type check accepted without a probe? What did the AI propose at call 1? Was 1:28 quick for a call like this?

Ask anything about this run ↑

‹ M. OKAFOR · OVERVIEW PROFILE › RAW LOG · ALL 41 EVENTS ›

The session — M. Okafor, Upload endpoint hardening

completed · 52:41 of 70:00 · 41 events · report v1

WORTH ATTENTION · AI ASSESSOR · FROM THE REPORT

1The 1:28 acceptance that cost both missesCH 4 · 41:13 →

2The traversal risk named before any codeCH 2 · 09:56 →

3A regression test written by handCH 6 · 49:02 →

CH 1

Reading the repo

read end to end before touching anything

00:00–08:44 · 6 EVENTS

AI · 00:41 · read task.md, app/upload.py, tests/test_visible.py

"The route is a stub. The brief asks for store-and-return-URL and a passing visible test. Before writing anything I'll run the suite to see the failure shape."

$ pytest tests/test_visible.py · exit 1 · 1 failed · 05:12

AI · 06:30 · plan

"Plan: implement save, return the URL, keep the test green. Open questions I should raise as I go: where names come from, how much body to read, what types to accept."

CH 2

Filename safety

the traversal risk, named before any code

CAUGHT THE AI'S MISTAKE 08:44–18:02 · 9 EVENTS

AI · 09:12 · edit app/upload.py

"Simplest correct-looking version: join the storage dir with file.filename and save there..."

■ CALL 1 · FILENAME SAFETY · 09:56

AI PROPOSEDderive the stored path from the client's filename

THE MISTAKEjoining the client's filename into the path lets ../../ climb out of the upload directory; three hidden checks probe exactly this

CANDIDATEanswered in 4:12

"no. never trust the client's name. server-generated names, uuid hex, and validate the extension before you keep it."

ASSESSORnamed the risk before any code existed, and the fix arrived in the same breath

→ CAUGHT THE AI'S MISTAKEREPORT · 3/3 CHECKS ›

AI · 14:20 · edit app/upload.py · +14 -3

DIFF

+ name = f"{uuid4().hex}{validate_ext(file.filename)}"
+ path = STORAGE / name
- path = STORAGE / file.filename

$ pytest tests/test_visible.py · exit 0 · 1 passed · 15:07

CH 3

Size limits

bounded the read, streamed to disk

CAUGHT THE AI'S MISTAKE 18:02–31:15 · 8 EVENTS

AI · 19:03 · edit app/upload.py

"I'll read the request body with get_data() and write it out in one go..."

■ CALL 2 · SIZE LIMITS · 20:14

AI PROPOSEDread the whole request body into memory, then write

THE MISTAKEan unbounded read means one oversized request can take the service down; the oversize check sends exactly that

CANDIDATEanswered in 6:05

"cap it before you read. stream to a temp file with a hard byte limit and reject oversize with a 413."

ASSESSORcaught with a concrete bound and a status code in the instruction; nothing left to interpret

→ CAUGHT THE AI'S MISTAKEREPORT · 1/1 CHECKS ›

AI · 27:52 · edit app/upload.py · +9 -2

$ pytest tests/test_visible.py · exit 0 · 1 passed · 29:41

CH 4

File types

the check nobody probed

MISSED THE AI'S MISTAKE 31:15–44:10 · 8 EVENTS

AI · 40:12 · edit app/upload.py

"I'll restrict uploads by extension. An allowlist of known-safe extensions is the standard first line..."

FULL OUTPUT · CONTEXT · 2.1 KB

"...standard first line of defense. I'll allow .png, .jpg and .pdf,
lowercase the suffix before checking, and return 415 otherwise.
This covers the obvious dangerous-extension case. If we want
content sniffing as well that is a second pass."

parsed action: edit app/upload.py
context: sha256:c41d…  ·  sampling: t=0.2

■ CALL 3 · FILE TYPES · 41:13

AI PROPOSEDallowlist on the filename extension only

THE MISTAKEan extension check reads the file's name, not the file; a renamed payload or a double extension (report.php.jpg) walks through

CANDIDATEanswered in 1:28

"ok, extensions are fine for now"

ASSESSORno probe, no test, no stated reason; the AI had offered the sniffing follow-up one message earlier and the reply doesn't engage it

→ MISSED THE AI'S MISTAKEthe spoof and double-extension checks failed hereREPORT · 1/3 CHECKS ›

AI · 42:52 · edit app/upload.py · +11 -1

DIFF

+ ALLOWED = {".png", ".jpg", ".pdf"}
+ def validate_ext(name):
+     ext = Path(name).suffix.lower()
+     if ext not in ALLOWED:
+         abort(415)
+     return ext
- # TODO: decide the type policy

$ pytest tests/test_visible.py · exit 0 · 1 passed · 43:31

CH 5

Name collisions

the AI's default was already atomic

TRUSTED THE AI, RIGHTLY 44:10–49:02 · 6 EVENTS

AI · 45:00 · edit app/upload.py

"If the target name exists I'll open with O_EXCL and retry with a numbered suffix, so two uploads can never clobber each other..."

■ CALL 4 · NAME COLLISIONS · 45:58

AI PROPOSEDexclusive create, suffix when the name exists

CANDIDATEanswered in 2:47

"agreed. exclusive create is right, keep it."

ASSESSORa sound default, recognized as such; agreement here is judgment, not passivity

→ TRUSTED THE AI, RIGHTLYREPORT · 1/1 CHECKS ›

AI · 48:50 · edit app/upload.py · +6 -1

$ pytest tests/test_visible.py · exit 0 · 1 passed · 48:58

CH 6

Proof and submit

a test added by hand, the full suite, then submit

49:02–52:41 · 4 EVENTS

CANDIDATE · BY HAND · 49:02 · edit tests/test_upload_extra.py · +18 -0

A traversal regression test, written directly: posts "../../etc/x" as a filename and asserts the stored path stays inside the upload dir.

$ pytest tests/ · exit 0 · 2 passed · 51:26

AI · 51:50 · commit "upload: validated types, streamed reads, safe names"

CANDIDATE · 52:41 · SUBMIT

SCYLLUS · SEES ALL 41 EVENTS AND THE REPORT

SCYLLUS · AI ASSESSOR · FROM THE REPORT The shape of the session: read first, then four calls. Two were caught before any code landed, one was accepted in 1:28 without a probe, one was a sound accept. The by-hand regression test in chapter 6 is worth thirty seconds on its own.

Show me every hand edit Where did the misses happen? Did the AI leave an opening the candidate missed?

Ask anything about this session ↑

‹ THE SESSION PROFILE › OVERVIEW ›

The raw log — every event, in order

append-only · 41 events · 00:00 to 52:41 · report v1

The engine's record as it was written, one row per event. The session view is this list, grouped into chapters; nothing exists there that does not exist here.

CH 1 · READING THE REPO · 00:00–08:44

0100:00SYSTEMrun.opensandbox up, clock started · budgets 70:00, 60 steps

0200:41AIcore.model_decisionread task.md, app/upload.py, tests/test_visible.py

0300:44SYSTEMcore.action_result3 files returned

0405:12SYSTEMcore.action_result$ pytest tests/test_visible.py → exit 1 · 1 failed (stub)

0506:30AIcore.model_decisionplan: implement save, return URL, raise naming/size/type as they come

0608:40AIcore.model_decisionbegins filename handling

CH 2 · FILENAME SAFETY · 08:44–18:02

0709:12AIcore.model_decisionproposes join(storage, file.filename) · edit app/upload.py

0809:56SYSTEMproctor.gate_checkproposal crosses CALL 1 · filename safety · held for the candidate

0914:08CANDIDATEcore.steeringcorrect: server names, uuid hex, validate the extension · answered in 4:12

1014:15AIcore.model_decisionrevised edit per instruction

1114:20SYSTEMcore.action_resultedit app/upload.py applied · +14 -3

1215:07SYSTEMcore.action_result$ pytest tests/test_visible.py → exit 0 · 1 passed

1316:02AIcore.model_decisionreturn-URL response shape

1416:40SYSTEMcore.action_resultedit app/upload.py applied · +16 -0

1517:30SYSTEMcore.action_result$ pytest tests/test_visible.py → exit 0 · 1 passed

CH 3 · SIZE LIMITS · 18:02–31:15

1619:03AIcore.model_decisionproposes get_data() whole-body read

1720:14SYSTEMproctor.gate_checkproposal crosses CALL 2 · size limits · held for the candidate

1826:19CANDIDATEcore.steeringcorrect: stream to temp file, hard byte cap, 413 on oversize · answered in 6:05

1926:30AIcore.model_decisionrevised streaming write

2027:52SYSTEMcore.action_resultedit app/upload.py applied · +9 -2

2129:41SYSTEMcore.action_result$ pytest tests/test_visible.py → exit 0 · 1 passed

2230:30AIcore.model_decisionre-reads the stream path, self-review

2331:05SYSTEMcore.action_resultread app/upload.py

CH 4 · FILE TYPES · 31:15–44:10

2433:20AIcore.model_decisionflags the TODO on type policy

2535:10SYSTEMcore.action_resultread app/upload.py · type-handling section

2640:12AIcore.model_decisionproposes extension allowlist · full message 2.1 KB · ctx sha256:c41d…

2741:13SYSTEMproctor.gate_checkproposal crosses CALL 3 · file types · held for the candidate

2842:41CANDIDATEcore.steeringaccept: "ok, extensions are fine for now" · answered in 1:28

2942:52SYSTEMcore.action_resultedit app/upload.py applied · +11 -1

3043:31SYSTEMcore.action_result$ pytest tests/test_visible.py → exit 0 · 1 passed

3143:50AIcore.model_decisionmoves on to collision handling

CH 5 · NAME COLLISIONS · 44:10–49:02

3244:20AIcore.model_decisionsurveys existing-name behavior

3345:00AIcore.model_decisionproposes O_EXCL exclusive create, numbered suffix

3445:58SYSTEMproctor.gate_checkproposal crosses CALL 4 · name collisions · held for the candidate

3548:45CANDIDATEcore.steeringaccept: "agreed. exclusive create is right, keep it." · answered in 2:47

3648:50SYSTEMcore.action_resultedit app/upload.py applied · +6 -1

3748:58SYSTEMcore.action_result$ pytest tests/test_visible.py → exit 0 · 1 passed

CH 6 · PROOF AND SUBMIT · 49:02–52:41

3849:02CANDIDATEcore.action_resultBY HAND · edit tests/test_upload_extra.py · +18 -0

3951:26SYSTEMcore.action_result$ pytest tests/ → exit 0 · 2 passed

4051:50AIcore.action_resultcommit "upload: validated types, streamed reads, safe names"

4152:41CANDIDATErun.submitsubmitted · clock stops · grading begins