The moment a build stops being personal, the risks change character. A tool only its builder touches can fail quietly, but a public one exposes secrets, data, and money to everyone on the internet, and recent incidents show how quickly working apps go wrong once strangers arrive. The builders in those stories didn't lack intelligence; they lacked a pass.

This lesson turns that pass into a short, repeatable habit: 4 plain-language questions about secrets, login, data exposure, and spending that anyone can answer without security expertise. It covers why the checklist form reduces harm even among experts, how to recruit AI as an adversarial reviewer of its own output, and where the checklist's protection honestly ends.

What is a pre-ship checklist?

A pre-ship safety checklist is a short, fixed set of questions a builder answers before making software public. For vibe-coded projects, 4 questions cover the ground that matters: whether the secrets are out of the code, whether anything that stores data or costs money requires login, whether a stranger can see data they shouldn't, and whether the spending is capped.

There's no single canonical version in the literature. This lesson synthesizes one from the checks Simon Willison laid out in March 2025 for people just getting started building software, covering low stakes, secrets, data privacy, and money on the line [1]. Andrej Karpathy states the norm behind it plainly: "You are not allowed to introduce vulnerabilities because of vibe coding. You are still responsible for your software, just as before." [2]

The checklist turns that responsibility into a few minutes of concrete work. It sits between building and sharing: once a project crosses the personal-tool line, the pass runs before anyone else can reach it.

Why checklists work

Checklists are a borrowed safety technology, not a software invention. The lineage runs through aviation's pre-flight procedures of the 1930s into modern medicine, where a landmark study of the WHO Surgical Safety Checklist found that deaths and complications dropped substantially after a 19-item checklist entered operating rooms.

The mechanism matters more than the history: checklists reduce harm because experts and novices alike skip steps they already know when working under time pressure. The vibe coder's version of that failure has a familiar sound: "it works, ship it."

That's why a pre-ship pass isn't an insult to anyone's competence. Knowledge doesn't prevent skipped steps under pressure; a short, fixed list does.

The secrets check

The first question: are all secrets out of the code? A secret is anything shaped like a password, most commonly the API keys that connect an app to paid services. The safe home for them is the hosting platform's environment variable settings, never the code itself, and especially never frontend code that every visitor's browser downloads.

The risk is measured, not hypothetical. GitGuardian found that AI-assisted commits leak secrets at roughly double the GitHub-wide baseline rate [3]. When a founder known as leo launched a SaaS built entirely with Cursor in March 2025, the API keys sat in the frontend where anyone could read them; within 2 days strangers had maxed out his usage and he shut the app down [4].

Karpathy hit the correct version of this while deploying MenuGen: his keys lived in a local file that never reached GitHub, and deployment broke until he added them manually in the platform's settings [5]. That friction is what a passing secrets check feels like.

The auth check

The auth check

The second question: does anything that stores data, costs money, or acts on a person's behalf require login? And critically, is that rule enforced by the server or database, not just drawn on the screen? A hidden button, a locked-looking page, or a paywall screen is decoration if the server behind it still accepts requests.

The pattern shows up repeatedly. leo's paywall existed only in the interface, so visitors bypassed the subscription within days of launch [6]. A May 2025 scan of 1,645 apps from the Lovable platform's own showcase found 170, roughly 1 in 10, where the interface looked normal while entire user tables sat readable underneath because the generated database configuration shipped without row-level security [7].

The general rule follows: the interface only looks like the app. The auth check passes when the rule holds even for someone who never loads the interface at all.

The stranger test

The third question: what can a stranger see? The stranger test means opening the app while logged out and trying to reach data, or asking the AI directly: "what can an unauthenticated visitor read or write in this app?" If the answer includes anything private, the check fails.

The Tea app made the archetype famous in July 2025: identity selfies sat in a storage bucket that required no login, only the URL, and roughly 72,000 images plus about 1.1 million private messages were scraped and posted publicly [8]. No one needed to break in; the door was never locked.

The test also counters a comfortable assumption: data isn't private just because no link points to it. Anything reachable without credentials should be treated as already public.

Pro Tip! A private browser window is the quickest stranger: it shows the app with no session, no cookies, and no login.

The cost cap check

The fourth question: is the spending capped? Every usage-billed service wired into a public app needs a spending cap or billing alert on the account, because a public app lets strangers spend the builder's money through it.

Willison flagged this from observed damage: "I've seen horror stories about people who vibe coded a feature against some API without a billing limit and racked up thousands of dollars in charges." [9] leo's incident followed the same shape: the exposed keys were his, but the usage that maxed them out came from strangers [10].

Caps feel like an operations chore, which is exactly why they get skipped. The check reframes them as part of shipping: no cap, no launch.

Pro Tip! Free tiers hide the risk until real traffic arrives, so the cap goes on before the service connects, not after the first bill.

Asking AI to attack the app

The checklist ends with an adversarial pass: asking an AI to attack the app and review it for security problems nobody asked about during the build. Functional testing proves the features work; it says nothing about what else the code allows.

The gap is well measured. Veracode's 2025 report found that roughly 45% of AI-generated code samples failed security tests, and that newer, larger models didn't produce measurably safer code [11]. The countermeasure is now institutional practice: Anthropic ships a security review command in Claude Code and an open-source GitHub Action that scans pull requests for injection, cross-site scripting, auth flaws, and exposed secrets, and it has caught vulnerabilities in Anthropic's own codebase [12]. As reported in May 2026, that review now also runs in real time while code is written [13].

Willison adds the human backstop: when other people will use the build, a quick check-in with someone more experienced belongs at the end of the pass.

"Nobody will find my app"

"Nobody will find my little app" is the misconception that keeps the other checks from happening. It feels true because the link was never shared. It fails because exposure is found mechanically, not socially.

Researchers located the 170 vulnerable Lovable apps by scanning 1,645 of them automatically [14]. Tea's open storage bucket needed only its URL to be found and dumped. Hosting platforms publish to public addresses by default; Karpathy found Vercel had auto-deployed his private, unfinished project to what he called "a totally public and easy to guess url" [15]. And the same agentic AI that builds apps now scans for exposed buckets, keys, and open tables at scale [16].

One nuance keeps this honest: obscurity genuinely does lower the odds for trivial personal tools. That's what the personal-tool line is for, and crossing that line is precisely what triggers this checklist.

A floor, not an audit

A completed checklist is a floor, not an audit. It catches the known killers, exposed secrets, missing auth, open data, and uncapped spend, which are the vibe coding equivalents of operating on the wrong site. It doesn't certify an app as production-ready, and treating it as certification recreates the false confidence that checklists are sometimes accused of producing.

The floor matters because the underlying risk is confirmed from several independent directions: static analysis, secret scanning, CVE attribution, and a large-scale academic analysis of AI-generated code in public repositories that reached directionally consistent conclusions [17]. None of that evidence says the 4 questions cover everything; they cover the common failure modes.

Above the floor, the graduation rule applies: anything holding other people's sensitive data at scale needs professional security review, and Karpathy's accountability norm doesn't expire once the pass is complete [18].