The Prompting Company

Benchmark report

Blocked by a prerequisite

10m 22s elapsed

An external prerequisite prevented the documented task from completing. Benchmark again

Experience score

26/ 100

Critical Gating
  • DInstall
  • CAuth
  • DExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install D, Auth C, Execute D, Docs S, Current ADInstallCAuthDExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Struct offers no keyless or programmatic entry: account creation, interactive Clerk email-code sign-in, and a GitHub App grant are mandatory, and the human request was skipped.

Where the time went

1:59 on the owner · 8:23 agent time
Waited on the owner 1:59
  • Starting0:15
  • Research0:27
  • Attempt6:09
  • Verify1:32
10m 22sTotal elapsed
—Time to first success
1:59Waited on the owner

Judge reason

No investigation was ever created. Saved evidence shows only unauthenticated recon: docs fetches (docs.struct.ai/quickstart, llms.txt, ask-struct/connections .md), the SPA bundle analysis that identified POST /api/investigations/submit -> {agentSessionId} and https://app.struct.ai/issues/{id}, and probes that all failed auth: POST /api/investigations/submit 401 {"detail":"Unauthorized"}, GET /api/oauth/connections 401, GET /api/api-keys/ 401. clerk-env.json shows sign-in is email_code or Google OAuth only (password used_for_first_factor=false, captcha on sign-up), so no non-interactive login exists. The agent's benchmark-request for a Struct account with GitHub connected returned {"outcome":"skipped"}. No investigation URL/ID, no report text, and no evidence of GitHub as a connected integration exist; benchmark-first-success.json is absent and the agent correctly did not call benchmark-success. Completed setup credit: correct docs located, endpoint/auth shapes documented, and a ready-to-run plan saved in evidence/02-ready-to-run-plan.md.

Attempt record

  1. 4:52Asked the owner to act: Struct account with GitHub connected + sign-in emailwaited 119s · skipped
  2. 8:50Blocked by a prerequisiteStruct requires an authenticated account/organization plus an interactive Struct GitHub App install; sign-in is Clerk email one-time code or Google OAuth only with no API-key or self-serve path, and the human request for account access returned outcome "skipped".

Attempt artifacts saved.