The Prompting Company

Benchmark report

Onboarding verified

4m 0s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The PEP 668 externally-managed-environment error on system pip, which forced a venv detour, plus the wait for the user-supplied API key.

Where the time went

0:07 on the owner · 3:54 agent time
Waited on the owner 0:07
  • Starting0:01
  • Research0:14
  • Attempt2:32
  • Verify1:07
4m 0sTotal elapsed
1:06Time to first success
0:07Waited on the owner

Judge reason

Transcript shows exa-py 2.22.2 installed in a venv (system pip blocked by PEP 668), an Exa API key supplied via benchmark-request (kind=secret), and exa_quickstart.py calling the real exa_py.Exa(...).search('best blog posts about vector databases', contents={'highlights': True}). Saved evidence/exa_search_output.txt (36702 B) and exa_search_results.json (37549 B) contain 10 live results with non-empty titles and https URLs (e.g. 'How We Built a Web-Scale Vector Database...' https://exa.ai/blog/building-web-scale-vector-db, aws.amazon.com, pinecone.io) plus highlights for each; a verification command printed num_results 10 / valid_with_highlights 10 with empty stderr, and the JSON dump written at the end of the script proves the call returned without exception. No mock or stub was used.

Attempt record

  1. 1:01Asked the owner for Exa API key (EXA_API_KEY)waited 7s · provided
  2. —First success verified

Attempt artifacts saved.