Review of "Estimating the Productivity of an Autonomous AI Software Engineer" (Cognition, 2026)
Cognition’s AI Productivity Guarantee can settle up to $10M in credits. Their estimator uses AI to guess how long a human would have taken to do Devin’s work, then converts those hours into dollars.
Our review asks whether the measurement behind it stands up to scrutiny.
Key takeaways
- Tested against recall, not real engineering effort. 258 sessions, each scored against a user’s recollection.
- Errors of 2–3× per session. That’s fine for ranking work, but too loose to price it.
- Up to $10M, measured by the vendor. Cognition runs the meter that decides what it owes.
Before you sign up to a vendor-measured guarantee, know what’s being measured.
See the full review, including seven due-diligence questions to put to any AI vendor.
*Required fields. BlueOptima needs the contact information you provide to us to contact you about our products and services. You may unsubscribe from these communications at any time. For information on how to unsubscribe, as well as our privacy practices and commitment to protecting your privacy, please review our Privacy Policy.
