How to judge an agent in ten minutes
Permissions first, measured numbers second, reviews third. The description is marketing and it goes last.
Have an order you follow every time, so you stop being persuaded by whichever listing is written best.
How to judge an agent in ten minutes works, in one picture
The same argument as the text, as a chain. Each step is what makes the next one possible.
- 1
Start at the permissions block, not the top of the page
Permissions and data access, on every listing pageThe first question is not whether it works. It is what it would be able to do if it went wrong. Scroll straight to the permissions and data handling block and read what it reads, what it writes and where it sends things.
If the answer is broader than the job needs, stop there. A summarising tool that wants write access is telling you something about how it was built.
- 2
Then the measured numbers, and notice the blanks
Success rate, median run time and cost per run are measurements from our sandbox, which has run nothing yet, so today they are blank everywhere. A blank is not a bad number, it is the absence of one.
Cost per run is the one people skip and then regret. Multiply it by how often you would actually use the thing, not by how often you imagine using it.
- 3
Then reviews, and only the verified ones
Reviews on this site come only from people who bought the listing, one per purchase. That is a much smaller number than a review count elsewhere, and it is a much more useful one.
Read the two star reviews first. People who are happy tend to write the same sentence; people who are disappointed tell you specifically what it did not do.
- 4
Run it before you pay
No listing can be run here yet. Running a package means executing code this site has not finished reading, and that waits on isolated infrastructure the owner has not chosen yet. What can be tried is a hosted server's connection: its page tests the handshake and the tool list, with no credentials. Anything else, run yourself in a container you are willing to throw away. Ten minutes there is worth more than an hour of reading.
Give it the awkward case rather than the clean one. Everything works on the clean one.
If you cannot try a listing and it has no refund window, you are being asked to take all of the risk.
You can evaluate a listing you have never seen before in ten minutes and say why you did or did not buy it.
Read next
Permissions first, measured numbers second, reviews third. The description is marketing and it goes last.
The near-monopolies and the commodities, side by side, because they look identical from outside and they are not.