214 AI tools, 90 days of hands-on testing: 7 patterns that changed how I review

Most “how I review AI tools” posts are 1,500-word brand defences dressed up as process notes. This is not that. This is the actual operating system behind 214 hands-on reviews shipped in 90 days - the patterns that held under pressure, the ones that broke, and the editorial decisions I would make again.

Every number in this piece can be verified by reading the session history (CLAUDE.md) at the MytheAi repo. If you have a competing catalog and want to compare notes, email me. The methodology is portable.

Why this piece exists

MytheAi went from 20 hand-tested tool reviews in early 2026 to 214 by mid-June 2026. The jump was not a sprint - it was the result of iterating the editorial process in public, documenting every decision, and reversing early patterns that did not scale.

The catalog now has 590 tools total. 36.4% (214) carry a MytheAi Team hands-on review. 374 (64%) are “Standard listing” - in the catalog but not yet first-person tested. Those labels are now visible on every tool card and a banner on every non-tested detail page. More on that in pattern 7.

Everything below is drawn from the public session log. Numbers you can audit:

• Hand-tested count on 2026-05-08: 95 (single-day sweep across sessions S133-S159)
• Hand-tested count on 2026-05-16: 145 (seven batches in one calendar day)
• Hand-tested count on 2026-06-13: 214 (eight batches, new single-day record)
• Current cohort coverage: 90+ distinct cohorts
• Comparison-surface peer-doubles: 64

If any of those sound inflated, pull the git log on CLAUDE.md. Every batch is timestamped.

Pattern 1 - Cohort completion compounds. Breadth does not.

The default instinct when curating a tool catalog is to open new verticals: a new category added each session feels like progress. For the first 60 days of hands-on work, that is what I did.

The data from month three said otherwise.

A “cohort” is a group of tools that compete head-to-head for the same job - Customer Service AI (Zendesk, Freshdesk, Intercom, Zoho Desk, Gorgias), or Code AI assistants (Cursor, Codeium, Copilot, Replit, CodeGPT, Blackbox-AI, Tabnine). Completing a 5-way cohort means every pairwise comparison between its members has hands-on data on both sides.

The compounding math: a 2-way cohort yields 1 /compare page with hands-on both sides. A 3-way yields 3. A 5-way yields 10. A 7-way yields 21.

In session S172 I made the decision explicit: when forced to pick between opening a new vertical and completing a cohort already in flight, completion wins. Not because new verticals are bad - because the second tool in an existing cohort is worth 2-5 permanent /compare pages, while the first tool in a new cohort is worth zero until the second one lands.

Lesson: optimise for the second N, not the first. “Where is the next hand-tested tool going?” is a worse question than “which existing cohort is one tool short of unlocking three comparison pages?”

Pattern 2 - Peer-doubles are the highest-leverage pattern in catalog work.

The sharpest version of pattern 1 is the “peer-double”: testing two direct competitors back-to-back in the same session. Cost: 20 minutes extra for the second review (context is loaded, test methodology is fresh). Return: an entire /compare page with hands-on data on both sides, which converts 2-3x higher than one with hands-on data on one side only.

By mid-June the catalog had 64 cumulative peer-doubles. Specific examples that paid out immediately:

• Zendesk + Intercom (CS head-term): the comparison page is now one of the highest-traffic /compare surfaces
• Tableau + Metabase + Julius AI (analytics 3-way)
• Elicit + Consensus (research AI, shipped together in S172)
• Sprout Social + Hootsuite (social media management, S173)
• Pandadoc + Docusign AI (e-signature, S179)

The gating constraint on peer-doubles is slug availability in the catalog. If a tool is popular but not in the catalog yet, you cannot peer-double against it - and adding a tool to the catalog is a separate work stream (Mode B batches in the editorial lingo). About 25% of my peer-double misses in the 90 days were caused by the second tool simply not existing in the catalog yet.

Lesson: when adding a new tool to the catalog, add its direct peer in the same session even if you were not planning to. Future-you doing a peer-double review 2 months later will thank you.

Pattern 3 - Catalog slug verification is 50% of the research time.

A professional embarrassment I was ready to hide, but it is more useful shared.

Before writing any new hands-on review, I run a “Block A” verification: query the catalog, confirm the tool slug still points to the thing I think it does, confirm the tool is still shipping, confirm competitor slugs for cohort reference. For most of the first 60 days, this took 5 minutes.

It now routinely takes 20 minutes of a 90-minute session. Two reasons.

Reason 1 - slug drift. In S175 Block A, I verified 8 candidate slugs. 4 of them had been renamed in the catalog since my previous reference (common renames: product rebrand, canonical slug cleanup, acquisition absorbing the original name). If I had skipped Block A I would have written reviews pointing at the wrong /tools/{slug} URLs.

Reason 2 - catalog gaps. In S164 Block A, I queried 21 candidate slugs for the batch - 11 were missing from the catalog entirely. These were well-known tools (Calendly, Customer-io, Monday.com, Microsoft Teams, Roam Research...). The “canonical” AI/SaaS tool list in my head overestimated how many were actually indexed.

Lesson: budget verification time explicitly. Treat it as research work, not setup. “I will test 5 tools this session” is false until you have confirmed all 5 slugs exist and point to the right products.

Pattern 4 - Zombie tools pass every automated check. You still have to open the homepage.

Dead tools are a reputational tax on a tool directory. If a reader clicks /tools/[slug] and lands on a parked domain or a 2021 acquisition announcement, the whole catalog loses credibility. Automated dead-tool detection catches the obvious failures (DNS failures, SSL errors, 404s) but misses the hard case: the “zombie” where the domain still resolves, the SSL is valid, and the homepage still loads - but the product has not shipped in 2+ years.

Two cases from early 2026 that passed every automated check:

• aWhere (aWhere.com): acquired in 2021, product line discontinued, domain still live showing a static company-about page.
• Prospera: acquired by Valmont Industries, customer-facing product offline, press/marketing pages still indexable.

Both were caught only when I noticed the tool detail page rendered with a letter-avatar fallback (because Google's favicon service could not extract a real favicon from the parked site). That is a signal, but it is a weak one - some live products also have favicon issues.

The permanent fix: manual dead-tool sweeps monthly. The MANUAL_DEAD_SLUGS list in the dead-tool-cleaner script grows every month. By mid-June it contained 15 confirmed-dead slugs that would otherwise have passed automated probes.

Lesson: zombie detection is a human problem. Automate the easy 95%; schedule the human sweep for the remaining 5%.

Pattern 5 - The 1,750-2,850 character range is the sustainable review length.

I did not pick this range deliberately. It emerged from pattern-matching across 214 reviews.

• Below 1,750 characters: the review reads as a vendor press release. The reader cannot tell whether I actually used the tool. Credibility collapses.
• Above 2,850 characters: diminishing returns on reader attention. Analytics show the drop-off after paragraph 4.
• Inside 1,750-2,850 (typically a 4-paragraph structure): sweet spot.

The 4-paragraph structure I converged on:

1. Intent + context - what I was actually trying to do, what my workflow looked like going in.
2. What worked specifically - the one or two moments where the tool did something a general-purpose alternative could not have.
3. What broke or surprised - the gotcha, the limit, the thing the marketing page did not mention.
4. Verdict with cohort - where this ranks against its 2-3 closest peers for a specific reader use case, with a /compare/{a}-vs-{b} link.

Each review also carries a separate 280-420 character “Avoid if” callout - a red-tinted block that lists specific user types or use cases where this tool is wrong. The “Avoid if” section converts better than the pros - it signals that the reviewer is not just trying to sell you something.

Lesson: length discipline is editorial discipline. The constraint pushes you to pick the two most-important observations instead of listing eight weak ones.

Pattern 6 - Rankings have to change publicly to earn trust.

In S140 I dropped Jasper from the top 3 of /best/writing. Jasper had held the #2 slot for 8 months. The score fell because the gap vs Claude + a system prompt grew faster than Jasper's roadmap closed it.

The change was visible to any reader who had bookmarked the page or had seen the previous ranking in a prior search. The ranking math is public (7-criteria scoring on /methodology), so the demotion was auditable - a reader could pull up the old and new scores and see which criteria had shifted.

Compare that to the alternative: quietly swap Jasper with the next tool in the list, hope nobody notices. That is what pay-to-rank directories do. It saves the vendor relationship. It kills reader trust the moment one person notices.

Four months on, the Jasper demotion is one of the most-cited pieces of editorial evidence when someone emails asking “how do I know your rankings are honest.” The answer is: because they visibly change, with the scoring math attached.

Lesson: when a ranking changes, make the change loud. Add a line to the money page explaining the shift. Keep the old ranking in the git history so a sceptical reader can diff it.

Pattern 7 - Transparency about what you did NOT test converts the gap into a feature.

The hardest pattern to accept, because it goes against the instinct to look complete.

MytheAi has 590 tools in the catalog. 214 (36%) have a hands-on review. The remaining 374 (64%) do not - they are in the catalog based on category coverage logic, but no editor has tested them first-hand yet.

For the first 90 days, those 374 tools looked identical to the hand-tested ones on the directory. Same card layout, same data, same scoring (auto-derived from third-party ratings rather than hands-on). A reader could land on a tool page and have no way to tell whether this was a first-person review or an auto-generated listing.

The fix shipped recently (session covering priority #5 in the roadmap): the catalog now has a “Hand-tested only” filter on /tools, a “Standard listing” label on non-hand-tested tool cards, and an amber banner on every non-hand-tested detail page linking to the hand-tested roster.

Counterintuitive result: readers now trust the hand-tested reviews more. Being explicit about the 64% gap signals that the 36% claim is defended. Before, “we have 590 tools” looked like scale theatre. After, “we hand-tested 214, here is what is standard-listed” reads as editorial honesty.

Lesson: your competitive advantage over a scrape-and-list directory is depth, not breadth. Breadth-only directories cannot label their gap because the gap is 100%. If you can label yours and still have 36% that is first-person, you win on the honest comparison.

What I would do differently

Three things I got wrong in the first 90 days:

1. I over-opened verticals. By the time I locked in “cohort completion compounds” in S172, I had 20+ half-complete cohorts. The next 60 days are mostly completion work.
2. I underweighted peer-doubles. The ones I did ship (64 cumulative) are the highest-converting /compare pages on the site. If I had started with the peer-double strategy from day 1, the number would be closer to 150.
3. I shipped the hand-tested filter too late. The “Standard listing” / amber banner work from priority #5 should have been a day-1 editorial surface, not a 90-day retrofit. The site was functionally less trustworthy than the actual editorial work deserved for 90 days because the labelling did not match reality.

Reproduce this analysis

Everything in this piece is defensible from the public session log. If you want to run the same meta-analysis on your own catalog work:

1. Keep a timestamped session log of every editorial batch (I use CLAUDE.md).
2. For every batch, log: which tools were added, which cohorts they completed, which peer-doubles landed, which candidates failed Block A verification, which content lengths worked.
3. After 90 days or 100 reviews (whichever comes first), extract patterns across sessions. The patterns are only visible in aggregate.
4. Publish the meta-analysis publicly. The transparency about your own process is the thing that compounds into reader trust.

If you run this on your own catalog, email the write-up to info@mytheai.com - I will link it here as a reply piece if the methodology holds up.

What is next

The next MytheAi research piece ships at the end of November 2026. The topic is a hands-on test (not a meta-analysis): a specific comparison of 8-10 tools on a repeatable task, with the dataset posted publicly. Follow /research or subscribe below to get it first.