We Could Have Ranked Higher. We Chose Not to Pretend We Knew More Than We Did.

There is a particular kind of test that makes a young software company nervous.

Not a security audit. Not a production incident. A public benchmark.

Someone takes your API, tests it against a set of real addresses, publishes the results, and places you on a leaderboard next to companies with much larger teams, bigger budgets, and far more established reputations.

For a small company, that can be a remarkable opportunity. A strong result can bring attention, credibility, new users, and perhaps new revenue. A weak result can make people dismiss the product before they have even looked at how it works.

We recently found ourselves in exactly that situation with our email validation API, when DataCircle benchmarked us against dozens of other providers in a public email-validation comparison.

The benchmark exposed a product decision that we had not fully thought through from the customer's perspective. We changed that decision. It also exposed something more complicated: a difference in what we consider an honest answer when an email address cannot be verified with certainty.

This is not an article about who was right or wrong. The reviewer had a reasonable methodology, and we respect the work involved in creating and publishing the benchmark. The purpose of this article is simply to explain what happened, what choice we faced, and why we made the choice we did.

The opportunity was real

The benchmark result was not flattering.

On DataCircle's October 31 rerun, our API scored 50 out of 100 and ranked 65th out of 91 providers. That is never pleasant to see, especially when you are still building trust and trying to convince people to use your product.

Then we looked more closely at the test cases.

Some of the misses involved addresses on a known catch-all domain. A catch-all mail server accepts mail for almost any recipient address during the SMTP conversation. From the outside, person@example.com and definitely-not-a-real-mailbox@example.com may receive exactly the same response.

That creates a fundamental limitation: the server has confirmed that the domain accepts the message, but it has not confirmed that the individual mailbox exists.

Our API reports that situation as risky.

The benchmark counted those cases differently. Based on the observed outcome of sending mail, the expected label for the tested addresses was invalid.

That difference had a meaningful effect on the ranking. If we changed our behaviour to return invalid for those cases, our score would rise substantially and our position on the leaderboard would improve.

The temptation was obvious.

The uncomfortable part was not a technical failure

Before discussing the disagreement, it is important to separate two things that can look similar from the outside: a technical failure and a product decision.

The catch-all detection feature had originally been reserved for the Pro tier and above. That was intentional. We had decided that the Starter and Free tiers would use a more limited feature set, while higher tiers would receive more advanced detection and fresher results.

So when the benchmark account did not identify the catch-all domain, the system was behaving according to the product rules we had chosen. It was not a hidden bug that we had accidentally shipped.

But the benchmark forced us to look at that decision from the customer's perspective.

If a Free or Starter customer receives a result that says an address is valid because the system did not perform catch-all detection, while a Pro customer receives risky because the feature is enabled, then the difference is not merely about speed or convenience. It affects how much truth the customer gets about the address.

That conflicted with our transparency principle.

We do not want accuracy-relevant context to be hidden behind a paywall when a customer is making a consequential decision about whether to contact someone. We can reasonably charge for faster checks, more frequent refreshes, higher limits, priority processing, and other operational advantages. But we became uncomfortable charging for the basic information that an address belongs to a catch-all domain.

So we changed the product decision.

Catch-all detection moved from Pro-and-above only to every tier — Free, Starter, Pro, and Enterprise all receive the core signal now. The differences between the tiers can still relate to speed, SMTP timeout settings, priority modes, and freshness—for example, how often catch-all status is re-tested—but the core signal is available to everyone.

After this change, our benchmark score did not move. The Free and Starter tiers now correctly identify catch-all domains, but they still return risky rather than invalid—and the benchmark's scoring model counts risky as a miss. The tier change was about transparency, not about gaming the ranking.

This was not a bug fix. It was a deliberate change made because the old tier boundary no longer matched the standard of transparency we want to uphold.

That distinction matters to us. We are happy to charge for more capacity and more convenience. We are less comfortable charging customers to be told about a limitation that materially changes the meaning of an email validation result.

After making that change, another question remained: what should the API return when catch-all detection is working correctly, but the individual mailbox still cannot be verified?

What does "invalid" really mean?

For an email validation product, labels are not cosmetic. Customers use them to decide what to do with an address.

If an API returns invalid, most users reasonably understand that to mean:

This address has been determined not to exist or not to be deliverable.

If an API returns risky, the meaning is different:

There is a meaningful delivery risk, but the address has not been proven invalid.

That distinction matters.

On a catch-all domain, the SMTP server may accept every recipient. It may later reject some messages, or it may accept them and route them in a way that is not visible to the validator. A later bounce can provide evidence about a particular message, but it does not change what the validator could know at the moment it performed the check.

A pre-send validation API is not the same thing as sending a real message and observing its final delivery outcome.

That does not make one method automatically correct and the other automatically wrong. They are answering slightly different questions.

The benchmark asked, in effect:

Did this address ultimately behave as deliverable when mail was sent?

Our API is designed to answer a narrower question:

What can we determine about this address before our customer sends mail, based on the signals available to us?

For a catch-all domain, the honest answer to the second question is often: we cannot verify the specific mailbox with confidence.

That is why we use risky.

The easy way to improve the score

We could have added a rule:

If a domain is catch-all, return invalid instead of risky.

Technically, that would not have been difficult.

It would also have improved our benchmark position. The result would look better in a comparison table, and more visitors might click through to investigate the product.

But the same rule would affect customers using our API in the real world. They might validate an address on a catch-all domain, receive invalid, and remove a legitimate prospect from their list.

That is a false negative.

It is not as visible as a false positive. A false positive may lead to a bounced message. A false negative can quietly cause someone never to send the message at all. The lost opportunity is harder to measure because it leaves no bounce, no error message, and often no obvious trace.

We did not want to optimise our product for a leaderboard by giving customers a level of certainty the underlying evidence did not support.

So we chose not to change the result merely to improve the ranking.

This is not an argument against benchmarking

It would be easy to turn this into an angry article about benchmarks. That is not what we want.

Public testing is valuable. It can uncover bugs, expose vague marketing claims, and give customers information they cannot get from a product landing page alone. We appreciated that the test inputs and results were visible enough for us to investigate what happened.

We also understand why a reviewer may prefer a simple scoring model. A benchmark needs rules. If every provider receives a custom interpretation for every edge case, the comparison becomes impossible to read and difficult to reproduce.

The disagreement is about whether a catch-all result belongs inside a binary valid/invalid model at all.

In our view, email validation has at least three meaningful outcomes:

1. Valid: the available evidence supports that the mailbox is likely to accept mail.
2. Invalid: the available evidence supports that the address is not deliverable.
3. Risky or unverifiable: the domain or mailbox behaviour prevents a reliable conclusion.

Our API also uses unknown for cases where the server cannot be reached at all—for example, because of a technical limitation, a rate limit, or an unavailable probe. That is different from risky: unknown means we could not obtain enough information to assess the address, while risky means the available signals indicate a meaningful delivery risk or uncertainty.

A three-state model does not make validation useless. It makes the uncertainty visible.

Customers can then choose their own policy. For example:

• remove addresses classified as invalid;
• send risky addresses only in small, carefully monitored campaigns;
• if you are sending from a Chinese domain, also see our analysis of why Chinese domains often look suspicious to Gmail — deliverability can fail for reasons unrelated to the address itself;
• ask customers to confirm risky addresses through another channel;
• keep risky addresses but avoid treating them as fully verified.

That is more useful than hiding the uncertainty inside a binary label. If you are evaluating email validation providers, our buyer's guide covers what to ask about catch-all handling and label semantics before you commit.

Catch-all domains are not automatically worthless

There is another important nuance here.

We are not saying that every address on a catch-all domain should be discarded. We are also not saying that catch-all detection means a domain is unusable.

Some providers, including large consumer email services, may sometimes give a useful negative answer during SMTP. A clear rejection can still be meaningful. If the server explicitly says that a recipient does not exist, that is evidence worth reporting.

The difficult case is a positive response from a catch-all server. A positive response tells us that the domain accepted the recipient during the check. It does not prove that the named mailbox belongs to a real person.

So our approach is not:

Never validate catch-all domains.

It is:

Do not describe an uncertain result as certain.

That may be a less exciting sentence than "we verify every email address," but it is a more accurate one.

What we learned from the experience

Three lessons stayed with us.

First: not every useful tier distinction is a good one. If a feature changes how the customer understands the evidence, paywalling it can quietly undermine the trust the product is supposed to build.

Second: product labels deserve as much attention as product features. "Valid," "invalid," "risky," and "catch-all" are not interchangeable words — customers build workflows around them.

Third: a ranking is only as meaningful as the question behind it. A benchmark can be fair and transparent while still measuring something slightly different from what a product is designed to do.

Why we accepted the lower ranking

The opportunity was real.

A better score could have put us near the top of the list. That could have generated visibility and commercial opportunities. For a small company, those benefits are not imaginary. They matter.

But our product is not only an API endpoint. It is also the explanation behind the result.

When we tell a customer that an address is invalid, we are making a claim about reality. When we tell them that an address is risky, we are making a different claim: that there is enough uncertainty or delivery risk that they should be careful.

We want those claims to be defensible.

Changing risky to invalid solely because it improved a public score would have created a gap between what we knew and what we told our customers. That is not a gap we were comfortable with.

So we accepted the lower ranking.

Not because rankings do not matter. They do.

Not because the reviewer was acting unfairly. We do not believe that is a useful conclusion.

We accepted it because a benchmark result is temporary, while customer trust is cumulative. A high position may bring someone to your website once. A trustworthy result is what gives them a reason to stay.

What would you have done?

There is no universal answer here.

If you were running a small API company and discovered that changing one classification rule could move you from the middle of a public ranking to near the top, would you change it?

Would you prioritise the benchmark's definition of a correct answer, or the uncertainty your customers face in production?

Would you use a binary result because it is easier to compare, or a third category because it better reflects the limits of the evidence?

We chose transparency over the ranking in this case. Another company may make a different choice, and that does not automatically make it dishonest. Different products can have different goals, and different benchmarks can measure different things.

For us, the principle is simple:

If we do not know for sure, we should not claim that we do.

That is not a promise that our API will never be wrong. No email validator can make that promise honestly. It is a promise that we will keep improving the detection, explain the limitations, and avoid manufacturing certainty just because certainty produces a better-looking number.

And yes, we will keep validating catch-all domains.

We will simply continue to label them honestly.

Questions?

I'm Andy, the founder. I'm just an IT guy who likes solving problems for people. If you have questions about the email validation API, the labels we return, or how catch-all detection works in practice, just drop me a line.

📧 info@parheliaweb.com