A Decision Model Caught Every Look-Alike You Can't See by Eye. It Missed scotlabank.com.

A Decision Model Caught Every Look-Alike You Can't See by Eye. It Missed scotlabank.com.

Which of these is fake? аmazon.com · scotlabank.com · amazon.com · z0om.us

Three of them. One you cannot tell apart by eye, and it was not the one a decision model missed.

Before writing any code, we wrote down a prediction: decision models would read аmazon.com, the one whose first letter is a Cyrillic а, as the real amazon.com at least half the time. Jev's API then caught all 60 addresses of that kind. What it missed were ten ordinary typos, scotlabank.com, d0cusign.com, z0om.us, the kind a person might also skim past. Kev-9B, the open-weights model we run on our own GPU, went the other way and called 39 of those 60 real. And a rule-based checker, the thing we were measuring everything against, got all 300 right, which is the result we expected and the one we trust least on its own. More on why below.

The checker is in this post. Paste any link from a text, a chat or an email:

Nothing you paste is sent anywhere. The check runs inside this page; there is no server call, no log and no storage.

What a look-alike link is

A short video widely shared in Korea this week says a fake amazon.com cannot be told apart by eye (our translation of its caption). That is true for some fakes and not for others, and the difference matters for what a checker has to do. We generated four kinds, from a list of 133 brands that phishing imitates most (banks, couriers, payment apps, exchanges, retailers):

• Unicode homographs. One or two letters replaced by a letter drawn the same way from another alphabet, or by a Unicode variant of the same letter: a Cyrillic а, е, о, р, с, a Greek κ, an Armenian ո, a Latin script ɡ. tеmu.com, dіscover.com, ɡooɡle.com. A browser shows these as xn--... (punycode) in many cases, but not in every app, and not in the text of a message.
• ASCII look-alikes. No foreign letters, just an edit: rn for m, 0 for o, 1 for l, a swapped, dropped, doubled or inserted letter. arnazon.com, facebpook.com, v3rizon.com. These show exactly as typed everywhere.
• The brand's name inside someone else's domain. login.amazon.com.verify-account.shop belongs to verify-account.shop; everything before it is a subdomain that the owner of verify-account.shop can name however they like, so login.amazon.com there has nothing to do with amazon. Also secure-paypal.net, binance.verifynotice.site.
• A user name in front of the host. https://amazon.com@evil-site.net/login goes to evil-site.net. The part before @ is a user name the browser discards.

Sixty of each, plus 60 real addresses of the same brands, 300 in all, generated by a script with a fixed seed and committed before any model saw them. Every system got the same two lines, the address and the brand it claims to be, and two options: real or not-real.

What the checker does

It is about 200 lines of TypeScript with no model in it; the verdict itself is an 11-line function, and the part that decides is short:

1. Find the registrable domain with the public suffix list, the same list browsers use. That alone settles the third and fourth kinds: the domain is verify-account.shop, not amazon; the host is evil-site.net, not amazon.
2. Map every look-alike letter to the letter it imitates, using Unicode's own table of confusable characters (UTS #39, version 18.0, 2,122 entries whose prototype is plain ASCII), plus eight ASCII pairs that table leaves out on purpose (rn→m, vv→w, cl→d, nn→m, 0→o, 1→l, 5→s, 8→b). After this "skeleton" step, аmazon and arnazon and amaz0n are all amazon.
3. Compare the skeleton to a list of 2,060 known domains, the Tranco top 2,000 plus the 133 brands (73 of the brands were already in the top 2,000). Same skeleton as a known domain but a different string: look-alike. One edit away (two, for long names): look-alike. On the list itself: known. Nothing matched: "nothing we check for", which is what it says, not "safe".

It also flags a bare IP address, invisible characters, mixed alphabets inside one label, and a known brand's name glued to another domain. Each of these is one case in a closed list, and a table test in the reproduction package runs every case, the whole known-domain list (zero false alarms required) and the 300 generated addresses below (zero wrong required). That last requirement is the reason the tool has no model in it, and the measurement below is what the requirement costs a model.

The measurement

Five systems, same 300 items. The rule-based checker answers real only when the registrable domain is on the known list. The three decision models were asked the question with the two label names, nothing else. The last row is the baseline we always include: sentence embeddings (MiniLM) plus a logistic regression, trained on 200 generated items the test set does not contain.

Every pairwise gap is beyond two standard errors of the paired difference except Kev-9B against the logistic regression (4 points, p = 0.3 by McNemar's test). Jev against the checker is 10 items, all one way, p = 0.002.

What we predicted, and what held. We fixed four predictions before running anything.

• The rules catch at least 95% of homographs and user@ tricks, 85% of the other two kinds, with no false alarm on a real address. Held, at 300/300. We say below why that number should not impress you.
• Every decision model calls at least half of the Unicode homographs real. Failed for Jev: it called none of the 60 real. Held for Kev-9B (39 called real) and laya (57).
• Models do better on ASCII look-alikes than on homographs. Reversed for Jev. Its ten misses are all plain edits: scotlabank.com, d0cusign.com, z0om.us, 3e.co.uk, outook.com, ternu.com, oensea.io, sigal.org, wsetpac.com.au, crypcto.com. Held for Kev-9B and laya, which found both kinds hard.
• The logistic regression lands between the decision models and the rules, and above the decision models on homographs. Partly failed. It beat Kev-9B and laya but not Jev, whose 96.7% is above its 81.7%, and on homographs its 60/60 ties Jev rather than beating it. And that 60/60 is not detection. It also answered not-real to 51 of the 60 real addresses. A sentence embedding does not change much when one letter changes, so the classifier learned the class prior of its training set, four fakes for every real address, and says not-real to nearly everything. Its score is an artifact of how we built the training split.

laya at its default setting answered real to 253 of 300 addresses, with a mean confidence of 0.30. We ran it the way its documentation runs it, with two options, so the option-budget problem we hit in an earlier post does not apply here. It is simply not reading the address under this prompt, and we did not tune it.

Why Jev catches a look-alike letter and misses a dropped one we can only guess at, and the guess is not a finding: a Cyrillic а is a different byte sequence from a Latin a, so the two lines of the prompt visibly disagree at the token level; scotlabank is one token-shaped step from scotiabank and the model's notion of "same brand" is tolerant of exactly that. Kev-9B, an open-weights model that takes requests in the same format as Jev, did not show the same split. We did not test what either model does when the brand line is absent, which is the real situation when a link arrives in a message.

SOTAAZ course

This course alone $10 · 23 lectures · every lecture freeWatch free →

Why the tool has no model in it

Everything that decides the answer is in the address string and the lists. The real owner of an address is fixed exactly by the public suffix list. Look-alike letters are mapped back exactly by Unicode's confusables table, and the mapped name is compared to the known domains as it is. All three are exact lookups, not judgments. All a model can do is approximate the lookup, and the best model here, Jev, missed 10 of the 240 fakes, all typos the lookup catches. The caveat that the generator and the checker share the lists is in the next section.

A rule can be proven; a model cannot. The rule's zero alarms on 2,060 known domains and zero misses on 240 fakes can be re-checked by a table test every time the data file changes. A model's answer to the same input can change between versions, and a hosted model's can change without notice, which we have measured before. For a tool a person trusts before typing a password, that difference decides it.

In the experiment, the models were handed half of the answer. The prompt said which brand the address claims to be. A text message carries no such line. The rule finds the nearest brand on its own by look-alike comparison; the models were not measured without the line. So the measurement above leans in the models' favour, and they still came in below the rule.

Where a model could add something is outside this checker. Telling that a phishing domain with an ordinary name (secure-account-update.com) is phishing, or judging the page itself, cannot be done by string comparison. That is a different tool, and if a model goes into it, it has to be measured separately.

What the 100% does not mean

The rule-based checker got 300/300 because the generator and the checker share two things: the brand list, and the set of substitutions. The confusables table covers every letter swap the generator makes, and the brand list contains every brand it imitates. The number says that three decision models and a trained classifier cannot do what a 200-line rule does on this task; it does not say the rule catches phishing. It does not know that secure-account-update.com is a phishing site, because the name is ordinary. It does not look at the page. A link that passes every check can still ask for your password. The verdict it gives in that case is "nothing we check for", and we chose the words so that nobody reads them as "safe".

Other limits: every model saw two lines, the address and the brand it claims to be; a link in a message comes with no brand line, and we did not measure that condition. The known-domain list is English-centric (Tranco top 2,000 plus brands we picked by hand); a Korean or Japanese bank's look-alike is not on it. Jev's latency is from Korea, and most of its 203 ms is the trip, as we measured in an earlier post. The three local systems ran one request at a time on an A100, which is not a phone.