Designing a Risk Score That Your Team Will Actually Trust

Every fraud stack ends up with a number between 0 and 100. The difference between a good score and a bad one is rarely the maths — it is whether the people acting on it believe it.

Score the evidence, not the vibe

Start by writing down the raw signals and what each one is actually worth in your traffic:

| Signal | Typical weight | Why | | --- | --- | --- | | TOR exit node | High | Deliberate anonymity, rare among ordinary customers | | Hosting / datacenter | Medium-high | Servers are not shoppers | | Commercial VPN | Low-medium | Very common among honest users | | Residential proxy | High | Purpose-built to look legitimate | | Mobile carrier | Neutral | Shared, but overwhelmingly real people | | ASN abuse history | Medium | Cheap networks attract cheap attacks |

Those weights should come from your own outcomes, not from a blog post — including this one. Take a month of confirmed chargebacks and confirmed good orders, and check which flags actually separate them in your data.

Keep the reasoning attached

A score of 78 with no explanation is a coin flip with extra steps. Return the contributing reasons alongside the number, and show them in the review UI. Two things happen: analysts start catching rule bugs for you, and disputes get resolved in one message instead of five.

Calibrate with three bands, not one cut-off

• Low — proceed silently.
• Medium — add friction: email verification, 3-D Secure, a delay before payout.
• High — hold for review, or decline where policy allows.

Single thresholds force a binary choice on a continuous signal and guarantee that most of your mistakes are expensive ones. Bands let you spend friction instead of trust.

Never let IP decide alone

IP intelligence answers "what kind of network is this?" It cannot answer "is this person who they claim to be?" Combine it with account age, velocity, device consistency, billing/shipping agreement and payment history. The IP verdict should shift a decision, rarely make it.

Measure the false positive cost explicitly

Blocking a fraudster saves the order value. Blocking a genuine customer costs the order and the relationship, and it happens far more often than teams estimate. Track both sides:

• Confirmed fraud caught per week.
• Support tickets caused by risk actions per week.
• Reversal rate on those tickets — how often review says the block was wrong.

If your reversal rate is above roughly one in five, your thresholds are too aggressive regardless of what the fraud numbers look like.

Review on a schedule

Attack patterns rotate. A weight that worked in spring can be noise by autumn. Put a recurring calendar entry on score review, replay recent labelled cases against the current rules, and adjust. A score nobody has revisited in a year is a legacy system, not a control.

The trust test

Ask an analyst to explain a random high-risk decision from yesterday in one sentence. If they can, the score is working. If they shrug and say "the system flagged it", you have a number, not a signal.