Table of Contents
Last spring a mid-sized logistics firm signed a three-year deal for a route-optimisation platform. The demo was flawless. Nine months on, dispatchers still keep the old spreadsheet open in a second window because the new tool can’t handle same-day rebookings. The software wasn’t broken. Nobody had tested it against the one thing the business does all day.
That failure mode is common, and it rarely comes from bad products. It comes from measuring a technology against its own brochure instead of against your operations. Working out how to evaluate whether a new technology is worth adopting is mostly disciplined scepticism: knowing what to measure, who to ask, and when to walk away.
Start With the Problem, Written Down
Before you take the first meeting, write one sentence describing what’s broken and what it currently costs. Not “we need better analytics” but “our month-end close takes 11 days and two people, and we want it under four.”
If you can’t attach a number to the pain, you don’t have a business case. You have a curiosity. That’s fine, but curiosities should be funded from a small discretionary budget, never from a multi-year commitment.
This step also hands you a rejection list. When the problem is specific, half the market disqualifies itself in the first call and you save weeks of demos.
Put It in Real Hands, on Real Work
Demos are theatre. The sales engineer knows exactly which path to take through the product, and the demo data has been cleaned by someone who cares. What you need to see is a Tuesday afternoon with a tired user and a deadline.
Run a trial of two to four weeks with the people who will live inside the tool. Give them live data and a real deadline. Then watch what happens around day eleven, once novelty fades and shortcuts appear. The workarounds people invent tell you far more about fit than any feature matrix.
Decide what “better” means before the trial starts
Pick two or three numbers and commit them to paper: quote turnaround from 4.2 hours to under 90 minutes, support tickets per 100 orders, hours per week of manual reconciliation. Two or three, not ten. A pilot without a defined success test quietly becomes a subscription, because nobody wants to kill a project that hasn’t technically failed yet.
Book the decision meeting at the start, not at the end. Organisations that do this kill weak initiatives in weeks instead of letting them drift for years.
Price the Whole Thing, Not the Licence
The subscription is the smallest line on the bill, and it’s the only number the vendor volunteers. Budget the rest before anyone signs:
- Implementation and data migration. Moving five years of records into a new schema is rarely a weekend job.
- The productivity dip. Experienced staff typically run 15-25% slower for the first four to six weeks. That cost lives in payroll, not in the contract, which is why finance teams miss it.
- Integration work. Anything touching your ERP, CRM or warehouse needs an engineer’s time, priced at engineer rates.
- Security and legal review. Weeks of calendar time, plus whatever the remediations cost.
- Renewal pricing. Year-one discounts are a customer acquisition cost. Ask what year three looks like and get it in writing.
Switching away from something you already run complicates the maths further, because you pay twice during the overlap. There’s a useful breakdown of that arithmetic in this guide to when technology migration actually pays off, with real ROI examples.
Watch What People Actually Do With It
Request usage telemetry at the 60-day mark. Not logins, since logins lie. Look at weekly active seats against seats paid for, and at the ratio of automated to manual steps in the workflow you bought the tool to fix.
Under 50% seat utilisation after two months usually means one of three things: the training was too thin, the tool solves a problem the team doesn’t have, or a manager bought it and never explained why. All three are recoverable, but only if you find out while the vendor still cares about your renewal.
The Moving-Target Problem
AI-based tools behave differently. Models get swapped, prompts that worked in March return nonsense in September, and accuracy drifts in ways nobody announces. A one-off evaluation tells you what the product did on test day, which is not what it will do next quarter.
For anything model-driven, insist on a small evaluation set of your own: 50 to 100 real examples with known-correct answers, rerun after every vendor update. Teams that build this habit treat a model change as routine maintenance. Teams that don’t spend their evenings debugging production. The field guide to rapidly improving AI products covers how to set up that feedback loop without hiring a research team.
Be especially sceptical where the underlying data is messy. AI prospecting tools, for example, are sold on match rates that look excellent until you test them against your own territory. This look at the state of AI sales intelligence shows where those numbers tend to fall apart.
Read the Exit Clause Before the Entry One
Three questions: can I export my data in a usable format, what does it cost, and how long after termination do you keep it? Vague answers mean no.
Then look at lock-in that isn’t contractual. Custom workflows built inside the vendor’s platform, scoring models that only exist in their cloud, staff who have learned one interface and no other. Switching costs compound quietly and never appear in a procurement spreadsheet. Designing for portability from day one is covered well in this piece on building sustainable digital solutions that survive a vendor change.
Ask the Questions Vendors Hope You Won’t
What happens to pricing if we double usage? Where does our data go, and is it used for training? Which subprocessors touch it? What was your uptime across the last four quarters, and can I see the incident log? Who on your team will still be assigned to us in six months?
The answers matter, but the willingness to answer matters more. A vendor that deflects on data handling will deflect on everything else, and you’ll discover it during your first outage. The argument for pushing hard here is made in the case for radical AI transparency.
Write the Decision Down, With a Kill Date
One page. The problem, the two or three metrics, the total three-year cost, the named owner, and the date you’ll reassess. Sign it and send it to whoever approved the budget.
Then do the part most organisations skip: book the reassessment now, six or twelve months out, and give someone the authority to cancel. Technology that was worth adopting in January can stop being worth it after a reorganisation, a price rise, or a better option arriving. A tool that solves your biggest bottleneck today may be solving a bottleneck you no longer have. The evaluation isn’t a gate you pass once, it’s a habit, and the teams that keep it spend noticeably less on shelfware than the ones that treat a purchase as a permanent decision.
