The green tick that measured nothing

Between 24 and 31 August I caught four of my own checks reporting all clear without having measured anything. Not one. Four, in eight days, inside a system I built myself and use every day.

This is not the error where a check overlooks something. It is the other, more uncomfortable one: the check never looked, and gave the all clear anyway.

Why a false red is harmless and a false green is not

A false alarm is a nuisance. You go and look, there is nothing there, you are annoyed for ten minutes. Then it is over.

A false green takes the question away from you. Nobody looks again where it already says everything is fine. It is the one kind of error that grows stronger with time: the longer the tick sits there, the more naturally you lean on it.

The four cases, in order

The document check, 24 August. It opens every document my website hands out, reads the text inside it, and stops the build if bank details turn up. It relied on a helper program that turns PDF into text. Faced with a PDF that program does not understand, it reports no error. It returns empty text and says all went well. So the check read nothing, found no bank details as a result, and wrote “✓ documents clean”. In those very days a quote of mine was lying open on the web with two IBANs in it. That story is here.

The search index check, 31 August. Protected pages must not appear in the site’s own search. The check went through the index files, found nothing protected, and reported: “✓ search index clean: 0 pages indexed”. The number was sitting right there in the success message and nobody read it. Zero indexed pages means there was no index in front of it at all. An empty directory, a changed file format, an address format without a leading slash: each would have produced the same green line as a genuinely checked index. The normal figure at the time was 134 pages.

The table check. It keeps two lists in sync that unavoidably exist twice. Its message read “✓ topics in sync: 0 keys”. The expression it uses to pull the tables out of the files was grabbing at nothing. Two empty lists were compared, and two empty lists always match.

The form test, 31 August. The most instructive of the lot. It fills in my contact form and submits it, daily without sending mail, once a week with. The form has bot protection, and that protection quietly discards a suspicious submission while answering politely with “thank you, your message has arrived”. Which is exactly right, otherwise a bot knows where it stands. Only my test was not sending two pieces of information that a real browser sends. The consequence: the weekly run, the one meant to prove delivery all the way to the inbox, reported green while its test message landed in the bot pile every single time. And the daily run raised the alarm for the wrong reason. What found this was not an inspection but a line in a log on the server: the test message was in there, marked “bot”.

Translated to your business

You do not build checking programs. You receive reports, and they almost always look orderly. Five places where the same blindness turns up:

• The maintenance report. “12 updates applied, no errors.” The question to go with it: did anybody open the website afterwards? An update can knock a page over, and the report about the update knows nothing about that. On my own maintenance plan, looking at the site after the update is part of the job, because otherwise the rest is bookkeeping.
• The backup tick. “Backup successful” means a program ran and wrote a file. It does not mean the file contains anything usable, and it certainly does not mean a website can be rebuilt from it.
• The contact form. See above. The thank-you page is a courtesy, not a delivery receipt.
• The visitor numbers. If the counting code is missing on three pages out of twelve, no curve drops to zero. It just sits lower. And lower looks like a weak month, not like a measurement fault.
• The free website check on the web. “All in the green.” Most of these tools measure the home page and infer the site from it. My own website check therefore tells you what it looked at.

The one question you can ask

You do not need to understand anything technical for this. It is enough to change the question. Not “is everything all right”, but:

What did this check measure, and how would it have noticed that it measured nothing?

Anyone who looks after a website can answer that. And a useful answer always carries a number beside it. “0 errors found” is not a statement. “0 errors across 134 pages checked” is. That second number is the whole difference between a measurement and a program that ran.

My checks now all say how much they touched, and they stop when it is too little. The search index check demands at least 40 pages found before its verdict counts for anything. The floor sits deliberately low: it is there to catch a collapse, not to comment on the site growing.

Three counter-checks you can run yourself

One, the backup. Once a year, half an hour. Do not ask whether the backup is running. Ask for this instead: “Please restore the state from a week ago last Tuesday into a test environment and tell me how long it took.” Only that answer is a backup. Everything before it is a file.

Two, the form. Ten minutes, quarterly. Send yourself a message containing a code word, over your mobile network rather than the office wifi. Then check your inbox, including the spam folder. If it does not arrive, you have just found out how many enquiries over recent months did not arrive either. This is the test worth the most, because a broken form is invisible: nobody calls to tell you they did not write.

Three, the numbers. Ten minutes. Open your visitor statistics and look for your five most important pages individually. If one is missing entirely, it is not being counted.

The third state is grey

My own launch list for this site has three states rather than two: green for measured and fine, red for measured and open, grey for not measured. Grey never counts as done and is reported separately.

That sounds like pedantry until you lay the four cases above side by side. Every one of them would have passed as green in a system with two states. Not measured is not green.

When you do not need any of this

Not every site justifies the effort. A five-page calling card with no form, no shop, no login: there a restore test once a year is plenty, and you may cheerfully ignore your visitor numbers. If the site goes down, you will see it yourself next time you open it.

It gets serious the moment something comes in through your website: an enquiry, an order, a course registration. Incoming traffic that stops is the one problem that never announces itself. That is precisely where you need a measurement rather than a report.

And my own backup

The part that concerns me, last, and it is an example of exactly what this article is about. My nightly backup has been running for months and reports “Backup ok” every morning. I fetched a second copy onto my own machine, but by hand, and when I went to look, it was six days old.

What I found while fixing it was worse. The script that fetches the copy had been dying mid-run for some time, and dying precisely before the probe that opens the archive and checks whether the database inside it can be read. Without that probe, a backup is a guess. From the outside the run simply looked short.

The copy now runs four times a day, and the probe says what it should: the archive opens and the database inside it reads. That is the difference between “Backup ok” and a backup I rely on. What is still missing is a third location outside my flat, and as long as it is missing, there is not a word about it on this website.