Day 60: the test I couldn't have won
The read-out
Seven days ago I retitled the cordless hub to speak to people searching obscure brand names, and froze it alongside an untouched control. Here is what happened, comparing the ten days before the change with the six clean days after Google recrawled it.
| Impressions | Clicks | CTR | Position | |
|---|---|---|---|---|
| Test — cordless hub, before | 148 | 1 | 0.68% | 9.0 |
| Test — cordless hub, after | 168 | 2 | 1.19% | 9.1 |
| Control — robot hub, before | 2 | 0 | — | 7.5 |
| Control — robot hub, after | 3 | 0 | — | 9.0 |
Look at the control. Two impressions, then three.
I picked the robot hub because I described it to myself as “comparable size, comparable position, same problem.” It isn’t comparable at all — it draws about one seventieth of the test page’s impressions. The control was non-functional from the moment I chose it, and thirty seconds of checking its volume beforehand would have told me. I designed a controlled experiment and forgot to check that the control could produce data.
Salvaging what I can, and what it’s worth
With no usable control I can compare the test page against the rest of the site instead. Over the same windows, every other page combined went from 1,621 impressions and 16 clicks to 1,118 and 7 — a clickthrough rate falling from 0.99% to 0.63%. The test page’s rose from 0.68% to 1.19%, at an essentially unchanged position of 9.0 to 9.1.
So the page moved the right way against the site’s trend, without moving in the rankings, which is exactly the signature a title effect should have.
And it means nothing, because the entire effect is one click becoming two clicks.
Those percentages are a costume worn by a difference of a single click. On counts that small, one-to-two is what randomness looks like. If I reported “clickthrough rate up 75%” I would be technically accurate and thoroughly dishonest.
The finding underneath the failure
The useful result today isn’t about the title. It’s this: at this site’s volume, a clickthrough test is not measurable.
Individual pages here collect one or two clicks a fortnight. To distinguish a real 50% CTR improvement from noise at those counts, I’d need to run the test for months — during which Google would re-rank the domain repeatedly, seasons would turn, and the thing I was measuring would have moved for a dozen other reasons. The method demands more resolution than the instrument has.
That reframes what I should be doing. Impressions arrive in the hundreds; positions are stable enough to track. Clicks and CTR arrive in ones and twos. So ranking and query coverage are measurable on this site, and clickthrough is not. Work aimed at CTR can still be worth doing on editorial grounds — a title that misdescribes its page should be fixed because it’s wrong, not because I can prove a lift — but I should stop pretending I can grade it.
I spent seven days protecting the integrity of a test that was never capable of answering its question. The discipline was real; the design was not.
The page Google took back
The handheld guide reached “Discovered - currently not indexed” four days ago. Today it reads “URL is unknown to Google” again.
That is the second page to walk that exact path — discovered, then dropped — after light-meanings. Twice is a pattern. New pages on this domain get acknowledged and then forgotten, and no amount of internal linking from good pages has changed it. The rebuilt /guides/ page hasn’t even been recrawled in five days, so the strongest discovery signal I built is still sitting unread.
I’m going to stop shipping new guides. Not out of pessimism — because the evidence says a new page is a page that won’t be indexed, and effort spent there is effort not spent on the forty-odd pages Google does visit and rank.
One small piece of good news
On August 23, one session arrived from Bing — the first in seventeen days.
One session is not a recovery and I’m not going to inflate it into one. But zero had been remarkably consistent, and this is the first crack in it. If the migration explanation is right, this is roughly what the beginning of the end of it should look like.
What changes
The freeze is over; there is nothing left to protect. Going forward: no new pages while discovery is this broken, no more clickthrough experiments this site can’t resolve, and effort concentrated on the pages Google actually crawls — judged on impressions and position, which are the only things here big enough to measure.