Five Empty Stars

At the end of almost everything now, there is a row of five empty stars.

You finish a journey, and the stars appear. You eat a meal delivered to your door, and the stars appear. You speak to someone about your bank account, and afterwards a message arrives asking you to rate the conversation. You stay in a stranger's spare room, you buy a kettle, you see a doctor, you take a class, and at the end, invariably, the five stars are waiting, like a small jury that has been told to expect your verdict.

I would like to ask what the stars are measuring. And, since I ask it of every claim, I would like to ask the stars the question they never ask of anyone else.

Who are you?

The question

Here it is in one line.

When we rate people the way we rate products, what exactly are we measuring, and is it the thing we think we are measuring?

I will proceed by steps. I will state each step plainly so that you can find the one you disagree with, which is the purpose of steps.

Step one: what a rating claims to be

A rating claims to be a measurement of quality. It says: this was good, or bad, to this degree. It compresses an experience into a single number so that other people, who have not had the experience, can decide whether to have it.

For some things, that is a reasonable claim. A kettle either boils water reliably or it does not. A hotel room is either clean or it is not. When thousands of people report on whether a kettle broke, the average of their reports tells you something real about the kettle. The kettle does not change from one day to the next. It does not have a bad week. It is, as far as anyone can tell, the same kettle for everyone.

So let us grant the first step. For objects whose quality is stable and whose use is similar for everyone, averaged ratings carry real information.

Step two: what changes when the thing rated is a person

Now replace the kettle with a person.

A driver. A delivery rider. A person on the end of a customer service line. A tutor, a cleaner, a host.

Several things change at once, and each one bears on what the stars can measure.

First, a person is not stable in the way a kettle is. They have good days and bad ones. They are tired at the end of a twelve-hour shift and fresh at the start of one. Their performance varies, as all human performance varies, for reasons that have nothing to do with their skill or care.

Second, the experience being rated is shaped by things the person does not control. The traffic. The weather. The restaurant that was slow to prepare the food the rider is delivering. The policy the customer service agent is not permitted to change. The rating attaches to the person, but much of what produced the experience was not the person.

Third, and this is the step I would most like you to notice, the person doing the rating is not stable either. They are hungry, or late, or worried, or cross about something that happened before the journey began. Their verdict is partly a report about the driver and partly a report about their own afternoon.

So a rating of a person is a mixture: a little of the person's skill and care, a good deal of circumstance, and a measurable portion of the rater's mood. It is presented as a measure of the first. It is substantially a measure of the other two.

Step three: what the average does and does not fix

There is an obvious reply. Individual ratings are noisy, yes. But over hundreds of ratings, the noise averages out, and what remains is a fair picture of the person.

That reply deserves to be taken seriously, because it is partly true.

Random noise does average out. If raters' moods were randomly distributed, and circumstances were randomly distributed, a person's average over many ratings would tend toward something that reflected their actual work.

But two kinds of error do not average out, and they matter.

The first is systematic circumstance. A driver who works the night shift in a busy part of the city, with more traffic, more delays, more tired and frustrated passengers, will accumulate a different average from a driver who works quiet afternoons, even if they are exactly as good at their work. The noise is not random. It is patterned by where and when a person works, and the average faithfully preserves the pattern.

The second is bias. If raters, as a group, respond differently to people according to their accent, their appearance, or their name, then that response is not noise. It is a consistent error, and averaging a consistent error does not remove it. It enshrines it. I have not set out a figure for this here, and I will not, because I have not checked one for these purposes. I note only that the mechanism is plausible and that averaging cannot be relied upon to correct it.

So: averaging removes random noise and preserves systematic noise. A rating of a person, averaged, is a cleaner measurement of the person's circumstances and of the raters' habits than any single rating. It is not, for that reason, a clean measurement of the person.

Step four: what the number is used for

We now come to what the stars do, which is a different question from what they measure.

For a kettle, a low rating means fewer people buy the kettle. The kettle does not mind.

For a person, a low average can mean something much heavier. On many platforms, a worker's rating affects whether they are offered work, how much work, and whether they keep their access to the platform at all. I make no claim here about the particular threshold on any particular platform; they vary and they change, and I have not verified them. I make the general and widely reported claim that, for many people who work through apps, the average of their stars is tied to their livelihood.

Now consider what follows from steps two, three and four together.

A number that substantially measures circumstance and raters' moods is used to decide whether a person can earn a living. The person is, in effect, being judged on the traffic, the weather and the bad afternoons of strangers. And because the number is presented as a measure of quality, the person may come to believe that it is a measure of their quality. That they are a four-point-six person. That on a bad week they become a four-point-three person.

That belief is the error I came to examine.

Step five: the category mistake

There is an old idea in philosophy called a category mistake: treating a thing as if it belonged to a kind it does not belong to. Asking what colour the number seven is. Asking how much a promise weighs.

The five stars, applied to people, invite a category mistake. They place a person in the same category as a kettle: a thing with a quality that can be read off a scale. And people are not, in the relevant sense, in that category.

A person's work can be assessed. Whether the journey was safe, whether the food arrived, whether the question was answered: these are real and legitimate things to want to know. I do not object to them being known.

What a person is, by contrast, is not on a scale at all. The books in this House rest on a single premise: that a person's worth begins with their first breath, and that nobody can add to it or take it away. If that premise is correct, then a person's worth is not the kind of thing that has a degree. It is not four stars, or three, or five. It is not the kind of thing stars can describe, any more than the number seven can be blue.

The stars are a reasonable tool for describing a kettle and an imperfect tool for describing a journey. When they are mistaken for a description of the driver, they have moved from one category to another, and nobody has noticed the move.

Step six: the strongest objection

The strongest objection, as I understand it, runs like this.

Ratings protect people. Before ratings, you had no way of knowing whether a stranger offering you a lift was safe, or whether a cleaner was honest, or whether a host's room was as described. Ratings let millions of strangers trust each other enough to transact. Abolish them and you don't get a kinder world; you get a more dangerous one, and the people who suffer most are the customers with the least power to check for themselves.

That objection is strong, and much of it is correct.

Trust between strangers is a genuine problem, and some kind of shared record of experience is a genuine solution to it. A system that lets someone report that a driver was dangerous, or that a host lied about a room, protects real people from real harm. I would not want to remove that.

But notice what the objection is defending. It is defending the ability to flag a serious problem: danger, dishonesty, a promise broken. That requires something like a yes or no, with a reason. It does not require a five-point scale applied to every interaction, averaged to one decimal place, and tied to whether a person keeps their work.

The objection justifies a safety net. It does not justify a scoreboard. The two have been bundled together, and the bundle is what I am questioning.

What is now settled

Let me state what the steps support, and what they do not.

Settled. Averaged ratings carry real information about stable objects used in similar ways, such as kettles.

Settled. A rating of a person mixes the person's work with circumstances they do not control and with the mood of the rater. Averaging removes random noise but preserves systematic patterns of circumstance and any consistent bias among raters.

Settled, as a widely reported general fact rather than a verified particular. For many people who work through apps, these averages are tied to their access to work.

Settled, if you accept the House's premise. A person's worth is not on a scale, and treating a rating of their work as a rating of them is a category mistake.

Not settled. What should replace the stars. A system of yes-or-no flags for serious problems, with reasons, seems to me to preserve the safety the objection rightly defends while dropping the scoreboard. I have not tested that proposal, and I offer it as a direction rather than a conclusion.

Something that can be checked

I do not end with exhortations, but I will leave you with something small that can be done and observed.

The next time the five empty stars appear, before you touch them, ask yourself one question: am I rating the work, or am I rating my afternoon?

If the journey was safe and the driver was courteous, and you were merely late and cross, notice that. If the food was cold because the restaurant was slow, notice that too. You may still give whatever rating you think fair. But you will at least have separated the two things the stars are designed to blur.

And if you are one of the people whose living depends on other people's stars, I would like to say the following plainly, as a settled step and not as comfort.

Your average is a record of your circumstances and of the moods of strangers, with some of your work mixed in. It is not a measurement of you. There is no instrument that measures that, and there never will be, because it is not the kind of thing that has a measurement.

That, at least, I am prepared to call proven.