We build and run a news site for a client that gathers stories from many outlets, and its political ratings for those outlets are now reviewed by hand. Its coverage balance view groups coverage of each story by the political leaning of the outlets reporting it, so you can see when a story is being covered almost entirely by one side. It only works if the leanings are right and I can explain each one.
For a while, that second part was not true.
How the ratings used to work
There were two sources. A short list in the code mapped well-known outlet names to a lean from -2 (left) to 2 (right). For outlets not on the list, a background job sampled recent headlines and asked an AI model to place the outlet on the same scale. The answers were stored and used as a fallback.
It was clever, and it filled the gaps quickly. It was also the wrong tool for the job, for three reasons.
- Nobody could defend an AI rating. If a reader asked why an outlet was rated centre-right, the honest answer was "a model read some headlines". That is not good enough for something that frames how people read the news.
- It rated things that are not news. Gaming sites, music magazines and trade titles were getting political leanings from their headlines. A games review site does not have a position on the spectrum, and giving it one skews the picture.
- It could not be repeated. A rating depended on whichever headlines happened to be sampled, so the same outlet might be placed differently on another day.
Political ratings from one list, reviewed by a person
The AI fallback is gone. There is now one list, which I review by hand, and an outlet that is not on it is unrated. Unrated is a perfectly good answer. Only general news and politics outlets belong on the list at all.
When I add feeds from new outlets, I rate them on the list first, before their stories start arriving, so the balance never drifts while I catch up.
Matching names properly
The old matching checked whether the outlet's name contained a short string. That caused quieter errors than the AI did, but they were errors all the same:
| Problem | What happened | Fix |
|---|---|---|
| A short name inside a longer one | "The Times" matched "The Times of India", a different paper | A blank entry for the longer name stops it falling through to the shorter one |
| A common word | "Independent" matched any outlet with "independent" in its name | Match "The Independent" and its domain, not the word |
| A brand with many outlets | A short brand name could catch titles other than the news outlet it was meant for | Rate the specific news outlet by its full name |
| Part of a word | A short name could match inside another word | Names now match as whole words only |
| Links submitted by readers | These arrive as a bare host, such as an outlet's domain, with no spaces | Hosts are matched separately, because their words run together |
While I was there I also revisited several ratings that had been set too far towards the ends of the scale. A list you review by hand gets better each time you read it. A model's guess does not.
Counting coverage fairly
Two smaller changes made the view itself more honest. Coverage is now counted by outlet rather than by article, so one outlet publishing five versions of a story does not look like five sources. And it only looks at the last 30 days, so a story is judged on how it is being covered now, not on an old flurry.
Where AI still helps, and where it does not
The site uses AI a great deal: summaries, explainers, answers to common questions. All of those are things a reader can check against the source in front of them. A political rating is different. It is a judgement about an organisation, it shapes how readers see everything else on the page, and it needs someone who will stand behind it.
My rule now is simple. If a decision is a judgement that people will rely on and might challenge, a person makes it and writes it down. AI can help gather the evidence. It does not get the final word.