Let’s start with the obvious: artificial intelligence is controversial.
There are genuine concerns about its environmental impact, its use of copyrighted material, the effect it could have on jobs, and the sheer volume of AI-generated slop currently flooding social media.
There’s also no shortage of people on LinkedIn keen to tell us that AI will revolutionise everything from accountancy to dog walking.
But whether you love it, hate it, or are simply bored of hearing about it, the technology isn’t going anywhere.
Like plenty of newsrooms, we’ve gradually started using it as part of our workflow at Express.
We’ll sometimes use it to proofread longer articles, help analyse lengthy States reports, check if we’ve missed anything, or come up with a list of keywords.
Sometimes it’s useful. Sometimes it isn’t.
But what would happen if we trusted AI to write a whole, complex, article?
With Guernsey’s AI Sprint about to get under way it felt like a good opportunity to find out.
Could AI actually do a reporter’s job? And if not, how close is it?
Three chatbots are better than one
The experiment was deliberately simple.
I took all the material I’d gathered for a feature about the Guernsey AI Sprint – including my interview transcript, press releases, and background information – and asked three different chatbots – also called large language models (LLMs) – to turn it into an article.
I chose ChatGPT (the one everyone’s heard of), Microsoft Copilot (the one everyone has at work), and Google’s Gemini (the one at the top of your search engine).
I chose to use three chatbots to see if any failures – or successes – were unique to one product or inherent to the tech itself.
The experiment
- Create a transcript: The first piece of AI I used actually wasn’t any of the chatbots, it was an AI tool called Otter.ai, which turns speech into text. I fed my interview with Katie Inder, the AI Sprint’s Programme Manager, in and got a pretty accurate transcript out.
- Generate a prompt: Next, I asked one of the chatbots (Gemini) to help me write a prompt for all three. I explained how I wanted to experiment to work, and gave it lots of information about the style of Express articles.
- First draft: Then I fed the prompt it gave me into all three chatbots, along with the transcript and a couple of press releases. The three chatbots then wrote their versions of the article.
- Review competitors and rewrite: After that, I shared the ‘competitor’ articles with each chatbot, and asked them to evaluate them and then rewrite their own article, to make it even better.
- Compare articles and pick a favourite: Next, I shared all three articles with each chatbot and asked them to pick a favourite.
Interesting, two of them picked the same article. - Read the articles: Up until this point, I hadn’t even read the articles, I’d just followed the process to see what they came up with.
- Pick my favourite: I picked my favourite, which was the same one that two of the chatbots had selected.
- Ask for improvements and corrections: The article had a couple of minor mistakes (such as calling Miss Inder ‘Mrs’), and was also way way too long and covered minor things in too much detail. So I asked one of the chatbots (ChatGPT) to make some changes.
- Move to another chatbot: As an Express employee, I have a paid Copilot account, but don’t have one for the others. So eventually I hit my ChatGPT limit and had to switch to Gemini.
- Get frustrated: When I asked Gemini to make a change, it did an OK job. Then I asked for another change, and it totally rewrote the article, ignoring what I’d asked in the last prompt.
- Move again: I moved over to CoPilot because Gemini was ignoring what I asked, and reverting to previous versions of the article.
- Swear a bit: After some initial success, Copilot started doing similar things. So I swore at it. (I’m hoping when the AI terminator-style robots take over they won’t hold this against me)
- Back and forth: While the initial drafts were really quick, this part of the process started to take ages. I had to break things down bit by bit, repeat myself and so on. Even then, it still kept on making basic mistakes.
My frustration levels were somewhere between getting the wrong meal at a pub and trying to get through to an airline’s customer service team when your plane’s gone tech. - Call it a night: I decided to take a well earned rest and try again in the morning.
- Switch back to ChatGPT: By the time I’d woken up, my ChatGPT limits had reset to I switched back to it.
- Carry on: I carried on with the same process, asking it to make small changes. Maybe I was less tired, or maybe ChatGPT is better than the others, but I seemed to make a bit better progress.
- Finish and prepare to publish: Once I’d got the article to a state I was happy with, I copied and pasted it into the system we use to publish our articles.
There were a few minor formatting issues to sort, and I had to add in links, pictures, and captions, but the words had all been generate by my chatbot tag team.
My verdict
Getting from a 45 minute interview to three first drafts was remarkably quick.
I had three versions of a long article in probably less than 15 minutes.
Writing it myself would have been a few hours.
We regularly use Otter.ai for transcription anyway, but if you add in listening back to a 45 minute interview and picking out quotes – which is how we’d have done it a few years ago – and you’re probably looking at at least half a day.
The issue was that while the articles all looked superficially decent, they were way, way too long, pretty dull, and spent way too much time on tiny, uninteresting details.
There were lots of words, in a sensible order, but it felt like it’d been written by first year journalism student with no personality of nose for what makes news.
Instead of telling readers about the cool AI stuff they could learn about, or using a killer quote from Miss (not Mrs!) Inder, the article started by telling us which organisations arranged the event. Hardly Pulitzer-winning stuff.
It’s the journalistic equivalent of leading with ‘The White Star Line’s offices are in central London’ rather than ‘Titanic Sinks: Great Loss of Life’.
Promising start, but frustrating from then on
Despite the mediocre first passes, I was hopeful I could help get the article into a decent shape.
After all, I know how to write a killer news story and I understand what Guernsey’s AI sprint is.
This is where it got infuriating.
Other than a couple of basic mistakes, it often seemed to ignore what I’d ask it.
I’d say “change this bit” and it’d rewrite the whole article.
Or I’d ask it to change one thing, and it would revert to a previous version of the story.
Occasionally it’d give me a lecture on why my approach was wrong.
Not a time saver… yet
I used to be a software developer and I understand how LLMs work in reasonable detail, so I know they’re not really ‘intelligent’ in the normal sense.
They’re just really good at prediction – which can often be very useful.
I’ve used AI in the past for software development, using things like Anthropic’s Claude or Microsoft PowerPlatform, and they’re pretty useful and save you a lot of time.
For an article like this, though, it wasn’t a time saver at all. I’d have been quicker writing it myself.
It also needed serious adult supervision.
If I didn’t know the story inside out and just trusted it there would be a real possibility of publishing something factually inaccurate or even wildly misleading.
Early days
While none of the three chatbots I used are ready to write an article unsupervised yet, it’s important to remember that this is still an emerging technology.
Just a couple of years ago people were laughing at Will Smith’s face melting when he ate pizza, in an AI meme.
Now, some people have AI girlfriends, which – odd as it may seem – shows how rapidly the technology is progressing.
I’m pretty sure the three chatbots I used could handle a simple press release from the Police asking for witnesses.
But for anything longer, they’re a useful tool for research and proofreading.
Of course, there’s every possibility I need to get better at learning how to prompt AI – especially when it comes to writing stories. (Luckily, the AI sprint has a session on just that)
But I think it’ll be a few years before AI is good enough to replace an experienced reporter entirely.
Of course, that brings us back to the ethical question.
To quote Jeff Goldblum in Jurassic Park: “Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.”
