So that generative AI thing still seems pretty popular

Three years of generative AI

In this Normal Deviance column, Hugh reflects on a rapid few years of progress.

A colleague reminded me that it’d been three years since this article, written about six months into the AI boom (which I’ve arbitrarily declared to be ChatGPT’s release). I’m not sure we’ve topped the image of the Pope in a puffer jacket, but it did seem a timely opportunity to reflect on the intervening period.

How much has generative AI actually improved in three years?

First, models have gotten significantly better. The tagline ‘today’s AI is the worst you will ever use’ recognises the period of rapid progress we are in. Actually articulating how much models have improved is a pretty tricky task, with attempts ranging from the dramatic to the avian . I quite like Metr’s attempt to estimate how many hours of human work (as measured in human time) a model can do before falling over. This figure has been doubling every four months, and we’re now comfortably in the ‘few hours project’ territory, which means models can take on more complex pieces of work and get most parts right.

Figure: METR’s tracking of the complexity of tasks that LLMs can reliably undertake

Source: https://metr.org/  

What does generative AI mean for knowledge workers?

Second, for knowledge workers at least, AI does genuinely seem to be a general-purpose technology (or GPT, although a different acronym from the one hiding in ChatGPT’s name). Technologies such as computers or electricity are termed GPTs because they provide a foundation to do a variety of different tasks. For generative AI models, the proof appears to be in the pudding - people are using the technology in quite different ways, including people with nominally similar jobs. Every week I see people using it for writing code, drafting emails, proofreading docs, checking compliance against reference documents, speeding up a literature search, designing slides, building apps, transcribing calls, producing documentation, or role-playing meetings. This (and more) points to ongoing changes in our work lives, particularly for white-collar workers.

Are AI hallucinations and broader risks still a concern?

Third, we seem to be talking about hallucinations a bit less. While particularly impactful hallucinations can still attract headlines (and in the legal profession it is even worthy of a database , which includes 74 Australian cases of spurious AI references), I think this issue is now well-understood by users and the public. Facts and references need to be verified (if you’re lazy, just ask a different AI tool), and we can expect people to be intolerant of such AI-induced mistakes.

Fourth, the robots are not (quite) here yet.  While the internet did get very excited about the 200 hour livestream of robots sorting packages, humanoid robots are yet to go mainstream.

Interacting with the world and making good judgements on how to act is a tricky task, which continues to be a topic of intensive research. I’m not sure we should have started with martial arts though.

Fifth and finally, people are aware of the broader impacts of AI. There is lots of attention to the labour market impacts, although at this point it does not appear to be a jobs apocalypse. But we see impacts elsewhere too. People purchasing computer equipment have seen the painful impacts of higher memory prices, particularly RAM , as new data centres fuel demand. And the energy and water demands of new compute have rightly caught the attention of the public.

What does this all mean for us? Happily, I think the lessons from three years ago remain intact. The internet is ballooning with AI-generated content, meaning that we should rigorously guard our attention. Transparency and regulation will continue to be important as to how AI tools shape our lives. And we are high on the AI hype-cycle , so there will still be areas of our lives where the impact proves to be less than imagined.

Figure: Google AI with its trademark lack of self-awareness

It remains a time that will reward those who are curious about emerging technology and tools. But more importantly, we continue to live in a world that still needs human-led, evidence-based advice. Our role as actuaries seems pretty safe.

Hopefully safe enough that I get to write another reflection article in 2029.