AI worse than humans in every way at summarising information, government trial finds

[Introversion]

https://www.crikey.com.au/2024/09/03/ai-worse-summarising-information-humans-government-trial/

Amazon conducted the test earlier this year for Australia’s corporate regulator the Securities and Investments Commission (ASIC) using submissions made to an inquiry. The outcome of the trial was revealed in an answer to a questions on notice at the Senate select committee on adopting artificial intelligence.

The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.

Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.

These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.

…

 

[Brigid Barry]

I'm so sorry but all I can think of to say is "no shit".

AI is mediocre at best at everything it tries to do.

 

[dickson]

https://www.crikey.com.au/2024/09/03/ai-worse-summarising-information-humans-government-trial/

These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.

Dog. Bites. Man.

 

[be frank]

Preaching to the choir here, but since LLMs don't and can't understand what they're "reading," they're not capable of summarising.

They don't summarise text, they only shorten it. I wish more people understood that difference.

 

[Brigid Barry]

The ONE thing it's good at: it goes through Amazon reviews to find repeated words and condense it into a blurb. If you’re looking at pillows and everyone says "very soft" and a handful of people say it smells funny, it'll spit out "customers found this soft but some customers found it smelled funny".

Amazon has rolled out an AI assistant (has something to do with Alexa and cannot be removed/disabled in the app) that's supposed to be able to make recommendations and answer questions and I can't wait to read the article about how Rufus recommends products based on how much the seller/manufacturer pays.

 

[dickson]

The ONE thing it's good at: it goes through Amazon reviews to find repeated words and condense it into a blurb. If you’re looking at pillows and everyone says "very soft" and a handful of people say it smells funny, it'll spit out "customers found this soft but some customers found it smelled funny".

Amazon has rolled out an AI assistant (has something to do with Alexa and cannot be removed/disabled in the app) that's supposed to be able to make recommendations and answer questions and I can't wait to read the article about how Rufus recommends products based on how much the seller/manufacturer pays.

Yet another reason for me to radically curtail the amount of money I give to Jeff Bezos.