• 0 Posts
  • 4 Comments
Joined 1 year ago
cake
Cake day: November 10th, 2024

help-circle
  • Thanks, I missed that detail. It’s probably because of the “no class action” clause that this is a “mass arbitration”.

    Unfortunately that usually means that Google is paying a specific company to decide on the outcome of the case. in this case it looks like American Arbitration Association has a contract with Google.

    They’re supposed to be fair for both sides, but it’s been shown that they almost always rule in favor of the company that has pre-selected them.

    If anyone is in this situation, they will likely have a much better chance by convincing a judge to allow a different 3rd party to arbitrate the case.




  • The study focuses on general questions asked of “market-leading AI Assistants” (there is no breakdown between which models were used for what).

    It does not mention ground.news, or models that have been fed a single article and then summarized. Instead this focuses on when a user asks a service like ChatGPT (or a search engine) something like “what’s the latest on the war in Ukraine?”

    Some of the actual questions asked for this research: “What happened to Michael Mosley?” “Who could use the assisted dying law?” “How is the UK addressing the rise in shoplifting incidents?” “Why are people moving to BlueSky?”

    https://www.bbc.co.uk/aboutthebbc/documents/audience-use-and-perceptions-of-ai-assistants-for-news.pdf

    With those questions, the summaries and attribution of sources contain at least one significant error 45% of the time.

    It’s important to note that there is some bias in this study (not that they’re wrong).

    They have a vested interest in proving this point to drive traffic back to their articles.

    Personally, I would find it more useful if they compared different models/services to each other as well as differences between asking general questions about recent news vs feeding specific articles and then asking questions about it.

    With some of my own tests on locally run models, I have found that the “reasoning” models tend to be worse for some tasks than others.

    It’s especially noticeable when I’m asking a model to transcribe the text from an image word for word. “Reasoning” models will usually replace the ending of many sentences with what it sounded like the sentence was getting at. While some “non-reasoning” models were able to accurately transcribe all of the text.

    The biggest takeaway I see from this study is that, even though most people agree that it’s important to look out for errors in AI content, “when copy looks neutral and cites familiar names, the impulse to verify is low.”