Bad uses / over-promised of AI

Forums › Artificial Intelligence (AI, LLM, ML) › Bad uses / over-promised of AI

1 | ... | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11

12007 posts

Uber Geek
+1 received by user: 12926

Subscriber

#3453824 17-Jan-2026 17:24

Cybersecurity researchers have disclosed details of a new attack method dubbed Reprompt that could allow bad actors to exfiltrate sensitive data from artificial intelligence (AI) chatbots like Microsoft Copilot in a single click, while bypassing enterprise security controls entirely.

"Only a single click on a legitimate Microsoft link is required to compromise victims," Varonis security researcher Dolev Taler said in a report published Wednesday. "No plugins, no user interaction with Copilot."

"The attacker maintains control even when the Copilot chat is closed, allowing the victim's session to be silently exfiltrated with no interaction beyond that first click."

Researchers Reveal Reprompt Attack Allowing Single-Click Data Exfiltration From Microsoft Copilot

Microslop Copilot got hacked

SomeOrdinaryGamers

Dunnersfella

4100 posts

Uber Geek
+1 received by user: 887

#3453826 17-Jan-2026 17:31

jonherries:

No real surprise here, it is like you are asking a BA-English honors student to do high school math. The answers are often not going to be pretty or add up (the hint is in the name: Large Language model).

They can be quite good at helping with suggesting excel formulae, but regular basic math isn’t a statistics exercise (unless you are a Bayesian ;)

I suppose the BA(Hons) student would know to spell 'honours' with a 'u' if they were New Zealand based...
Or at least be pedantic enough to double-check it?

jonherries

1434 posts

Uber Geek
+1 received by user: 317

Trusted

Subscriber

#3453830 17-Jan-2026 18:06

Dunnersfella:

jonherries:

No real surprise here, it is like you are asking a BA-English honors student to do high school math. The answers are often not going to be pretty or add up (the hint is in the name: Large Language model).

They can be quite good at helping with suggesting excel formulae, but regular basic math isn’t a statistics exercise (unless you are a Bayesian ;)

I suppose the BA(Hons) student would know to spell 'honours' with a 'u' if they were New Zealand based...
Or at least be pedantic enough to double-check it?

The use of licensed models in an enterprise context allows you to deal with these issues - adding a statement like “Your users are in New Zealand and your spelling should be consistent with British English.” tends to work well.

We have also had some pretty good success with Te Reo spelling - but not macrons.

Jon

Website: herri.es

Linkedin: jonherries

MadEngineer

4597 posts

Uber Geek
+1 received by user: 2577

Trusted

#3453832 17-Jan-2026 18:20

Dynamic:

Has anyone else found Copilot App Skills in Excel to be unreliable? This is the first time I'm really giving Copilot App Skills a try. My M365 account is licensed for Copilot for 365.

I'm opening a 250 line supplier billing CSV file and had Copilot for 365 help me generate a prompt to transform the file into something that I can upload to our quoting software so we have up to date pricing for our Sales Admin to use in quotes. Across a dozen tests, Copilot actually only completed the request half the time. The other half it only did part of the job and then errored and stopped.

I tried the same prompt with ChatGPT Pro, uploading the source file at the same time. Got a good result in less than half the Copilot processing time with no issues.

Sounds like it is struggling with the amount of data. does it work if you use an invoice with less lines?

Also, see if the supplier can provide CSV files preformatted for you. It's a feature that we offer our customers.

You're not on Atlantis anymore, Duncan Idaho.

kingdragonfly

12007 posts

Uber Geek
+1 received by user: 12926

Subscriber

#3453845 17-Jan-2026 20:23

Microsoft launches Copilot AI function in Excel, but warns not to use it in 'any task requiring accuracy or reproducibility': August 2025

Certain Windows 365 Copilot users now have access to a new feature in Excel that lets you use Microsoft's Copilot AI to generate formulas, but you might want to be careful how you use it. Microsoft is already warning users that the AI might not always be accurate.

The new "COPILOT" function allows you to skip writing Excel formulas yourself by telling Copilot what you want to do and the cells you want to use. For instance, you could type in "=COPILOT("Summarize this feedback", A2:A20)" to have Copilot generate a formula that summarizes the content in a column of cells. The examples Microsoft suggests in the support page for the COPILOT function focus on classifying, summarizing, and generating content.

However, Microsoft specifically warns not to use it for "any task requiring accuracy or reproducibility," like numerical calculations. Microsoft also advises against using the feature for "financial reporting, legal documents, or other high-stakes scenarios," so basically most of the things people turn to Excel for.
...

kingdragonfly

12007 posts

Uber Geek
+1 received by user: 12926

Subscriber

#3453846 17-Jan-2026 20:24

For a deep read, from UC Berkeley

Why Hallucinations Matter: Misinformation, Brand Safety and Cybersecurity in the Age of Generative AI: UC Berkeley

We may now be finding that the AI "ghost in the machine" that we all should fear is not sentience, but simple hallucination. As Sophocles almost said, “Whom the gods would destroy, they first make hallucinate”.

In the present day, Mark Twain’s (or Benjamin Disraeli’s?) supposed quote might better be recast as, “There are three kinds of lies: lies, damned lies, and hallucinations”.

In our age of generative AI, the technology’s propensity to create false, unrelated, “hallucinated” content may be its greatest weakness.

Major brands have repeatedly fallen victim to hallucination or adversarial prompting, resulting in both lost brand value and lost company value.

Notable examples include the chatbot for the delivery firm, DPD, aspersing the company; Air Canada having been found financially liable by the Canadian courts for real-time statements made by its chatbot; inappropriate image generation issues at Midjourney and Microsoft; and of course, Google losing $100 billion in market value in a single day following a factual error made by its Bard chatbot.

In each instance, brand value that was carefully accreted in the age of static content did not prove resilient to hallucination from the age of generative AI content.
...

Dell

Shop now for Dell laptops and other devices (affiliate link).

kingdragonfly

12007 posts

Uber Geek
+1 received by user: 12926

Subscriber

#3453851 17-Jan-2026 20:46

From the devil itself, Microsoft

Correction capability helps revise ungrounded content and hallucinations

Today, we are excited to announce a preview of "correction," a new capability within Azure AI Content Safety's groundedness detection feature. With this enhancement, groundedness detection not only identifies inaccuracies in AI outputs but also corrects them, fostering greater trust in generative AI technologies.

What is Groundedness Detection?

Groundedness detection is a feature that identifies ungrounded or hallucinated content in AI outputs, helping developers enhance generative AI applications by pinpointing responses that lack a foundation in connected data sources.

Since we introduced groundedness detection in March of this year, our customers have asked us: “What else can we do with this information once it’s detected besides blocking?” This highlights a significant challenge in the rapidly evolving generative AI landscape, where traditional content filters often fall short in addressing the unique risks posed by Generative AI hallucinations.
...

“Quis custodiet ipsos custodes?”
Commonly translated as who watches the watchers?”

Tinkerisk

4814 posts

Uber Geek
+1 received by user: 3684

#3453871 18-Jan-2026 05:10

I want my DOS 3.31 back! 😁

Qui nihil scit, omnia credere debet. - He who knows nothing must believe everything.
Firewalls do NOT stop dragons!
I avoid Big Tech, they try hard to dictate technology and culture across borders.
In effect we have everything to hide from someone, and no idea who someone is.

cddt

1981 posts

Uber Geek
+1 received by user: 1927

#3454008 18-Jan-2026 19:58

kingdragonfly:

The new "COPILOT" function allows you to skip writing Excel formulas yourself by telling Copilot what you want to do and the cells you want to use. For instance, you could type in "=COPILOT("Summarize this feedback", A2:A20)" to have Copilot generate a formula that summarizes the content in a column of cells. The examples Microsoft suggests in the support page for the COPILOT function focus on classifying, summarizing, and generating content.

However, Microsoft specifically warns not to use it for "any task requiring accuracy or reproducibility," like numerical calculations. Microsoft also advises against using the feature for "financial reporting, legal documents, or other high-stakes scenarios," so basically most of the things people turn to Excel for.
...

Thought I'd try this the other day, after having made an additional payment on my mortgage. I entered the details of my different tranches in an Excel document, and asked the inbuilt "Copilot" to tell me how long it would be to fully pay off my mortgage. It thought for a minute, wrote some python code, and told me I will finish paying it off in 15.4 months. Unfortunately, the bank disagrees and thinks I should be paying for another 13 years and 9 months.

ezbee

2662 posts

Uber Geek
+1 received by user: 3110

#3454038 19-Jan-2026 09:52

kingdragonfly

An Ai needs a Ai groundedness detector to detect hallucinations and such.

I see where this is going.

Ai needs an Ai groundedness detector which is Ai that needs a groundedness detector, oh wait.

So its groundedness detectors all the way down,

Luckily, we have an oversupply of GPU's RAM, and Gigawatts to burn all these tokens. :-)

kingdragonfly

12007 posts

Uber Geek
+1 received by user: 12926

Subscriber

#3454582 20-Jan-2026 11:44

Americans given "non-disclosure agreements" NDAs from AI data center company

NBC News

Mighty Ape

Shop now at Mighty Ape (affiliate link).

kingdragonfly

12007 posts

Uber Geek
+1 received by user: 12926

Subscriber

#3456386 27-Jan-2026 12:18

Creator starts on a long tangent about game piracy, then get into AI companies like actively encouraging "shadow libraries", basically illegal piracy sites, by giving them free high-speed access.

A number of "shadow libraries" were dying, until AI companies started pirating copyrighted materials on a massive scale. I don't think "massive" truly describes the scale.

Court filings allege Nvidia accessed pirated datasets from Anna’s Archive, with leadership approval. Estimated 500 terabytes of copyrighted books were used.

It's Only Illegal For Me and You...

SomeOrdinaryGamers

gzt

18756 posts

Uber Geek
+1 received by user: 7883

Lifetime subscriber

#3456851 28-Jan-2026 23:41

AI generated pictures of Mt Maunganui disaster spreading online:

https://www.nzherald.co.nz/nz/mount-maunganui-slip-officials-warn-of-fake-ai-disaster-images-online/D4RB35B4NBGHHCVROG66SAQ7QA/

Sometimes I wonder if some of these people even know they are using an image generator.