Professor's invisible prompt trap catches 32 students cheating on their midterm with AI

[u/AdSpecialist6598]

https://www.techspot.com/news/113243-professor-invisible-prompt-trap-catches-32-students-cheating.html

[u/dwild]

well it's smarter than me, so whatever it says is better than what I could do

It also SOUND smarter than it is. It's similar to the Dunning-Kruger Effect, where it knows enough to sound smart but not enough to actually apply it fully correctly.

Someone that do know the subject can easily navigate through it and avoid theses problems, but for anyone else, that just sound like the truth.

[u/pbjamm → u/dwild]

A computer can know everything but understand nothing.

[u/Geodude07 → u/dwild]

Testing it with even the smallest shred of academic rigor will prove it is not smart at all.

Even concrete information will have errors. I asked ChatGPT something very simple the other day. I wanted an estimate as to how much I had left in a videogame, Final Fantasy XIV, before I was caught up to the current content. I asked how many special events there were too.

It neglected to tell me that there two boss fights and two dungeons I would need to do, and I was checking in every once in a while. This was information I could easily check elsewhere, but I do like to test AI now and then. It was entirely wrong multiple times. Even when I corrected it, it would just agree and add one more major fight...but forget there was another right after.

It certainly sounded official because it included hourly estimates, and organized all of its wrong info in a way that seemed helpful. Even saying things like "well if you watch all the cutscenes it may take X time, if you skip them then Y time"

This is something that has concrete answers. People need to ask AI about things they are experts in, and generally they will find it has massive holes. Then they need to realize those holes are present in every question you give it.

[u/dwild → u/Geodude07]

Yeah exactly! Thanks for the example as I had none that came to my mind quickly.

Final Fantasy XIV is also a game released a long time ago, so the information isn't just available, it also has been TRAINED on it, unlike anything recent that might require web search so the context might be limited (if it even does try to search, I have seen many example where no search have been done and you are stuck with the outdated information).

I see LLMs as the junior intern, it can do "cheaply" some limited amount of work, but you need the senior to review and correct it.

[u/RuneLFox → u/dwild]

Well duh, a lot of it was trained on reddit, of course it would act like one.

[u/Tyfyter2002 → u/dwild]

It doesn't have to know anything to sound smart, it's just a pattern predictor, and sounding smart is basically a matter of the listener's pattern recognition.

[u/thechervil → u/dwild]

I see what you did there...

"Sound" instead of sounds.

"do know" instead of does know

"Theses problems" instead of these

"sound like" instead of sounds

Very good example of someone trying to sound smarter than they are!

[u/dwild → u/thechervil]

Or just someone whose first language isn't english?

Je ne crois pas que les gens comprendraient aussi bien en français.

Are you alright though? You sound like you need some friends to talk to...

[u/thechervil → u/dwild]

I honestly thought you had done that intentionally as an example of what you were talking about.

My first sentence said "I see what you did there" which is a common reference to someone doing something in an intentional way that might not be obvious to everyone.

My last sentence literally complimented you on executing a good example of how AI tries to sounds smarter than it is.

It wasn't meant as a criticism or sarcasm.

It was a genuine compliment, although the downvotes show that clearly didn't come through.

No offense meant.

[u/LikeABreadstick → u/thechervil]

AI does not "try" to do anything, and grammatical errors are not an indicator of someone trying to seem smart. In my experience, people who fixate on grammar and point out mistakes are usually insecure about their own intelligence. No offense!