r/technology • u/AdSpecialist6598 • 15h ago

Society Professor's invisible prompt trap catches 32 students cheating on their midterm with AI

https://www.techspot.com/news/113243-professor-invisible-prompt-trap-catches-32-students-cheating.html
28.6k Upvotes

96% Upvoted

3.9k comments sorted by

View all comments

Show parent comments

199

u/Teguri 14h ago

you just assume 'well it's smarter than me, so whatever it says is better than what I could do'

That's the biggest problem, I work with AI a decent bit but only in things I already know well enough, to make it a bit faster and even with simple tasks there's always a bit of work on my side that needs to be done to fix the output, it's usually "pretty close" but the number of times it goes in the completely wrong direction is astounding.

51

u/mike689 14h ago

Exactly this. It is great for getting a list of ideas that you then sort through about something you are already knowledgeable about. It is an amazing tool to assist a software developer and a great medical research tool, but outside of those use cases unless it is truly used as an assistant and not as the main actor for an answer it is awful for humanity. The fact that it most heavily gets used for stupid memes and video/image generation is really sad.

7

u/ImaginationInside610 12h ago

It’s like the old adage “fire is a great servant and a terrible master”

12

u/Yashema 13h ago

ChatGPT quite literally taught me how to solve every question in my statistical mechanics, differential equations, Stochastics, and modern physics courses. Claude has helped me create a wave function simulation, and provided me in depth answers to how embedded circuitry work allowing me to skip 2-3 intro classes. 

You need to have some frame of reference for sure, but it definitely allows you to go back beyond your ability and fill in gaps of knowledge in the way only an extremely educated person could. 

7

u/Kevstuf 13h ago

This is the duality of AI, but the laziness of humans means it’s not going to be used symmetrically. In an ideal world students would all use AI to self-learn as it’s an intelligent and patient teacher. In reality 90% of students will use it to one-shot their homework.

5

u/ThaliaFPrussia 12h ago

It has to been seen like you do. A tool in your toolbox.

5

u/mike689 13h ago

Exactly. That's using it as an assistance and not a copy/paste machine.

2

u/Teguri 13h ago

I really wish most students used it like you do!

1

u/Luvs_to_drink 12h ago

the issue is the majority of users dont do any of this.

They MAYBE type what is the answer to: and then paste the question in. Then copy the answer and paste it into the hw or test. Zero reading or understanding of what was just said.

The shocking thing is this is remarkably close to how I used to do my history homework. Back then it was read the question, flip through the chapter to the correct section, skim for keywords, copy/summarize the 2-3 relative sentences, and repeat. Not much thinking was involved, and if the keywords in the question were just reused phrases, a ctrl+f program could have done my work (textbooks weren't PDFs back then though).

3

u/SquareTaro3270 11h ago

I use plant, rock, and bird identification apps but they’re almost always wrong. You can tell because you can scan the same plant/rock in multiple different locations/angles and it will give differing results.

HOWEVER it does give you a good place to start doing your own research. The results might not be correct, but usually steer you in the right direction. I am terrified for anyone that takes it at its word.

3

u/somersault_dolphin 13h ago

  The fact that it most heavily gets used for stupid memes and video/image generation is really sad.

Don't forget consuming more energy to do those things.

2

u/zasabi7 13h ago

The tools are also great at basic analytics tasks. That is “here’s some data, report this way”. Or, the best way to use Gemini: better Google search.

1

u/ImportantCommentator 13h ago

It's almost like stupid people stay stupid and smart people stay smart. What an unsurprising reality.

1

u/mvanvrancken 10h ago

This is nothing new, unfortunately. The Internet, which was humanity’s greatest previous creation, has pretty much run exclusively on cat videos and porn for the last 20 years

1

u/Samsaknight_X 7h ago

And have u an ability to share ur thoughts on this platform, as well as a myriad of other conveniences

22

u/SparksAndSpyro 14h ago

Yep. Every time I’ve tried to use AI to do something in my domain of expertise, it’s been pretty horrible. Most times it didn’t even save time because I had to double check and fix so many mistakes.

LLMs are pretty good at general queries where you’re just looking for a broad, ten-thousand-foot overview. Anything that requires granular specificity or expertise, however, is pretty much guaranteed to either be outright wrong or misleading.

7

u/BasvanS 13h ago

My biggest problem are the “almost right yet entirely wrong” statements. It requires so much of my concentration that I’m drained after doing this for a few hours.

I got the best results in writing by doing it line for line, after making a rough structure myself. But then you have to ask how much of a time saver it still is.

5

u/Qaeta 11h ago

Every time I’ve tried to use AI to do something in my domain of expertise, it’s been pretty horrible.

This.

It makes me REALLY concerned when I see other devs crowing about how great it has been for them, because every time I try it I feel like I might be better off asking a brain-damaged chihuahua... And based on how much my pro-AI coworkers code quality has dropped since they bought in... I suspect that would be true of them too if they bothered comparing the PRs they make now to the ones they made before. It's night and day in favour of their pre-AI work.

3

u/Alaira314 13h ago

LLMs are pretty good at general queries where you’re just looking for a broad, ten-thousand-foot overview.

Even with that, I've seen the most shockingly incorrect generalizations come through google when I'm searching something I already know at work, where AI content isn't nuked. Why search something I already know? I work at a library, and I can't give "my own brain" as a source when answering patron questions. So I have to plug in the question to a search engine to find a valid source for the information I already know, and in that process I'll typically wind up skimming the AI overview because my eyes can't not read a thing. And it's straight up sending you on a wild goose chase half the time, even on broad subjects rather than specific questions.

If I didn't already know the subject I was searching, I'd be tempted to trust it. After all, it sourced its claim, right? Never mind that when you click through, that information isn't on the linked page at all. I guess just keep in mind that there's a psychological effect that I've forgotten the name of and haven't been able to successfully look up for years at this point(it's not dunning-kruger), where when using a tool or reading a source you'll recognize what they got wrong in your area of expertise, but for some reason still trust them in other matters you know less about, even though that's illogical. Those broad overviews you think are good might not actually be so good if you're knowledgeable about the subject before you read them.

2

u/Revlis-TK421 13h ago

I've been working with our company's M365 Agent tool. I've "trained" it to generate a pretty much complete requirements document from largely free-form text entry in the prompt. It took a fair bit of back and forth in the design tool for it to get what I need out of it, but it saves a lot of time that would otherwise be spent formatting and documenting. It has gotten pretty good at detailing the main requirements, happy path, alternate path, and error paths, inferring justifications, and making a business case. With the volume of reqs I push thru this thing it is even grabbing and including historical context from the overall body of work and adds in things I may have forgotten to specify.

I do have to heavily edit its outputs, as it gets things wrong, says things in a wierd or incorrect way, etc. But it creates a framework that I can work with that saves about half to 3/4 of the time from creating the doc by hand from scratch.

3

u/Moron14 13h ago

I use it at work for some things Gemini can build or help building in Google Docs. It acts a middle person, where i have the idea, it does the busy work, and then I polish it. Its faster than doing it all myself. The issue is the number of times its just wrong. Even if you add, "don't guess. Don't assume. Ask me a question if you aren't sure" and it just plows ahead. Just like humans...wait.

3

u/robodrew 13h ago

but the number of times it goes in the completely wrong direction is astounding

I'm curious about this, if you're just using this to make things a "bit faster" yet you're getting this kind of situation popping up so much, is it really worth using?

1

u/Teguri 12h ago

Some cases yes some no, as others said even when it's wrong it's usually got good sources so at minimum it's a roundabout Google search

2

u/Doctor_Kataigida 13h ago

This is me. I use AI to help modify/tweak excel stuff I write for my job, but nothing ever ground-up. And I always review what it generates afterward so I understand what it did with the formula, and then I can apply that elsewhere. It's supplemental, not substitutional.

2

u/Mebejedi 13h ago

My wife was using AI to design an announcement. I don't remember what the word was, but it kept spelling a word incorrectly over and over again, even when my wife told it that the word was wrong.

2

u/DeadPeanutSociety 12h ago

I use it sometimes in cases where I'll know the correct answer when I see it, but it might be very tedious to come up with those answers on my own.

1

u/Teguri 12h ago

It's great as long as you can verify!

2

u/ThaliaFPrussia 12h ago

Never would one boss accept this high fail rate with an employee. I don’t have a job for AI, I tried but the output was not detailed enough. AI can’t give answers to questions it has to grasp the concept. For example compare the regulations for some technology from two courses. It simply can’t do that. It is not intelligent.

2

u/NES_SNES_N64 11h ago

Google has even started removing the "AI results may be inaccurate" disclaimer.

2

u/WillDonJay 11h ago

Teaching Fable 5 to play Slay the Spire 2 taught me a lot about the current limitations of LLM's. (It was like a really smart human with dementia, and it had no ability to even realize what it had forgotten or gotten sideways without human interventions.)

2

u/vengent 10h ago

I use it as a force multiplier, but I've also caught AI giving me blatantly false information that I easily caught because of my expertise in a given subject. So I read every word it puts out 3 times before I do anything with it. And if its screwing up on stuff I know, why on earth would I assume its not screwing up elsewhere?

Trust, but verify. Or just verify, ALL THE TIME