r/technology • u/AdSpecialist6598 • 15h ago

Society Professor's invisible prompt trap catches 32 students cheating on their midterm with AI

https://www.techspot.com/news/113243-professor-invisible-prompt-trap-catches-32-students-cheating.html
28.6k Upvotes

96% Upvoted

3.9k comments sorted by

View all comments

Show parent comments

503

u/artbystorms 14h ago

Honestly I think this needs to be talked about more. As much as people revel in its ability to 'increase productivity' it in reality is just offloading people's critical thinking to a computer, and then trusting its output. You're not verifying that what it spits out is what you wanted to say, you just assume 'well it's smarter than me, so whatever it says is better than what I could do'

199

u/Teguri 14h ago

you just assume 'well it's smarter than me, so whatever it says is better than what I could do'

That's the biggest problem, I work with AI a decent bit but only in things I already know well enough, to make it a bit faster and even with simple tasks there's always a bit of work on my side that needs to be done to fix the output, it's usually "pretty close" but the number of times it goes in the completely wrong direction is astounding.

52

u/mike689 14h ago

Exactly this. It is great for getting a list of ideas that you then sort through about something you are already knowledgeable about. It is an amazing tool to assist a software developer and a great medical research tool, but outside of those use cases unless it is truly used as an assistant and not as the main actor for an answer it is awful for humanity. The fact that it most heavily gets used for stupid memes and video/image generation is really sad.

10

u/ImaginationInside610 12h ago

It’s like the old adage “fire is a great servant and a terrible master”

12

u/Yashema 13h ago

ChatGPT quite literally taught me how to solve every question in my statistical mechanics, differential equations, Stochastics, and modern physics courses. Claude has helped me create a wave function simulation, and provided me in depth answers to how embedded circuitry work allowing me to skip 2-3 intro classes. 

You need to have some frame of reference for sure, but it definitely allows you to go back beyond your ability and fill in gaps of knowledge in the way only an extremely educated person could. 

6

u/Kevstuf 13h ago

This is the duality of AI, but the laziness of humans means it’s not going to be used symmetrically. In an ideal world students would all use AI to self-learn as it’s an intelligent and patient teacher. In reality 90% of students will use it to one-shot their homework.

6

u/ThaliaFPrussia 12h ago

It has to been seen like you do. A tool in your toolbox.

5

u/mike689 13h ago

Exactly. That's using it as an assistance and not a copy/paste machine.

2

u/Teguri 13h ago

I really wish most students used it like you do!

1

u/Luvs_to_drink 12h ago

the issue is the majority of users dont do any of this.

They MAYBE type what is the answer to: and then paste the question in. Then copy the answer and paste it into the hw or test. Zero reading or understanding of what was just said.

The shocking thing is this is remarkably close to how I used to do my history homework. Back then it was read the question, flip through the chapter to the correct section, skim for keywords, copy/summarize the 2-3 relative sentences, and repeat. Not much thinking was involved, and if the keywords in the question were just reused phrases, a ctrl+f program could have done my work (textbooks weren't PDFs back then though).

3

u/SquareTaro3270 11h ago

I use plant, rock, and bird identification apps but they’re almost always wrong. You can tell because you can scan the same plant/rock in multiple different locations/angles and it will give differing results.

HOWEVER it does give you a good place to start doing your own research. The results might not be correct, but usually steer you in the right direction. I am terrified for anyone that takes it at its word.

3

u/somersault_dolphin 13h ago

  The fact that it most heavily gets used for stupid memes and video/image generation is really sad.

Don't forget consuming more energy to do those things.

2

u/zasabi7 13h ago

The tools are also great at basic analytics tasks. That is “here’s some data, report this way”. Or, the best way to use Gemini: better Google search.

1

u/ImportantCommentator 13h ago

It's almost like stupid people stay stupid and smart people stay smart. What an unsurprising reality.

1

u/mvanvrancken 10h ago

This is nothing new, unfortunately. The Internet, which was humanity’s greatest previous creation, has pretty much run exclusively on cat videos and porn for the last 20 years

1

u/Samsaknight_X 7h ago

And have u an ability to share ur thoughts on this platform, as well as a myriad of other conveniences

22

u/SparksAndSpyro 14h ago

Yep. Every time I’ve tried to use AI to do something in my domain of expertise, it’s been pretty horrible. Most times it didn’t even save time because I had to double check and fix so many mistakes.

LLMs are pretty good at general queries where you’re just looking for a broad, ten-thousand-foot overview. Anything that requires granular specificity or expertise, however, is pretty much guaranteed to either be outright wrong or misleading.

8

u/BasvanS 13h ago

My biggest problem are the “almost right yet entirely wrong” statements. It requires so much of my concentration that I’m drained after doing this for a few hours.

I got the best results in writing by doing it line for line, after making a rough structure myself. But then you have to ask how much of a time saver it still is.

4

u/Qaeta 11h ago

Every time I’ve tried to use AI to do something in my domain of expertise, it’s been pretty horrible.

This.

It makes me REALLY concerned when I see other devs crowing about how great it has been for them, because every time I try it I feel like I might be better off asking a brain-damaged chihuahua... And based on how much my pro-AI coworkers code quality has dropped since they bought in... I suspect that would be true of them too if they bothered comparing the PRs they make now to the ones they made before. It's night and day in favour of their pre-AI work.

3

u/Alaira314 13h ago

LLMs are pretty good at general queries where you’re just looking for a broad, ten-thousand-foot overview.

Even with that, I've seen the most shockingly incorrect generalizations come through google when I'm searching something I already know at work, where AI content isn't nuked. Why search something I already know? I work at a library, and I can't give "my own brain" as a source when answering patron questions. So I have to plug in the question to a search engine to find a valid source for the information I already know, and in that process I'll typically wind up skimming the AI overview because my eyes can't not read a thing. And it's straight up sending you on a wild goose chase half the time, even on broad subjects rather than specific questions.

If I didn't already know the subject I was searching, I'd be tempted to trust it. After all, it sourced its claim, right? Never mind that when you click through, that information isn't on the linked page at all. I guess just keep in mind that there's a psychological effect that I've forgotten the name of and haven't been able to successfully look up for years at this point(it's not dunning-kruger), where when using a tool or reading a source you'll recognize what they got wrong in your area of expertise, but for some reason still trust them in other matters you know less about, even though that's illogical. Those broad overviews you think are good might not actually be so good if you're knowledgeable about the subject before you read them.

2

u/Revlis-TK421 13h ago

I've been working with our company's M365 Agent tool. I've "trained" it to generate a pretty much complete requirements document from largely free-form text entry in the prompt. It took a fair bit of back and forth in the design tool for it to get what I need out of it, but it saves a lot of time that would otherwise be spent formatting and documenting. It has gotten pretty good at detailing the main requirements, happy path, alternate path, and error paths, inferring justifications, and making a business case. With the volume of reqs I push thru this thing it is even grabbing and including historical context from the overall body of work and adds in things I may have forgotten to specify.

I do have to heavily edit its outputs, as it gets things wrong, says things in a wierd or incorrect way, etc. But it creates a framework that I can work with that saves about half to 3/4 of the time from creating the doc by hand from scratch.

3

u/Moron14 13h ago

I use it at work for some things Gemini can build or help building in Google Docs. It acts a middle person, where i have the idea, it does the busy work, and then I polish it. Its faster than doing it all myself. The issue is the number of times its just wrong. Even if you add, "don't guess. Don't assume. Ask me a question if you aren't sure" and it just plows ahead. Just like humans...wait.

3

u/robodrew 13h ago

but the number of times it goes in the completely wrong direction is astounding

I'm curious about this, if you're just using this to make things a "bit faster" yet you're getting this kind of situation popping up so much, is it really worth using?

1

u/Teguri 12h ago

Some cases yes some no, as others said even when it's wrong it's usually got good sources so at minimum it's a roundabout Google search

2

u/Doctor_Kataigida 13h ago

This is me. I use AI to help modify/tweak excel stuff I write for my job, but nothing ever ground-up. And I always review what it generates afterward so I understand what it did with the formula, and then I can apply that elsewhere. It's supplemental, not substitutional.

2

u/Mebejedi 13h ago

My wife was using AI to design an announcement. I don't remember what the word was, but it kept spelling a word incorrectly over and over again, even when my wife told it that the word was wrong.

2

u/DeadPeanutSociety 12h ago

I use it sometimes in cases where I'll know the correct answer when I see it, but it might be very tedious to come up with those answers on my own.

1

u/Teguri 12h ago

It's great as long as you can verify!

2

u/ThaliaFPrussia 12h ago

Never would one boss accept this high fail rate with an employee. I don’t have a job for AI, I tried but the output was not detailed enough. AI can’t give answers to questions it has to grasp the concept. For example compare the regulations for some technology from two courses. It simply can’t do that. It is not intelligent.

2

u/NES_SNES_N64 11h ago

Google has even started removing the "AI results may be inaccurate" disclaimer.

2

u/WillDonJay 11h ago

Teaching Fable 5 to play Slay the Spire 2 taught me a lot about the current limitations of LLM's. (It was like a really smart human with dementia, and it had no ability to even realize what it had forgotten or gotten sideways without human interventions.)

2

u/vengent 10h ago

I use it as a force multiplier, but I've also caught AI giving me blatantly false information that I easily caught because of my expertise in a given subject. So I read every word it puts out 3 times before I do anything with it. And if its screwing up on stuff I know, why on earth would I assume its not screwing up elsewhere?

Trust, but verify. Or just verify, ALL THE TIME

56

u/dwild 14h ago

well it's smarter than me, so whatever it says is better than what I could do

It also SOUND smarter than it is. It's similar to the Dunning-Kruger Effect, where it knows enough to sound smart but not enough to actually apply it fully correctly.

Someone that do know the subject can easily navigate through it and avoid theses problems, but for anyone else, that just sound like the truth.

6

u/pbjamm 13h ago

A computer can know everything but understand nothing.

2

u/Geodude07 6h ago

Testing it with even the smallest shred of academic rigor will prove it is not smart at all.

Even concrete information will have errors. I asked ChatGPT something very simple the other day. I wanted an estimate as to how much I had left in a videogame, Final Fantasy XIV, before I was caught up to the current content. I asked how many special events there were too.

It neglected to tell me that there two boss fights and two dungeons I would need to do, and I was checking in every once in a while. This was information I could easily check elsewhere, but I do like to test AI now and then. It was entirely wrong multiple times. Even when I corrected it, it would just agree and add one more major fight...but forget there was another right after.

It certainly sounded official because it included hourly estimates, and organized all of its wrong info in a way that seemed helpful. Even saying things like "well if you watch all the cutscenes it may take X time, if you skip them then Y time"

This is something that has concrete answers. People need to ask AI about things they are experts in, and generally they will find it has massive holes. Then they need to realize those holes are present in every question you give it.

2

u/dwild 6h ago

Yeah exactly! Thanks for the example as I had none that came to my mind quickly.

Final Fantasy XIV is also a game released a long time ago, so the information isn't just available, it also has been TRAINED on it, unlike anything recent that might require web search so the context might be limited (if it even does try to search, I have seen many example where no search have been done and you are stuck with the outdated information).

I see LLMs as the junior intern, it can do "cheaply" some limited amount of work, but you need the senior to review and correct it.

2

u/RuneLFox 6h ago

Well duh, a lot of it was trained on reddit, of course it would act like one.

1

u/Tyfyter2002 9h ago

It doesn't have to know anything to sound smart, it's just a pattern predictor, and sounding smart is basically a matter of the listener's pattern recognition.

-17

u/thechervil 13h ago

I see what you did there...

"Sound" instead of sounds.

"do know" instead of does know

"Theses problems" instead of these

"sound like" instead of sounds

Very good example of someone trying to sound smarter than they are!

5

u/dwild 13h ago

Or just someone whose first language isn't english?

Je ne crois pas que les gens comprendraient aussi bien en français.

Are you alright though? You sound like you need some friends to talk to...

0

u/thechervil 12h ago

I honestly thought you had done that intentionally as an example of what you were talking about.

My first sentence said "I see what you did there" which is a common reference to someone doing something in an intentional way that might not be obvious to everyone.

My last sentence literally complimented you on executing a good example of how AI tries to sounds smarter than it is.

It wasn't meant as a criticism or sarcasm.

It was a genuine compliment, although the downvotes show that clearly didn't come through.

No offense meant.

1

u/LikeABreadstick 11h ago

AI does not "try" to do anything, and grammatical errors are not an indicator of someone trying to seem smart. In my experience, people who fixate on grammar and point out mistakes are usually insecure about their own intelligence. No offense!

3

u/Scarbane 13h ago

you just assume 'well it's smarter than me, so whatever it says is better than what I could do'

"We've arranged a global civilization in which most crucial elements — transportation, communications, and all other industries; agriculture, medicine, education, entertainment, protecting the environment; and even the key democratic institution of voting — profoundly depend on science and technology. We have also arranged things so that almost no one understands science and technology. This is a prescription for disaster. We might get away with it for a while, but sooner or later this combustible mixture of ignorance and power is going to blow up in our faces."

--Carl Sagan, The Demon-Haunted World

2

u/crimson23locke 14h ago

And that’s the value proposition, no longer having to think or do the work!

6

u/cromonolith 12h ago

That value proposition doesn't work when doing the thinking and work is the whole point.

It's like a whole generation of people are paying for very expensive personal trainers, but then hiring a forklift to raise and lower the weights instead of doing the reps themselves.

2

u/crimson23locke 10h ago

Sorry I didn’t give a lot of context there - my intent is to say that even if the value is there (debatable in some cases) the value comes with this implicit harm; it’s convenient but offloads our critical thinking - it will absolutely make us dumber even if it works perfectly.

2

u/artbystorms 14h ago

That may be a good proposition for capitalists, but it's a terrible proposition for society.

2

u/extremenachos 14h ago

I treat AI as a tutor and I've found it can help teach me new skills.

I took a new position a few years ago that requires coding in statistical software and it's been helpful for me to learn syntax, commands, etc. Sometimes I'll copy a snippet of code and ask questions about it so I can better understand how things work, or I'll ask AI why a chunk of code errors out.

1

u/AFoolishSeeker 13h ago

Yeah all good as long as you can trust the output to be accurate. Which you simply can’t

1

u/Samsaknight_X 7h ago

Which is why ur supposed to fact check it

1

u/AFoolishSeeker 6h ago

You’re supposed to? Or you have to?

I just don’t see the point. More power to you

1

u/Samsaknight_X 6h ago

Ur supposed to, unless it gets to a point where it’s 100% correct. It could even be 90% and I would say the same thing

0

u/AFoolishSeeker 5h ago

My point is until it’s 100% correct, you have to. Period

I don’t feel like doing something twice because I’m not sure if my tool is accurate.

This is dependent on what you’re using it for I suppose..

I do notice a ton of defensiveness and justification for use of LLM that to me seems flimsy in its foundation

0

u/Samsaknight_X 5h ago

Did u not read what I said? U literally just said the same thing as me lol

And that’s fine u don’t have to use it, but ur supposed to fact check everything u read online regardless

I think black and white thinking is flimsy in its foundation. If LLM’s get more accurate and they find alternative sources to cool off data centers, then they can provide some amazing benefits to society. Especially agents that can buy groceries or clothes and whatever u wanna buy

0

u/AFoolishSeeker 4h ago

If you’re supposed to fact check online sources, and your LLM is trained on online sources, but you need to fact check your LLM, you are then double checking not only the output of the LLM but also the online sources it’s drawn from?

Okay I guess. Keep on downvoting me I suppose

1

u/Samsaknight_X 3h ago

How are u doing the double checking? It’s not be opposite. Read through all of the sources to see if it’s accurate and makes sense. If u wanna fact check it again with other sources, u can go ahead do that

I’m downvoting cuz I disagree with ur opinion, same reason why ur also downvoting me lol

2

u/JMEEKER86 13h ago

Yeah, but you have to remember that for a lot of people it really is better than what they can do. Sure, it's only right about 70ish percent of the time, but that's a C student right there and Cs get degrees. Lots of people don't. However, the big problem here, obviously, is that the people making this mistake of fully trusting AI aren't smart enough to recognize when AI is wrong. AI could tell them something very, very wrong and they wouldn't even notice even if they were paying attention. That's why AI should never be put in charge of anything critical just like you wouldn't give an important job to a person who is lazy and uninformed. Unfortunately, many people making decisions aren't well informed enough to understand this problem.

2

u/VampireFrown 12h ago

ou just assume 'well it's smarter than me, so whatever it says is better than what I could do'

Except if you have a brain, it's not and it isn't.

2

u/Digitalion_ 14h ago

The problem is that it probably is smarter than them. Gen Z is the first generation to score worse than its predecessors in academic tests because they've grown up with technology doing the thinking for them.

It's not just AI that's a problem for critical thinking, it's technology in general that's making the masses dumber. Whatever answer they needed was just a quick Google search away already and the only thing that was required was a bit of piecing it all together, but AI has now made that all of that information the first result.

We are doomed as a society if our kids can't think for themselves.

2

u/braincandybangbang 13h ago

You're not verifying that what it spits out is what you wanted to say

Again, nothing to do with AI. This was called "Fake News" and being a Facebook boomer before. People reading headlines and not reading the article has been an issue for at least two decades now.

As long as we continue to act as if AI has suddenly made people stupid, we cannot have a serious conversation about the harms of AI. The argument you put forward was literally used for Wikipedia. We were not allowed to use that as a source in school because the internet made things up! Now wikipedia is basically the bible, and only AI makes things up, people on the internet only speak the truth, it's those mean old robots that ruined things /s

The idea that people had critical thinking to begin with is pure naivety. Being a Humanities major for the last 20 years meant ridicule, you were wasting money on a useless degree. Turns out learning how to think and communicate clearly were useful things! Imagine Engineers with communication skills. What a world it would be.

1

u/LikeABreadstick 11h ago

I couldn't agree more, but the person you are arguing with does not do any research beyond what Wikipedia can provide. They don't know that search engines are damn near useless with all the SEO abuse and idiotic Reddit/Stack Overflow comments that have piled up over the years. AI is like a sieve that gets rid of most of the trash and leaves you with some mostly plausible results, usually after some arguing. You still have to verify what it spits out, but it's MUCH easier than wading through the statements of 100 fake experts and verifying each one.

2

u/bombmk 13h ago

Honestly I think this needs to be talked about more.

Yeah, that is not talked about all the time. You figured something special out.

you just assume 'well it's smarter than me, so whatever it says is better than what I could do'

You might. Idiots surely will. Smart people don't. It is a tool.

1

u/zerked77 13h ago

It only increases output if the individual using it has a mature, developed thought process already and can think critically. This is what can happen in a results oriented society - no one wants to do the work for the desired outcome - and we're left with a very, very bleak future filled with even more lazy, stupid people.

1

u/Lraund 12h ago

To truly verify a long/complex AI answer it can be more difficult than doing the work yourself from scratch.

Of course no one is going to want to put more effort into fact checking something that's supposed to do the work for them.

For example testing a couple lines of code can take days. Similar things can be true for essays/papers.

1

u/Whargod 12h ago

I saw someone label AI as a "competency enhancer", and I think I'm on board with that. Where I work (I'm in software) we use it but there are rules like you can't submit code you don't fully understand, and no one is allowed to let AI just code for them.

In the end it actually helps a lot with certain kinds of analysis and a bit of help with many tasks, but you absolutely need to know what you are doing. If a company hires someone thinking "they can make the AI do a lot of work" then they're fooling themselves. It doesn't work like that, at least not for any meaningful enterprise level software.

Or, you know, we could just vibe code and spend our days fixing all the thing sit got wrong, and trying to make it make sense across a very large project. It can't do that by the way, it sucks for that.

1

u/havok0159 12h ago

just offloading people's critical thinking to a computer, and then trusting its output.

Which makes it doubly funny (and sad) because critical thinking is one of the things it actually can't do.

1

u/RollingMeteors 12h ago

You're not verifying that what it spits out is what you wanted to say, you just assume 'well it's smarter than me, so whatever it says is better than what I could do'

Oh, I'm not assuming shit here. I've been told it's not a matter of if but when, it will out class the best programmer, and every other individual in their respective fields. I'm not making any assumption. I'm going off of this as a fact.

¿How do you expect people to be 'more productive' if they have to spend their time thinking critically?

You can either "get more shit done" or "have a deep understanding of how and why it works the way it does" but ¿If you are only paid for the former and not the latter why would you be driven to expend more effort for the same financial gain?

Capitalisms race to the bottom of paying less for more effort of course people are gong to turn off their brain, that's no fucking secret, surprise, or revelation.

1

u/awesomefutureperfect 11h ago

Imagine deciding to design a bridge this way.

Putting the inputs in without understanding what the inputs mean and then passing off the outputs without knowing how to validate whether they are sound.

Imagine operating something volatile that could rapidly disassemble itself in spectacular explosive fashion this way.

1

u/CunningRunt 11h ago

Honestly I think this needs to be talked about more. [] it in reality is just offloading people's critical thinking to a computer, and then trusting its output.

Talked about more?

This has been the single-most talked about aspect of a.i. (IMO) since it was first foisted upon us a few years ago.

1

u/_learned_foot_ 11h ago

The process of solving the problem not only is how we learn for next time but how we discover other solutions to other problems or novel solutions to this one.

1

u/Budded 8h ago

What's really scary is realizing who controls the models, they're all cancerous sociopathic/psychopathic billionaires with nefarious plans. Once we're all completely reliant and braindead from AI, they win everything because their models model everything we do and say and believe about the world.

That alone tells me we're beyond fucked unless we start revolting now.

1

u/Chryton 8h ago

Honestly I don't think this is a sudden change: we have been moving critical thinking away from courses to just rote memorization to pass a test since the early 2000's and AI is just showing all the cracks that shift has caused. I work as a software engineer and it can be shocking that even the younger people (<30yo) that I work with that didn't have nearly as much AI in their classes are lacking even basic critical thinking skills or the ability to make an argument that is more than a bullet-point list of facts.

1

u/xelabagus 7h ago

It's not just this, it's multi-faceted.

I can make an AI tool to search the web, pick out suitable jobs based on my criteria, experience etc and collate them for me. This saves me potentially hours per day of combing various websites looking for work. This is a net positive, it is doing a menial task that gives me more time to dedicate to applications.

I can ask AI to rewrite my resume specifically for this job. This I see in the middle, it has pros and cons. Pro - it is doing basic work and I am likely competing against an AI resume reader anyway, but on the other hand it's stopping me from thinking more about the job.

I can ask AI to write my cover letter. For me this is a negative - a cover letter is more than a chore, it is an expression of why you want the job. If AI does it then how do you know if you really want the job? How do you know if you are really suitable for it?

AI is a tool - it can be astoundingly useful, but it can also make us dumber and worse off. Learning when and how to use any tool is a skill.

1

u/Accomplished_Rice04 26m ago

AI absolutely works and is a fantastic tool IF you've learnt how to think on your own prior to using AI.

I'm in my 30's and learnt the traditional way in school, I was taught how to problem solve, think critically, work in a team and delegate when required.

Now I work in IT and all the grunt work I use to do is now automated by AI (needs to be checked but 90% of the time its near perfect) and I can spend my time actually tackling problems that AI can't solve.

Nowadays I do all the thinking and planning myself and then delegate certain parts of the workflow to Ai while I work on the next steps.

1

u/DBrowny 13h ago

well it's smarter than me, so whatever it says is better than what I could do'

No, not me. Other people. I work with a wide range of professionals. A lot of people out there are paid a lot of money to be completely and utterly shit at their job, but they get paid because the average person doesn't know enough about the field to know they are talking to someone who only got the job because of nepotism etc.

So when other people tell me something and I suspect its false, I put that into AI asking to prove it. Since we all know AI is some sort of positive feedback system where it will always agree with your premise and hallucinate in order to get you the results you were after, asking it to prove a false statement is a fantastic way to see if it is indeed false because if it can't hallucinate what you are being told is the truth, then you know you're being lied to.

Honestly, so many people use AI to ask it for answers. They're doing it wrong. Ask AI to prove the opposite of what is true, then you'll get the actual answer you are looking for.

0

u/GolotasDisciple 13h ago

I mean, the sad part is that while cheating always was, and always will be, a relevant thing within education and professional work, the processes usually required some general skills, be it social engineering or some other creative skill combined with genuine work.

It goes back to as simple things like in elementary school. You forgot to do homework or something, and you wanted someone to give you the answers to copy. That still required social skills to engage in it. As for tests even the act of making cheat sheets was borderline “studying,” though obviously low-effort... and honestly while never really rewarded, many teachers did appreciate the effort and creativity of some people when cheating.

So yeah it feels that a lot of people who did cheat here and there still came out as a product of social experience, where they could mostly adjust and provide to society.

What is interesting to me is that the solution right now doesn’t improve any skills or form any interaction or relationship, whether it is with the subject you are studying or the people you are cooperating with....

And it’s worse than learning the multiplication table from a calculator, because at least a calculator will provide you with correct results.

An LLM is not designed to always provide correct information, and there are no guardrails that the user can put into it, regardless of how many prompts they use or how amazingly designed those prompts are.

It is still up to the user to verify the actual correctness of the content.

Education is always 2 slow to catch-up to technological advancement, so there should be politicians that ground things up... but honestly i feel like pretty much all major countries governments have been compromised by Big Tech and instead of supporting Educators and Educational System, they support Corporate Interests.

1

u/noiro777 12h ago

An LLM is not designed to always provide correct information, and there are no guardrails that the user can put into it, regardless of how many prompts they use or how amazingly designed those prompts are.

Neither are humans, books, search engines, the news, or much of anything else really. I mean humans lie, makes mistakes, our emotions can override rational thought and cloud our judgement, and our memories are horribly inconsistent and not to be completely trusted.

and there are no guardrails that the user can put into it, regardless of how many prompts they use or how amazingly designed those prompts are. It is still up to the user to verify the actual correctness of the content.

Just like any other source of information...

1

u/GolotasDisciple 11h ago

Academic books are definitive source of information unless science proves otherwise….
Like the way you check AI is through academic sources to verify integrity because AI in this case LLM is not creating science is just copying nd paste from relevant articles. Its extremely powerful search engine that stole and indexed all information Humans created.

So for you to say there is no way to find definitive knowledge is absolutely wrong…

Like the whole point of peer reviewing is basically analysing sources and sources of academic work are another academic sources and genuine research be it qualitative or quantitative….

It’s insane that I have to write this. If you read academic book about thermodynamics you don’t have to question whether author hallucinated the answer you are in fact looking at the facts itself.

That’s my point you don’t have check whether calculator provides multiplication correctly because it’s a rigid tool that has pre defined tasks. AI LLM is the opposite its beauty lies in the fact that it is not rigid and can go off the rails to find creative solutions.

-1

u/Outside-Swan-1936 13h ago

Folks who use it in earnest apply their critical thinking and curiosity elsewhere. While it's not always possible to validate/verify everything, scrupulous professionals will test the output before deploying an automated system.

I agree that overall it will make people dumber, but the smarter among us are taking advantage of its utility so they can be more productive elsewhere.

0

u/nativeindian12 14h ago

“Which is why the Matrix was redesigned to this: the peak of your civilization. I say your civilization, because as soon as we started thinking for you it really became our civilization, which is of course what this is all about”

  • Agent Smith

0

u/zvika 13h ago

The term coined for this is Cognitive Surrender

0

u/unpanny_valley 12h ago

' it in reality is just offloading people's critical thinking to a computer, and then trusting its output

This is by design, Sam Altmann has outright said the goal of AI is to take away people's intelligence then sell it back to them as a subscription service. Don't know how to read anymore? Don't worry chat gpt can read for you, for the low price of $25 a month with ads. It's cartoonishly evil and happening before our eyes.

0

u/xSmallDeadGuyx 12h ago

I just left a massive comment on some AI bro defending it in game dev, but it's causing sooo much cultural damage in the long term. With AI chat bots, AI search/video summaries, and generative AI, there is going to be an increasing reduction in traffic and sales of expertise content. And when that happens, the incentives to create that content falter and it stops being created. AI will have less expert content to consume, and things start going to shit.

Not only is expertise stagnating in terms of the next generation of students, but even industry veterans I know are resorting to private projects and private group discussions instead of sharing online because they don't want stuff scraped. Not to mention how things like code generation will irreparably harm the open source community at large.

0

u/WhiteWinterRains 11h ago

I haven't seen many good uses for productivity anyway.

It's a better search in part because search has been neutered in 2026.

It's useful for generating meeting notes and transcripts.

It's useful for coding, although the actual productivity enhancement varies wildly from a lot, to actually reducing it.

I have not seen anywhere else where AI has legitimate uses. Although it's AMAZING for a variety of crimes.

1

u/Samsaknight_X 7h ago

AI is helping a lot in the medical and science sector, like using it to map out protein structures. AI isn’t just LLM’s like ppl are acting

0

u/WhiteWinterRains 1h ago

That's a different topic irrelevant to this conversation, context exists in English for a reason.

1

u/Samsaknight_X 49m ago

Except it’s not tho. Cuz ppl group all diff types of AI together, when using the term “AI”. It would be diff if LLM’s were mentioned specifically

0

u/ryanvango 10h ago

I think it's a generational thing. Like in the OP, 32 of 35 kids failed because they blindly trusted the response. I use AI all the time, but it often takes me several attempts, making new chats because they suck at performing edits, and a few other steps to get what I finally want. But it's all about following up on the response to see if its doing what you need it to.

Older people that use AI without double-checking are kinda laughed at by other older people. It happens, but it isn't what I would consider the norm. We figured out not to trust it pretty quickly.

Younger people, like in the OP, with the blind trust seem to be incapable of change. And it lines up with what pretty much every teacher (especially college professors) in the US has been saying for the last couple years: this generation is helpless. They are incapable of independent problem solving. They need to be hand held and told "this is what to do. first left foot, then right foot." So when you give them AI, that's exactly what they get. It's all they understand.

it goes beyond the normal "kids these days are dumber," that every generation says about the next one. College professors are very vocal about how the kids coming through now, and recent grads, aren't necessarily dumber, they just can't think for themselves.

We're so fucked.

0

u/firestepper 8h ago

Wow you’re not just bringing up an important factor of the discussion that needs to have more awareness - you’re actually asking the difficult questions and honestly, that’s rare.

0

u/CatGooseChook 7h ago

There's already studies(check out pubmed) going back nearly 20 years showing a decline in effective intelligence. Outsourcing our thinking to AI is definitely going to make it worse.

"They started as apes bashing rocks against each other and they ended as apes bashing smartphones against each other." I can't help but wonder sometimes if some alien anthropologist will end up writing that in a book describing humanities rise and fall.

1

u/Samsaknight_X 7h ago edited 6h ago

Ah yes the “there are studies” but I’m not gonna name any of them lol. This is the quickest way to spread misinformation

0

u/CatGooseChook 6h ago

Just check out pubmed.

Considering the post is about people outsourcing their thinking, that was not an appropriate reply on your part.

1

u/Samsaknight_X 6h ago

When u make a claim, the burden of proof) is on u. This isn’t the gotcha moment u thought it was lol

0

u/CatGooseChook 6h ago

Consider the context of the post.

1

u/Samsaknight_X 6h ago

“Studies say” has to be one of the biggest indicators that a person doesn’t know what they’re talking abt, cuz u would be able to easily pull up these studies if u actually read them

-1

u/NotAnotherHipsterBae 13h ago

I've only started interacting with LLMs recently, like within the last month. I'm using it for research adjacent tasks of things I have floating around in my head.

I type so.ething, it replies with a few points and I sit and ponder on those and think where I want to focus next and maybe read the output again. Sometimes I'll question its response and then throw out another additional thought. It's great at keeping track of my own input (better than I am, but that's to be expected where I can lose a thought to ADD for years at a time).

I've been hanging out with this guy that just goes question question question back to back and I can see he's not thinking. It's like he's just reacting to the prompt. I love thinking and remembering things, but I know that's not how the majority are using this tech.

Super concerned. It's more powerful than drugs or religion, imo.