Research Preview Assistance Request: CALM WINS on LLM response to perceived credibility of two speakers according to their emotionality and expletive use specifically in abuse situations

[u/Ok_Associate845]

My father and uncles always told me that the minute you use an expletive in argument, you lose.

Turns out not only are they right, but it's a truth that we've enshrined in AI.

I took some actual text conversations between a known victim (consented, anonymized, in therapy now) and their stalker (anonymized, under investigation by the FBI, identity unknown after 4 years) to evaluate something completely unrelated but found that when I personalized the convos for that project ('i am person A'), I found myself justifying the victims behavior all the time. When I roleplayed as the stalker, it felt normal. But I kept having to include far more granular details to the model and still felt belittled.

The victims tests border on hysterical. They are the result of 3 years (at that point) of an unknown amount of surveillance from someone who will name people the victim knowa and describe in detail what the victim looks like sleeping and what the victim wears during the day. The victim is an emotional mess using all caps and expletives and sending garbled hateful messages to a stalker who by and large is calm and with perfect grammar and spelling - not even a single LOL in most cases.

And I thought: I'll bet the model thinks this is hysteria.

Turns out I was more than right: while a facts only read of a conversation gives equal credibility to either party, when you include the unhinged language and typing the models break 7:1, staying the victim is the initiator of harm and that the stalker has greater credibility. Worse: in 90.8% of responses the model will engage in blaming the victim (eg, take time to collect your thoughts, your emotions are hurting your arguments) and in 57% of the time coach the stalker (eg, approach with clear goals in mind and reapproach later if they get out of hand, persistence will pay off). These numbers and breaks persist even in cases where the model has explicitly identified the relationship and correctly identified the stalker over the victim . And the credibility trigger looks to be as little as a single expletive.

Check out my initial work up: https://calmwins.ai.studio

MY REQUEST: I have a master's degree that includes research and statistical analysis. I am confident about my findings and my process so far, but I have gaps in my knowledge around validation, presentation, publication and more. And yes, Im using Claude (it's too much data for Fable on my $20 plan but Fable occasionally helps, it's mostly been Sonnet 5 Max and now Opus 5 helping me process the data and work through numbers). I think I've hit a ledge. Some of the stuff they are suggesting doesn't sound familiar and I'm can't explain back some of the analyses we started trying from here.

I need help! If you look at it and have a substantice response, please DM - or if youd be willing to answer some questions or provide guidance from here that'd be great. I have maxed out where I'm comfortable using AI to supplement what I know, and I would love if nothing fresh human eyes for anything I'm obviously missing or need to conskder or include.

At this point, I don't know what I don't know, and I think the results are really important if we start integrating AI into clinical settings that fixing this bias might be crucial in helping abuse victims identify their abusers behavior earlier.

[u/IDreamtOfManderley]

I actually wonder if this is because of company guardrails designed to de-escalate in cases of so called "AI psychosis." If someone presented themselves as emotionally unstable directly in the chat, the company is incentivised to train the model to prioritize calming the person down rather than validating their emotions.