The Answer Looked Better Before I Opened the Source
I've had this happen enough times with AI answers that I've started noticing the sequence.
I ask something. The answer comes back and sounds reasonable. The argument hangs together. There are caveats. It considers another possibility. Nothing immediately makes me think this is bullshit.
Then I open one of the sources.
Sometimes it's a LinkedIn post by somebody whose experience doesn't seem particularly relevant to the claim being made.
And my confidence drops.
The strange part is that the answer hasn't changed. Same sentences. Same reasoning. Same conclusion. I've just learned where one of the pieces came from.
At first, that reaction seemed pretty sensible to me.
Weak source. Lower confidence.
Except I wasn't sure I had actually established the first part.
Apparently citations can earn some trust before we check them
I assumed citations made an AI answer more trustworthy because they gave me a way to verify it.
Which they do.
But one experiment found something I wasn't expecting. People reported more trust in AI-generated answers when citations were present even when those citations were random. The researchers also found lower reported trust among participants who actually checked the citations.
That second finding doesn't prove that opening a citation caused trust to fall. People who were already more suspicious may simply have been more likely to check.
Still, I recognised something in it.
I've definitely looked at an answer with citations and felt slightly reassured before opening a single one.
That is slightly horrifying lol.
Because the citation marker has already done something before I've established whether whatever sits behind it deserves to.
And sometimes, when I finally do open it, the opposite happens.
A real source can still be the wrong evidence
The obvious AI citation failure is easy enough to understand.
The paper doesn't exist. The case was invented. The URL goes nowhere.
What interested me more was the quieter version, where the source is completely real.
Stanford researchers use the term misgrounding for one version of this in legal AI: the system can describe the law correctly while citing a source that doesn't actually support what it says.
Nothing needs to be fabricated.
The source can exist. It can even be an excellent source.
It just isn't evidence for that particular claim.
A large study of Google AI Overviews makes the distinction even harder to ignore. Researchers broke generated answers into 98,020 individual claims and checked them against the cited pages. 11% weren't supported by the pages attached to them. More surprisingly to me, the researchers found that source quality and whether a source actually supported the generated claim were largely independent.
I'd been collapsing two questions:
Is this a good source?
Does this source support this claim?
They aren't the same question.
A prestigious paper that doesn't establish the sentence attached to it is still a bad citation for that sentence.
And then there is my LinkedIn post.
Suppose somebody writes:
We tried X at our company and this happened.
That could be perfectly good evidence for:
This person says their company tried X and this happened.
It becomes much weaker evidence for:
Companies generally experience X.
The post hasn't changed.
What I'm asking it to prove has.
Because sometimes I might just dislike who's on the other end
When I open a citation and see somebody who doesn't seem “qualified enough,” what exactly am I reacting to?
Sometimes I think lowering my confidence is justified. If an AI makes a sweeping claim about enterprise buying and the evidence underneath it is one person's anecdote with no obvious connection to enterprise buying, I probably should want better evidence.
But I can also see how quickly my judgment can turn into something much lazier:
That's not much of an improvement.
A fancy title doesn't make somebody right. Someone can be highly qualified and still be talking outside their expertise. A company can know its own product better than almost anybody while also having an obvious reason to present it favourably. And an unknown practitioner might have exactly the first-hand experience relevant to the question I'm asking.
So I've started mentally finishing the sentence.
Not:
Is this person qualified?
but:
Qualified to tell me what?
That doesn't remove judgment from the process.
It makes the judgment annoyingly more specific.
I think the order matters to me too
This part is just something I've noticed about how I use these tools.
When I research something myself, I normally encounter the source while I'm encountering the claim. I know I'm reading a paper, a company blog, Reuters, Reddit or somebody's LinkedIn post while I'm deciding what I think of what it says.
Those labels are imperfect shortcuts. My reaction to them can be biased too.
But at least the provenance is sitting there in front of me.
With an AI answer, I often experience the order differently.
First I read one coherent explanation.
Then, if I open the citations, I discover where its different pieces came from.
That's probably part of why opening a source can feel so abrupt. I already encountered the claim in one context. Now I'm seeing the evidence in another.
And sometimes the second context makes the first one look shakier.
But that still doesn't tell me whether my reaction is correct.
So I still open the LinkedIn post
I just don't think seeing “LinkedIn” settles anything anymore.
Maybe the post really is flimsy evidence for what the AI claimed.
Maybe it reports something useful but the AI stretched one person's experience into a general rule.
Maybe the post supports the sentence perfectly well and I only disliked the credentials attached to it. Don't @ me lol 🥲
Or maybe there was simply much better evidence available and I should wonder why the AI didn't use that instead.
Those are different failures, and I used to bundle them together as:
bad source.
I don't think I can do that anymore.
The citation itself isn't proof that the answer deserves my trust.
But my immediate dislike of the citation isn't proof that it doesn't either.
I started this with a fairly simple irritation: sometimes AI gives me an answer that sounds good, then I open a source and think, really? This is what we're relying on?
I still have that reaction. I'm just less certain now that I would have chosen better for the right reasons.