Why Last Week’s User Count Can Change
I thought old analytics numbers had one useful property: once the day was over, they stopped moving.
Monday happens. Events get recorded. Monday ends.
Maybe the dashboard needs a while to process everything, but eventually Monday becomes history. If I come back on Thursday and ask what happened on Monday, I might run a different analysis or notice something I missed. I wasn't expecting the same Monday report to quietly contain fewer users.
Apparently, that can happen.
Say one person uses a product anonymously from their phone on Monday morning. Later that day they open it anonymously from their laptop.
As far as the analytics system knows, there may be two separate identities:
phone → anonymous identity A
laptop → anonymous identity B
Then on Wednesday, the person logs into the same account from both devices.
The system has learned something it didn't know on Monday:
identity A ─┐
├→ same user
identity B ─┘
Nothing new happened on Monday.
But the system now understands Monday differently.
Depending on how the analytics product handles identity, the user count for Monday can change with it.
The events don't have to change
Amplitude documents this behaviour directly.
Someone can initially use a product anonymously and later become associated with a known User ID. When Amplitude learns that activity it had treated separately belongs to the same user, those identities can be merged. Amplitude also says historical user counts can decrease as this happens, and that counts may stabilise over time as anonymous users return and become identified.
That was the bit that caught me.
How does a historical user count decrease?
Amplitude's warehouse documentation helped me understand it. Its raw event data can retain the original IDs attached to the events, while a separate mapping records identities that have since been merged. The dashboard can then use that mapping when calculating users.
So Monday's raw history could still look roughly like this:
10:00 → event from identity A
14:00 → event from identity B
Later, the system learns:
A and B belong to the same user
The events themselves didn't have to be rewritten. What changed was the relationship between them.
And because a metric like "unique users" depends on that relationship, its result can change too.
I had been picturing a historical metric as something much closer to a stored fact.
In cases like this, it is partly a calculation over stored facts using what the system currently knows about identity.
Was the first number wrong?
My first instinct was yes.
If one human used the product and Monday originally showed two users, surely 2 was wrong and 1 corrected it.
I'm not completely comfortable with that description.
On Monday, the system may genuinely have had two anonymous identities and no information connecting them. It wasn't necessarily miscounting the identities it knew about.
By Wednesday it had more information.
If what we wanted to measure was distinct people, merging the two identities may give us a better reconstruction of Monday.
But later information isn't automatically better information.
Identity resolution can go wrong in the other direction too. Google Analytics warns, for example, that if a product assigns the same User-ID to two different people, Analytics can interpret their activity as belonging to one user.
So I don't think the useful distinction is simply:
wrong answer → correct answer
It is closer to:
answer using the identity information available then
↓
new identity information arrives
↓
revised answer using what the system knows now
The actual past is fixed.
Our reconstruction of it may not be.
Identity resolution has a time dimension
The technical problem underneath this is called identity resolution.
A product usually doesn't have some universal identifier that follows a person everywhere. It sees whatever identifiers its instrumentation gives it: a device ID, browser ID, account ID, login, email, or something else.
Identity resolution is the machinery that decides when different trails of activity should be treated as belonging to the same entity.
That itself isn't new or surprising. Analytics products have had to deal with people switching devices and moving from anonymous to logged-in usage for a long time.
What I hadn't thought about was the time dimension.
If an analytics system learns today that two identities belong together, how much of yesterday is it allowed to reinterpret using that information?
There isn't one universal answer.
Some history has a boundary
Google Analytics has limits around retroactive identity.
Its User-ID feature can connect behaviour across sessions and devices when a product supplies a consistent identifier. But Google explicitly says data collected before User-ID was implemented is not retroactively reprocessed and associated with those IDs.
There is a boundary beyond which that new identity information does not reach.
Google also lets a property change the reporting identity used in reports. It can rely on device identity or use combinations involving User-ID and other information. Changing that reporting choice can alter how users are deduplicated without rewriting the underlying collected events.
Amplitude has another kind of boundary, and this one made the trade-off much clearer to me.
For some plans, Amplitude uses a billing metric called Monthly Tracked Users. Identity merges can affect that calculation too. After the month ends, however, Amplitude eventually performs a final calculation that accounts for late-arriving events. Once that final calculation is done, later identity merges no longer keep changing the billed MTU count for that month.
For analytics, continuing to reconcile identities can make an old report better reflect what the system knows now.
For billing, letting last month's number move indefinitely would create another problem entirely.
At some point, that number needs to become final.
Better reconstruction or stable history?
That leaves two useful things pulling in different directions.
Let the system keep revising historical user counts and old reports can benefit from identity information that arrives later.
But rerunning the same report later may no longer reproduce the number someone originally saw.
Freeze the report immediately and you get stability, but you may preserve identities that later evidence suggests belonged together.
I don't think one of those is always better.
For an exploratory product dashboard, I might want the best reconstruction available now.
For an invoice, there has to be a point where the number stops moving.
An audit record, experiment, or financial report may have different requirements again.
Those examples aren't interchangeable, and I don't want to pretend they all need the same rule.
What they made me notice is that there are at least two different things I could mean by a historical metric:
What did the system know at the time?
or:
Given what the system knows now, what does it think happened then?
Those questions can produce different numbers.
Not every part of the past changes together
There is another boundary worth keeping straight.
Learning that two identities belong to the same user does not mean the system should rewrite everything it knows about old events.
So a system can effectively learn:
these two old events belonged to the same person
without changing:
what properties were attached to those events when they happened.
That means "does history change?" is probably too broad a question.
Different parts of the historical record can follow different rules.
Raw events can remain untouched.
Identity can be reconciled later.
Event-time properties can remain tied to the time of the event.
A billing number can eventually freeze.
Once I separated those things, the behaviour stopped looking quite as strange.
The number is about Monday. Part of the answer might come from Wednesday.
If I ran a report and saw:
Monday
1,020 users
then came back later and saw:
Monday
995 users
I'd probably assume something had gone wrong.
Maybe something did. But another possibility is that the system learned that some identities it previously counted separately belonged together. The report still describes Monday, even though part of the information used to calculate it may have arrived later.
I hadn't really thought about historical analytics that way. I assumed that, once enough time had passed, a metric simply became final. But an event being in the past and our understanding of that event being finished aren't necessarily the same thing.
How long that interpretation should remain open seems to depend on what the number is for. An analytics report may benefit from later identity information. A billing number eventually needs to stop moving. There doesn't seem to be one answer that works for both.
Stay in touch
Occasional reflections on mindfulness and intentional living.