Bug 1216691

Summary: SOURCE_DATE_EPOCH is no longer deterministic
Product: [openSUSE] openSUSE Tumbleweed Reporter: Bruno Pitrus <brunopitrus>
Component: BasesystemAssignee: Jan Zerebecki <jzerebecki>
Status: NEW --- QA Contact: E-mail List <qa-bugs>
Severity: Major    
Priority: P2 - High CC: Andreas.Stieger, bwiedemann, dmueller, jzerebecki
Version: Current   
Target Milestone: ---   
Hardware: x86-64   
OS: openSUSE Tumbleweed   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Bug Depends on:    
Bug Blocks: 1047218    

Description Bruno Pitrus 2023-10-30 07:02:17 UTC
Recently on x86_64 and ix86 Tumbleweed SOURCE_DATE_EPOCH has started to increase on every rebuild, defeating its entire purpose as build-compare now decides that the output has changed and pushes a spurious update to users.

I observed the problem with signal-desktop which embeds SOURCE_DATE_EPOCH in plaintext in the package:

[  508s] Extracting packages
[  510s] /usr/libexec/signal-desktop/config/local-production.json differs (JSON text data)
[  510s] --- old//usr/libexec/signal-desktop/config/local-production.json	2023-10-30 04:05:17.000000000 +0000
[  510s] +++ new//usr/libexec/signal-desktop/config/local-production.json	2023-10-30 06:03:43.000000000 +0000
[  510s] @@ -1 +1 @@
[  510s] -{"buildCreation":1698211517000,"buildExpiration":1705987517000}
[  510s] +{"buildCreation":1698211518000,"buildExpiration":1705987518000}

This seems to be caused by this change moving the setup from rpmbuild to OBS: https://github.com/openSUSE/post-build-checks/pull/58
Comment 1 Jan Zerebecki 2023-10-30 08:27:49 UTC
(In reply to Bruno Pitrus from comment #0)
> Recently on x86_64 and ix86 Tumbleweed SOURCE_DATE_EPOCH has started to
> increase on every rebuild

Not if you rebuild the package. Only when OBS changes the dependencies, which is a build input. Different input makes different output, which gets a different epoch according to its definition. I.e. it is deterministic.

> increase on every rebuild, defeating its entire purpose as build-compare now
> decides that the output has changed and pushes a spurious update to users.

It is the other way around, it was broken before and is now fulfilling its purpose.

> This seems to be caused by this change moving the setup from rpmbuild to
> OBS: https://github.com/openSUSE/post-build-checks/pull/58

That link is supposed to explain how to make build-compare still work for this use case.

Is there anything it did not explain? Any way to make it easier to understand?
Comment 2 Bruno Pitrus 2023-10-30 09:38:06 UTC
Yes — this does not break determinism for local builds. But it breaks the way build-compare is used on OBS if the package embeds SOURCE_DATE_EPOCH in any way.

This is the same issue as with putting the rebuild counter into the binary — it also breaks build-compare. Even if it's not a real indeterminism when you wear the rpmbuild glasses, it's an indeterminism if you wear the package maintainer glasses, as the rebuild counter is something you have no control over.
Comment 3 Bruno Pitrus 2023-10-30 09:49:11 UTC
(In reply to Jan Zerebecki from comment #1)
> It is the other way around, it was broken before and is now fulfilling its
> purpose.
The ONLY purpose of SOURCE_DATE_EPOCH is reproducible builds. See e.g. https://gcc.gnu.org/onlinedocs/cpp/Environment-Variables.html

The place which is broken is either %clamp_mtime_to_source_date_epoch or your tool which expects rpm timestamps to be monotonically increasing.
Comment 4 Jan Zerebecki 2023-10-30 10:11:35 UTC
While the tool that expects an increasing timestamp is broken, not increasing them still breaks the definition of SOURCE_DATE_EPOCH. Debian also fixed a bug like this in what they call binNMU.

Embedding SOURCE_DATE_EPOCH in the output is by definition of that variable a change in output. So if you expect the output to not have changed then the embedding is broken. There is AFAIK always a better way than embedding that variable. E.g. in case of the timebomb in Signal, they probably want the date of the last upstream source git commit. See the github link.

Did you read the link? Can you answer my question?
Comment 5 Jan Zerebecki 2023-10-30 10:16:38 UTC
"incorporating any packaging-specific modifications." - https://reproducible-builds.org/specs/source-date-epoch/

The OBS change to a new set of build dependencies is a packaging-specific modification.
Comment 6 Bruno Pitrus 2023-10-30 10:22:10 UTC
I see that mtime clamping has use for both rsync and bit-perfect local rebuilds. Still imho a better solution would be to introduce a new variable which monotonically increases and is used ONLY for clamping, and not abuse SOURCE_DATE_EPOCH which is expected to remain unchanged by build-compare.
I'm with Bernhard on this one:
> I think, we should keep this "adding of release" off by default as it will break build-compare and build-tree-pruning for a large number of packages that already use SOURCE_DATE_EPOCH as timestamp.
Comment 7 Bruno Pitrus 2023-10-30 10:23:14 UTC
(In reply to Jan Zerebecki from comment #5)
> The OBS change to a new set of build dependencies is a packaging-specific
> modification.
it's broken also if you just manually click “rebuild” — no changed build dependencies in that case, only the counter
Comment 8 Jan Zerebecki 2023-10-30 10:32:35 UTC
It is not an abuse of SOURCE_DATE_EPOCH, instead it now matches the definition. I spoke with multiple people in the reproducible builds project about this and they agreed.

Manually clicking rebuild may also change the dependencies.

The Signal build script needs to be fixed if their intention is different than this effect.