Project

General

Profile

Bug #14933

Feature #5630: Reproducible builds

stable branch is not reproducible: differences in some .fa.html website files

Added by intrigeri 12 months ago. Updated 9 months ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
-
Target version:
Start date:
11/08/2017
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
bugfix/14933-non-deterministic-inline-posted-timestamp
Type of work:
Code
Blueprint:
Starter:
Affected tool:

diffoscope.html View (15.3 KB) intrigeri, 11/08/2017 10:37 AM


Related issues

Related to Tails - Bug #12726: There should be a date on the notes in the News section of the website Resolved 06/17/2017

Associated revisions

Revision 0a484302 (diff)
Added by intrigeri 11 months ago

auto/build: normalize file timestamps in wiki/src before building (refs: #14933)

The copy of the website included in the ISO image has "Posted" timestamps that
apparently match when we cloned the Git repository. This is expected since we
build the website with the rcs setting disabled. Normalizing timestamps before
building should fix this.

What's surprising though is that only a few pages seem to be affected, while
I would expect that all inlined pages without an explicit meta date should
trigger the exact same problem. But anyway, normalizing mtime should fix it
everywhere, so if it works, let's not bother investigating this further.

One potential drawback of this commit is that it may make it harder to refresh
an older, cached build of the website when we reintroduce this option in the
future: ikiwiki will have a hard time knowing which pages it should refresh.
We'll see.

Revision cd1a448c
Added by bertagaz 11 months ago

Merge remote-tracking branch 'origin/bugfix/14946-mergebasebranch-breaks-reproducibility'

Fix-committed: #14946, #14933

Revision 3a622350 (diff)
Added by anonym 10 months ago

Vagrant: never build the wiki early.

This has caused several issues throughout the years, the lastest
instance being the reopening of #14933, so let's just remove this
feature. The optimization this allows for when rebuilding the same
branch frequently (together with the keeprunning option) was probably
only used by me, and rarely at that, so I'd argue it is not worth the
maintenance cost.

Will-fix: #14933

Revision 19befbd9
Added by intrigeri 10 months ago

Merge remote-tracking branch 'origin/bugfix/14933-non-deterministic-inline-posted-timestamp' into stable (Fix-committed: #14933)

History

#1 Updated by intrigeri 12 months ago

I'll have a very quick first look in the hope I can easily find out where the root cause is. If the fix is not trivial I'll coordinate with anonym to see who handles this.

#2 Updated by intrigeri 12 months ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to anonym
  • % Done changed from 0 to 10
  • Type of work changed from Research to Code

I think we didn't notice this earlier because the fix for #12726 had been merged into master but not into stable. Yesterday I've merged master into stable (precisely thinking about such potential problems) which is why we now see it. I'm glad we noticed this before we're closer to the 3.3 release.

So, in the source of affected pages we have [[!inline pages="doc/encryption_and_privacy/gpgapplet.warning" raw="yes" sort="age"]]. The mtime of wiki/src/doc/encryption_and_privacy/gpgapplet.warning.fa.po is used as the timestamp I see in the diffoscope output matches:

08:08:29 + TAILS_GIT_DIR=/home/vagrant/amnesia
08:08:29 + [ ! -d /home/vagrant/amnesia ]
08:08:29 + git clone /amnesia.git/.git /home/vagrant/amnesia
08:08:29 Cloning into '/home/vagrant/amnesia'...
08:08:34 done.

I think the mtime is used (instead of the time of last change in Git) because our ikiwiki.setup disables the rcs parameter. I suspect we should normalize such timestamps of files in wiki/src before building the website in auto/build, just like we already do for config/binary_local-includes and config/chroot_local-includes. anonym, wanna give it a try or should I?

Now, honestly I'm a bit confused: I don't understand why this problem only appears for this specific inlined page and only in Farsi. The way I understand the problem currently, all inlined pages without an explicit meta date should trigger the exact same problem.

Might it be the problem I was refering to in "non-deterministic inlined posts' date: any inline can be affected as long as it uses a template that displays the date; this is the problem that made us do 104d7f89ffd0a83962eb6eb891e1c2d390cf62b0 initially; your validation code addresses that… as long as it covers all affected inline directives; I think it does" (#12726)? Apparently I was wrong and we don't check all affected inline directives. But if merely normalizing mtimes works, let's not bother.

#3 Updated by intrigeri 12 months ago

  • Related to Bug #12726: There should be a date on the notes in the News section of the website added

#4 Updated by intrigeri 11 months ago

  • Assignee changed from anonym to intrigeri

#5 Updated by intrigeri 11 months ago

  • QA Check set to Ready for QA
  • Feature Branch set to bugfix/14933-non-deterministic-inline-posted-timestamp

(I have to make this RfQA otherwise Jenkins won't test reproducibility.)

#6 Updated by intrigeri 11 months ago

  • Assignee changed from intrigeri to anonym
  • % Done changed from 10 to 50

Woohoo, my fix works. The ISO is still not reproducible in our CI environment:
https://jenkins.tails.boum.org/view/Tails_ISO/job/reproducibly_build_Tails_ISO_bugfix-14933-non-deterministic-inline-posted-timestamp/1/artifact/build-artifacts/diffoscope.html but for a different reason, that I've reported on #14946. I think this problem won't affect release builds but I'm not 100% sure and anyway I've found a rather trivial fix. In any case that should not block fixing this problem.

#7 Updated by intrigeri 11 months ago

  • Blocks Bug #14946: Topic branches that lag behind their base branch don't build reproducibly with mergebasebranch build option added

#8 Updated by anonym 11 months ago

intrigeri wrote:

Woohoo, my fix works.

Yay!

The ISO is still not reproducible in our CI environment:
https://jenkins.tails.boum.org/view/Tails_ISO/job/reproducibly_build_Tails_ISO_bugfix-14933-non-deterministic-inline-posted-timestamp/1/artifact/build-artifacts/diffoscope.html but for a different reason, that I've reported on #14946. I think this problem won't affect release builds but I'm not 100% sure

I'm 99% sure this isn't a problem for releases: they are built from a tag which makes mergebasebranch into a noop.

#9 Updated by intrigeri 11 months ago

  • Assignee changed from anonym to bertagaz

#10 Updated by bertagaz 11 months ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (bertagaz)
  • % Done changed from 50 to 100
  • QA Check changed from Ready for QA to Pass

Apparently that seems to do the trick, so it's been merged together with #14946.

#11 Updated by bertagaz 11 months ago

  • Status changed from Fix committed to Confirmed
  • Target version changed from Tails_3.3 to Tails_3.5
  • % Done changed from 100 to 70
  • QA Check changed from Pass to Dev Needed

bertagaz wrote:

Apparently that seems to do the trick, so it's been merged together with #14946.

During the 3.3 release, we noticed this problem was not really gone: when $TAILS_MERGE_BASE_BRANCH is not set, the wiki is built in vagrant/provision/assets/build-tails, where the timestamps have not been yet cleaned. When it gets refreshed later in auto/build, the problem with the openpgp_applet fa.html files is not fixed.

#12 Updated by intrigeri 11 months ago

  • Assignee set to anonym
  • Priority changed from Elevated to High

#13 Updated by anonym 11 months ago

  • Blocks deleted (Bug #14946: Topic branches that lag behind their base branch don't build reproducibly with mergebasebranch build option)

#14 Updated by intrigeri 11 months ago

anonym will investigate this by December 11 and then might delegate it to me.

#15 Updated by anonym 10 months ago

intrigeri wrote:

anonym will investigate this by December 11 and then might delegate it to me.

I'll look at it today (off-by-one error, whoops! :)).

#16 Updated by anonym 10 months ago

bertagaz wrote:

bertagaz wrote:

Apparently that seems to do the trick, so it's been merged together with #14946.

During the 3.3 release, we noticed this problem was not really gone: when $TAILS_MERGE_BASE_BRANCH is not set, the wiki is built in vagrant/provision/assets/build-tails, where the timestamps have not been yet cleaned. When it gets refreshed later in auto/build, the problem with the openpgp_applet fa.html files is not fixed.

I can confirm that building with mergebasebranch produces the same ISO each time on my system:

md5:     25adfb21ffed211ac154aa6669d87163  tails-amd64-3.3.iso
sha1:    7bf7f11f206803d4690dfa1ce7d4da35735ea262  tails-amd64-3.3.iso
sha256:  82b1286442be5f72d68d36166a352de7b12472d0e94692ca64fcc6e6a113e343  tails-amd64-3.3.iso
sha512:  0ef9de2d3965c06738a4556d8f6d4fa5aa9fe1196069c0a624de23558bfb141b566b6473078e50a3f89c62fb0da792609415dcf73429c69181ee32adf97fe005  tails-amd64-3.3.iso

So, most likely Jenkins built that exact image, but the artifacts from all jobs built from that commit are absent. I've instead asked jvoison to rebuild Tails 3.3 with mergebasebranch set. If he gets the same SHAAAA as me, we're good, imho.

As for the actual fix, see 3a622350a5 which removes the early wiki build -- I'm probably the only one using it (together with keeprunning), which isn't very often, and if I ever really need it I can just drop ./build-wiki into build-tails temporarily.

#17 Updated by anonym 10 months ago

  • Status changed from Confirmed to In Progress

#18 Updated by cypherpunks 10 months ago

$ TAILS_BUILD_OPTIONS="mergebasebranch" rake build 
[…]
$ md5sum tails-amd64-3.3.iso
25adfb21ffed211ac154aa6669d87163  tails-amd64-3.3.iso
$ sha1sum tails-amd64-3.3.iso
7bf7f11f206803d4690dfa1ce7d4da35735ea262  tails-amd64-3.3.iso
$ sha256sum tails-amd64-3.3.iso
82b1286442be5f72d68d36166a352de7b12472d0e94692ca64fcc6e6a113e343  tails-amd64-3.3.iso
$ sha512sum tails-amd64-3.3.iso
0ef9de2d3965c06738a4556d8f6d4fa5aa9fe1196069c0a624de23558bfb141b566b6473078e50a3f89c62fb0da792609415dcf73429c69181ee32adf97fe005  tails-amd64-3.3.iso
$ lsb_release -a
Distributor ID:    Ubuntu
Description:    Ubuntu 17.04
Release:    17.04
Codename:    zesty
$ uname -a
Linux jv 4.10.0-42-generic #46-Ubuntu SMP Mon Dec 4 14:38:01 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$                                            

#19 Updated by anonym 10 months ago

  • Assignee changed from anonym to intrigeri
  • % Done changed from 70 to 80
  • QA Check changed from Dev Needed to Ready for QA

Success! Please review'n'merge into stable!

#20 Updated by intrigeri 10 months ago

Code review passes. I've updated the branch so it has a chance to build on Jenkins.

#21 Updated by intrigeri 10 months ago

  • Status changed from In Progress to Fix committed
  • % Done changed from 80 to 100

#22 Updated by intrigeri 10 months ago

  • Assignee deleted (intrigeri)
  • QA Check changed from Ready for QA to Pass

Merged! Let's see how it goes next time we release :)

#23 Updated by intrigeri 10 months ago

  • Target version changed from Tails_3.5 to Tails_3.4

#24 Updated by anonym 9 months ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF