Project

General

Profile

Bug #14924

Feature #5630: Reproducible builds

reproducibly_build_Tails_ISO_stable Jenkins job always fails

Added by intrigeri 5 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Continuous Integration
Target version:
Start date:
11/06/2017
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

We're quite unlucky: build_Tails_ISO_stable is scheduled daily, and with Jenkins' hashing system it happens to run at 07:15 so reproducibly_build_Tails_ISO_stable starts around 08:00. We update the Debian security APT snapshot at 7:41 so in practice, reproducibly_build_Tails_ISO_stable always gets a different snapshot and thus always produces a different ISO.

This prevents us from easily noticing when an update or branch merge broke reproducibility. And possibly worse, it teaches us to ignore email notifications from Jenkins (this has been going on for a week and nobody noticed or if they did, then they kept it to themselves for some reason).

Due to the way our Jenkins jobs are generated, we have little flexibility here: we can't simply reschedule build_Tails_ISO_stable (and only this one) to run at a different time. And if we start playing with the H symbol we'll be lucky if we don't break devel or testing.

The simplest option we have seems to slightly reschedule the APT snapshot update. Now, of course this will likely break "reproducibly" (sic) jobs for other branches, but until we have a better solution, not breaking stable/testing/devel gets higher priority by far. While implementing this I'll need to be careful not to break testing or devel, but at least I can easily reason about it, while I don't know how the Jenkins hash thing works exactly.

Long term, the only sane option is probably to teach our build system how to use snapshots specified on the command line (instead of "latest"), and to teach Jenkins to pass the correct parameters when trying to rebuild a given ISO image. Whoever feels responsible for this, please file a ticket about it.


Related issues

Related to Tails - Feature #12633: Lower the workload caused by reproducible builds Jenkins jobs Resolved 10/22/2017
Related to Tails - Bug #15107: Some topic branches are never built reproducibly during Jenkins daily runs => add option to specify which APT snapshot serials to use during build In Progress 12/26/2017
Blocked by Tails - Bug #14923: devel branch FTBFS since torbrowser-launcher 0.2.8-4 was uploaded Resolved 11/06/2017
Blocked by Tails - Bug #14946: Topic branches that lag behind their base branch don't build reproducibly with mergebasebranch build option Resolved 11/10/2017

Associated revisions

Revision f3fafe5a (diff)
Added by intrigeri 5 months ago

Revive the testing branch so I can take it into account (refs: #14924)

Revision aafdf8da (diff)
Added by bertagaz 4 months ago

Add APT_SNAPSHOT_SERIAL build option.

It enables to specify which time-based APT snapshot repositories will be
used as 'latest' during the build, and will set it accordingly in the
resulting ISO image. Useful to reproduce another ISO image.

Refs: #15107, #14944, #14924

History

#1 Updated by intrigeri 5 months ago

  • Parent task set to #5630

#2 Updated by intrigeri 5 months ago

  • Related to Feature #12633: Lower the workload caused by reproducible builds Jenkins jobs added

#3 Updated by intrigeri 5 months ago

  • % Done changed from 0 to 50
  • QA Check set to Ready for QA

Let's assume a 70 min delay between the time when the 1st build resolves the "latest" snapshot pointer and the time when the 2nd build does (currently we're around 52 minutes but let's be on the safe side: the longer this delay, the more chances the problem this ticket is about happens). Let's assume this pointer resolution happens 1st thing during a build to simplify (in practice it's more ~2 minutes later, whatever).

Frozen branches:

  • daily build of stable starts at 07:15 => the debian-security repo should not be updated between 07:15 and 08:25
  • daily build of testing starts at 15:23 => the debian-security repo should not be updated between 15:23 and 16:33

Non-frozen branches:

  • daily build of devel starts at 06:56 => no repo should be updated between 06:56 and 08:06
  • daily build of feature/buster starts at 09:15 => no repo should be updated between 09:15 and 10:25
  • daily build of feature/tor-nightly-master starts at 08:15 => no repo should be updated between 08:15 and 09:25

So ideally no repo should be updated between 06:56 and 10:25, and the debian-security repo should not be updated between 15:23 and 16:33. I've reworked the way we schedule these cronjobs so this is now guaranteed: all updates now start between 05:00 and 05:45, between 11:00 and 11:45, between 17:00 and 17:45, and between 23:00 and 23:45 (UTC).

The catch is that I've assumed the repo updates take 0 time, which is of course incorrect. But as long as no such update takes more than ~1 hour we should be good. Anyway, I think that this change will already improve things in the vast majority of cases. If practice proves me wrong I'll come back to it.

If Jenkins ever changes its hashing algorithm that determines when "daily" jobs are run for a given branch, we'll need to come back to it and possibly adjust automatic_refresh_delta_hours in tails::reprepro::snapshots::time_based.

I'll now wait ~36 hours so that all common cases have been tested.

#4 Updated by intrigeri 5 months ago

  • Status changed from Confirmed to In Progress

#5 Updated by intrigeri 5 months ago

  • Blocked by Bug #14923: devel branch FTBFS since torbrowser-launcher 0.2.8-4 was uploaded added

#6 Updated by intrigeri 5 months ago

So, at least this seems to be fixed for the stable branch :) I'll come back to it once the other ones don't FTBFS anymore.

#7 Updated by intrigeri 5 months ago

  • Blocked by Bug #14946: Topic branches that lag behind their base branch don't build reproducibly with mergebasebranch build option added

#8 Updated by intrigeri 5 months ago

  • % Done changed from 50 to 60

Daily build of devel is OK too. For feature/tor-nightly-master I'll wait for #14946 to be fixed before fixing the FTBFS (=> saving storage space). We'll see if this happens early enough (i.e. today) for me to have data here before the 3.3 release.

#9 Updated by intrigeri 5 months ago

  • Target version changed from Tails_3.3 to Tails_3.5

intrigeri wrote:

Daily build of devel is OK too. For feature/tor-nightly-master I'll wait for #14946 to be fixed before fixing the FTBFS (=> saving storage space). We'll see if this happens early enough (i.e. today) for me to have data here before the 3.3 release.

The merges happened too late for me to get data already, so postponing.

#10 Updated by intrigeri 5 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 60 to 100
  • QA Check changed from Ready for QA to Pass

Everything works as intended. For the testing branch, we'll see during the next freeze.

#11 Updated by bertagaz 4 months ago

  • Related to Bug #15107: Some topic branches are never built reproducibly during Jenkins daily runs => add option to specify which APT snapshot serials to use during build added

Also available in: Atom PDF