Project

General

Profile

Feature #5630

Reproducible builds

Added by Tails over 4 years ago. Updated 1 day ago.

Status:
In Progress
Priority:
High
Assignee:
Category:
Build system
Target version:
Start date:
09/23/2015
Due date:
% Done:

92%

QA Check:
Feature Branch:
Type of work:
Code

Description

To ensure integrity against build machine or developer compromise, we should be able to produce identical binaries when building the same source on two different (but possibly identically configured) machines.

Team: anonym, lamby, bertagaz, u, kibi and intrigeri


Subtasks

Bug #10232: Investigate vagrant-lxc for our build systemRejected

Feature #11966: Reproducible website buildResolved

Feature #11967: refresh-translations: don't update PO files unless something other POT-Creation-Date has changedResolved

Bug #11970: sys-subsystem-net-devices-multi-user.device/start times outResolved

Feature #11971: Consider migrating some of /lib/live/config/* to systemd unit filesResolved

Feature #11972: Switch our Jenkins ISO build system to vagrant-libvirtResolved

Feature #12017: Update our Vagrant ISO build basebox wrt. vagrant-libvirtResolved

Feature #12327: Upgrade Lizard's memory againDuplicate

Feature #11974: Reproducible IUK buildsResolved

Feature #11976: Migrate to mksquashfs that honors $SOURCE_DATE_EPOCHResolved

Feature #11979: Move Vagrant's apt-cacher-ng data to a dedicated diskResolved

Feature #11980: Create and provision a new Vagrant VM for every ISO buildResolved

Feature #11981: Delete the Vagrant VM used for an ISO build once it is finishedResolved

Feature #11982: Set up processes to update the Vagrant ISO build baseboxResolved

Feature #11983: Check if the test suite has more failures on the reproducible ISOResolved

Bug #11986: Generated APT auto-removal config file encodes the build system's kernel versionResolved

Bug #11987: ikiwiki includes "Posted" timestamps in some generated web pagesResolved

Feature #11988: Update Vagrant boxes management design doc with meeting notesResolved

Feature #12002: Estimate hardware cost of reproducible builds in JenkinsResolved

Bug #12032: The SquashFS creation is not deterministicResolved

Feature #12051: Use ikiwiki 3.20161219+ instead of our patched oneResolved

Bug #12329: /usr/share/locale/*/LC_MESSAGES/tails.mo's POT-Creation-Date depends on the build timeResolved

Bug #12330: initrd.img is not generated reproduciblyRejected

Feature #12338: Test ISO build reproducibility with varying number of CPU coresResolved

Feature #12339: Have the ISO build reproducibility regardless of the current timeResolved

Feature #12345: Test ISO build reproducibility with varying CPU typeResolved

Bug #12347: Investigate why the reproducible ISO is larger than 3.0~beta2In Progressanonym

Feature #12348: Review'n'merge the reproducible builds branch into feature/stretchResolved

Feature #12351: Test building an ISO with a fake system time set in 2018Confirmedanonym

Feature #12352: Error out if trying to build an ISO with a system time before SOURCE_DATE_EPOCHResolved

Feature #12409: Reconsider the need for publishing Vagrant baseboxesResolved

Bug #12453: Invalid MBR ID passed to isohybridResolved

Feature #12505: Switch isobuilders to vagrant-libvirt in PuppetIn Progressbertagaz

Bug #12527: Duplicate gpg-agent killing code introduced in the build-tails scriptResolved

Bug #12529: Vagrant box creation needlessly downloads Linux from debian-securityResolved

Bug #12530: Vagrant box creation fails: can't unmount chroot that's busyResolved

Bug #12531: ISO builds on Jenkins are fragile since the migration to vagrant-libvirtResolved

Bug #12577: Aborted Jenkins ISO builds can cause next ones on the same isobuilder to failResolved

Bug #12578: Abort if Vagrant create_box failsResolved

Bug #12599: /var/lib/libvirt/images gets filled on isobuildersResolved

Bug #12618: Retrieving ISO build artifacts sometimes fails on JenkinsResolved

Bug #13302: /var/lib/libvirt/images sometimes gets filled on isobuilders, take 2Resolved

Bug #12541: isobuilders memory check keeps switching between OK and WARNING since the switch to VagrantResolved

Bug #12565: Test failures on Jenkins due to lack of disk spaceResolved

Bug #12566: ikiwiki image size specification makes the ISO build unreproducibleResolved

Bug #12567: fontconfig cache is not generated reproducibly even with patch from Debian#857892Resolved

Bug #12574: isobuilders system_disks check keeps switching between OK and WARNING since the switch to VagrantResolved

Bug #12575: Fix basebox:clean_oldResolved

Feature #12576: Have Jenkins use basebox:clean_old instead of basebox:clean_allResolved

Bug #12579: reproducibly_build_Tails_ISO_* Jenkins job are brokenResolved

Bug #12595: Not enough space in /var/lib/jenkins on isobuildersResolved

Bug #12606: Better balance our isobuilders' I/O load over all available SSDsResolved

Feature #12608: Analyze what's still not reproducible on current testing branchResolved

Feature #12616: Document our vagrant based build setup in JenkinsResolved

Bug #12619: /usr/share/doc/tails/website/torrents/rss.html is not reproducibleResolved

Bug #12620: /usr/local/lib/tor-browser/omni.ja embeds build timestampResolved

Feature #12625: Make Ikiwiki resize images deterministicallyResolvedintrigeri

Feature #12626: Report back to the reproducible builds community about how we did itResolvedintrigeri

Feature #12628: Draft a "user" (aka. RM) story for the reproducible release processDuplicate

Bug #12629: Document reproducible release processIn Progressanonym

Feature #12630: Document how users can verify a reproducibly built ISO/IUKResolved

Feature #12633: Lower the workload caused by reproducible builds Jenkins jobsIn Progressbertagaz

Bug #12637: Deploy rake libvirt volumes clean up task on all Jenkins build jobsResolved

Bug #12641: Comment changes in POT files make ISO builds non-reproducibleResolved

Feature #12654: Introduce more variations in our reproducibility CI testsConfirmedintrigeri

Bug #12681: reproducibly_build_Tails_ISO_* Jenkins jobs are buggy when building from a tagResolved

Feature #12715: Decide what builds we will try to reproduce in JenkinsResolved

Bug #12725: Sort out the apt-snapshots-disk partition situation on apt.lizardResolved

Bug #12726: There should be a date on the notes in the News section of the websiteResolved

Bug #12735: live/initrd.img not reproducible in some environmentsResolved

Bug #12736: live/vmlinuz not reproducible in some environmentsDuplicatelamby

Bug #12738: Remove gconfResolved

Bug #12737: utils/ not reproducible in some environmentsDuplicate

Bug #12739: Metadata for directories inside the squashfs not reproducible in some environmentsRejected

Bug #12740: Various .cache files not reproducible in some environmentsResolved

Bug #12909: /var/cache/cracklib/src-dicts not reproducibleResolved

Bug #13439: mimeinfo.cache not reproducibleResolved

Bug #13440: GTK immodules.cache not reproducibleResolved

Bug #13441: giomodule.cache not reproducibleResolved

Bug #13442: gdk-pixbuf's loaders.cache not reproducibleResolved

Bug #12741: /lib/modules/*/modules.* not reproducible in some environmentsResolved

Feature #13436: Have Jenkins jobs that reproduce ISOs when a branch ticket is Ready for QAIn Progressbertagaz

Bug #13504: Rebase our custom squashfs-tools package on 1:4.3-3+deb9u1Rejected

Bug #13531: Use ikiwiki 3.20161219+, againDuplicate

Bug #13623: Executable bits of /etc/hostname not set deterministicallyResolved

Feature #13624: Analyze results of Tails 3.1 call for reproductionResolved

Feature #14512: Send second email call to test reproducibility of Tails 3.2alpha1Resolved

Feature #14520: Prepare & publish a blog post about testing Tails ISO reproducibilityResolvedintrigeri

Feature #14607: Analyze results of Tails 3.2~alpha1 call for reproductionResolved

Bug #14729: Fix gdk-pixbuf vulnerability (CVE-2017-2862)Resolved

Feature #14756: Drop update-ca-certificates.serviceIn Progressanonym

Feature #14757: Final report for SponsorT 2016Confirmedu

Bug #14767: ikiwiki does not order news items deterministicallyResolved

Feature #14875: Use conditional-buildstep plugin in reproducibly_build_Tails_ISO jobsConfirmedbertagaz

Bug #14924: reproducibly_build_Tails_ISO_stable Jenkins job always failsResolvedintrigeri

Bug #14933: stable branch is not reproducible: differences in some .fa.html website filesConfirmedanonym

Bug #14944: jenkins-data-disk is running out of diskspaceResolvedintrigeri

Bug #14946: Topic branches that lag behind their base branch don't build reproducibly with mergebasebranch build optionResolved


Related issues

Related to Tails - Feature #8511: Have all Debian packages we use build in a deterministic way In Progress 01/01/2015
Related to Tails - Feature #7100: Decide what to do with machine-id Confirmed 04/16/2014
Blocked by Tails - Bug #8125: Self-host the Tor Browser tarballs we need Resolved 10/15/2014
Blocked by Tails - Bug #9416: Stop shipping ssl-cert-snakeoil in the ISO Resolved 05/17/2015
Blocked by Tails - Bug #9419: eatmydata is not being used in the build chroot Resolved 05/17/2015
Blocks Tails - Feature #11990: In 2018, try reproducing an ISO that was released in 2017 Confirmed 11/22/2016
Blocked by Tails - Bug #11273: clean up libdvd-pkg build files Resolved 03/21/2016

Associated revisions

Revision 4dd6fe78 (diff)
Added by intrigeri about 1 year ago

Export last changelog entry's timestamp as SOURCE_DATE_EPOCH (refs: #5630).

Revision d52e4e9e (diff)
Added by intrigeri about 1 year ago

Don't include /var/cache/ldconfig/aux-cache in the ISO (refs: #5630).

It causes reproducibility issues, and is not needed strictly speaking.

Revision bfbfecc4 (diff)
Added by intrigeri about 1 year ago

Don't include monkeysphere private key in the ISO (refs: #5630).

It causes reproducibility issues and should not be shared among all
Tails systems. Thankfully it is only useful when using monkeysphere
to authenticate users connecting to the Tails system, and we don't
ship SSHd, so 1. shipping that key previously was not a security issue;
2. we don't have to generate this key at boot time.

Revision a8f5383f
Added by intrigeri about 1 year ago

Merge remote-tracking branch 'lamby/regenerate-fontconfig' into feature/5630-deterministic-builds

refs: #5630

Revision 70efd9dc (diff)
Added by intrigeri about 1 year ago

Don't ship /etc/console-setup/cached_setup_keyboard.sh in the ISO.

refs: #5630

It's useless since it refers to a file in /tmp that won't exist anyway,
and it causes reproducibility issues.

Revision d86ef16d (diff)
Added by intrigeri about 1 year ago

Don't ship /var/lib/monkeysphere/authentication/ in the ISO (refs: #5630).

Same rationale as in bfbfecc4a3e0979c862525fa81abe431c9bdecfe.

Revision c5868e09 (diff)
Added by intrigeri about 1 year ago

Don't ship /root/.gnupg/trustdb.gpg in the ISO: it's not needed and causes reproducibility issues (refs: #5630).

Revision b48826f3 (diff)
Added by intrigeri about 1 year ago

Pretend that tails-keyring.gpg is created at $SOURCE_DATE_EPOCH, to make its content deterministic (refs: #5630).

Revision 0d5d4d42 (diff)
Added by intrigeri about 1 year ago

Empty /etc/machine-id instead of deleting it (Closes: #11970, refs: #5630, refs: #7100).

Revision 0f56eea5 (diff)
Added by intrigeri about 1 year ago

Ensure the SquashFS creation time is $SOURCE_DATE_EPOCH (refs: #5630).

Revision 64dcfa72 (diff)
Added by intrigeri about 1 year ago

Drop mtime clamping: lb_chroot_reproducible does that for us already (refs: #5630).

Revision 9e2d1145 (diff)
Added by intrigeri about 1 year ago

Don't pass -mkfs-fixed-time to mksquashfs, instead rely on having it honor $SOURCE_DATE_EPOCH (refs: #5630).

This reverts commit 0f56eea534f6cde1bee912cc51ceeb435790df80.

Revision 13dee6f9 (diff)
Added by intrigeri 9 months ago

Don't include root's gpg-agent socket files in the SquashFS (refs: #5630).

Revision f9675a11 (diff)
Added by intrigeri 9 months ago

Normalize timestamps of files in config/chroot_local-includes before building.

refs: #5630

Revision 3cae08ce (diff)
Added by intrigeri 9 months ago

Ensure /etc/resolv.conf is owned by root:root in the SquashFS.

lb_chroot_resolv will "cp -a" it from the source tree, so it inherits its
ownership from the whoever cloned the Git repository. This has two problems.
First, this results in unsafe permissions on this file (e.g. a Vagrant build
results in the 'amnesia' user having write access to it). Second, building with
a different user results in a non-deterministic SquashFS.

refs: #5630

Revision 6fcb2323 (diff)
Added by intrigeri 9 months ago

Pass a fixed MBR ID to isohybrid (refs: #5630).

Otherwise, a random one is used, that makes the build unreproducible even when
the content of the ISO filesystem matches.

Revision dbc479a8 (diff)
Added by intrigeri 9 months ago

Reproducible builds post-processing: don't try deleting /etc/ssl/certs/java/cacerts that's not shipped anymore (refs: #5630).

Revision bd43b818 (diff)
Added by intrigeri 9 months ago

Reproducible builds post-processing: don't try deleting /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/classes.jsa, that's not shipped anymore (refs: #5630).

Revision ababfdf9 (diff)
Added by intrigeri 7 months ago

Revert "Ship the fontconfig cache in the ISO again, after making it reproducible"

This reverts commit d7dae73111ea2521ff98010b5ebd62a11e5cb3ef.

refs: #5630

History

#1 Updated by intrigeri over 4 years ago

  • Type of work set to Code

Type of work: Code

#2 Updated by BitingBird over 3 years ago

  • Description updated (diff)
  • Category set to Build system
  • Starter set to No

#3 Updated by intrigeri about 3 years ago

  • Blocked by Bug #8125: Self-host the Tor Browser tarballs we need added

#4 Updated by intrigeri almost 3 years ago

  • Related to Feature #8511: Have all Debian packages we use build in a deterministic way added

#5 Updated by intrigeri over 2 years ago

  • Feature Branch set to feature/5630-deterministic-builds

#6 Updated by intrigeri over 2 years ago

Requires our tails/debian-old-2.0+faketime branch of live-build, otherwise faketime has no effect on the commands run in the chroot set up by live-build.

#7 Updated by intrigeri over 2 years ago

  • Blocked by Bug #9416: Stop shipping ssl-cert-snakeoil in the ISO added

#8 Updated by intrigeri over 2 years ago

  • Blocked by Bug #9419: eatmydata is not being used in the build chroot added

#9 Updated by sajolida over 2 years ago

  • Description updated (diff)
  • Assignee set to intrigeri
  • Target version set to 2017

#10 Updated by intrigeri about 2 years ago

  • Priority changed from Low to Normal

(It's on our roadmap now.)

#11 Updated by intrigeri about 2 years ago

  • Description updated (diff)

#12 Updated by intrigeri almost 2 years ago

  • Subject changed from Deterministic builds to Reproducible builds
  • Blueprint set to https://tails.boum.org/blueprint/reproducible_builds/

#13 Updated by intrigeri almost 2 years ago

  • Description updated (diff)

#14 Updated by BitingBird over 1 year ago

  • Status changed from Confirmed to In Progress

#15 Updated by intrigeri over 1 year ago

  • Status changed from In Progress to Confirmed
  • Assignee changed from intrigeri to anonym

#17 Updated by intrigeri about 1 year ago

  • Description updated (diff)

#18 Updated by intrigeri about 1 year ago

  • Related to Feature #7100: Decide what to do with machine-id added

#19 Updated by intrigeri about 1 year ago

  • Description updated (diff)

#20 Updated by intrigeri about 1 year ago

  • Blocks Feature #11990: In 2018, try reproducing an ISO that was released in 2017 added

#21 Updated by intrigeri 9 months ago

  • Blocked by Bug #11273: clean up libdvd-pkg build files added

#22 Updated by lamby 9 months ago

fontconfig issues should be resolved with: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=857892

#23 Updated by intrigeri 7 months ago

  • Status changed from Confirmed to In Progress

#24 Updated by intrigeri 7 months ago

#25 Updated by intrigeri 6 months ago

  • Target version changed from 2017 to Tails_3.2
  • Feature Branch deleted (feature/5630-deterministic-builds)

(

  • This branch has nothing interesting now that we generate the fontconfig cache in a reproducible manner. I've renamed it to wip/feature/5630-deterministic-builds so it doesn't eat precious cycles on our CI infra. And anyway the only non-merge commit it has on top of testing is a trivial revert.
  • Setting a target version that's before the sponsor deadline.

)

#26 Updated by anonym 3 months ago

  • Target version changed from Tails_3.2 to Tails_3.3

It seems Tails 3.2 is reproducible! Woo!

But we have some testing, documentation and communication tasks remaining so => postponed.

#27 Updated by intrigeri about 1 month ago

(Some subtasks are meant to be done after the end of the contract.)

#28 Updated by anonym 26 days ago

  • Target version changed from Tails_3.3 to Tails_3.5

Also available in: Atom PDF