Project

General

Profile

Feature #11971

Feature #5630: Reproducible builds

Consider migrating some of /lib/live/config/* to systemd unit files

Added by intrigeri almost 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
-
Target version:
Start date:
11/20/2016
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
bugfix/11971-fontconfig-cache-in-iso
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

While working on reproducible builds we're adding quite some stuff in there, and some of it might be slow. If this affects boot time significantly (especially on slower hardware), we should migrate some of those bits to systemd unit files, so that they can be run in parallel.


Related issues

Related to Tails - Feature #12318: Have our test suite track detailed boot-up performance Confirmed 03/11/2017
Related to Tails - Feature #11983: Check if the test suite has more failures on the reproducible ISO Resolved 11/21/2016
Related to Tails - Bug #12567: fontconfig cache is not generated reproducibly even with patch from Debian#857892 Resolved 05/19/2017

Associated revisions

Revision 042b624e (diff)
Added by intrigeri over 1 year ago

Update /etc/ssl/certs with a dedicated service, not as part of live-config.service.

I've seen this operation take 10-15 seconds during boot, so better have systemd
parallelize it with other tasks, than run sequentially by live-config.

refs: #11971

Revision c08029e3 (diff)
Added by u over 1 year ago

will-fix: #11971 Move fontconfig generation to systemd at boot time for a deterministic build.

Revision fd16a6e7 (diff)
Added by u over 1 year ago

will-fix: #11971 Move fontconfig generation to systemd at boot time for a deterministic build.

Revision 739d8b98 (diff)
Added by u over 1 year ago

will-fix: #11971 Delete this file which has been moved in commits fd16a6e and c08029e

Revision 53866d70 (diff)
Added by intrigeri over 1 year ago

Start update-ca-certificates.service during early boot (refs: #11971).

We were previously running it as part of live-config, which means it can run
early. Without this commit this service would block some of the boot too,
just later.

Revision 7c70f834
Added by intrigeri over 1 year ago

Merge remote-tracking branch '451f/feature/11971+fontconfig_to_systemd' into feature/5630-deterministic-builds (refs: #11971)

Revision 9065331b (diff)
Added by intrigeri over 1 year ago

Run tails-reconfigure-fontconfig.service with a read-only /etc, by replacing ProtectSystem=yes with ProtectSystem=full (refs: #11971).

Revision 221c3ca7 (diff)
Added by intrigeri over 1 year ago

Start tails-reconfigure-fontconfig.service and update-ca-certificates.service after network.target.

Currently they fail to start with "Main process exited, code=exited,
status=226/NAMESPACE". I think that PrivateNetwork=yes requires the network to
be set up in the first place.

refs: #11971

Revision a4d4faf8 (diff)
Added by intrigeri over 1 year ago

Start tails-reconfigure-fontconfig.service and update-ca-certificates.service after systemd-tmpfiles-setup.service.

I think that PrivateTmp=yes requires /tmp to have been set up already.

refs: #11971

Revision bdacff0e (diff)
Added by intrigeri over 1 year ago

Start tails-reconfigure-fontconfig.service and update-ca-certificates.service earlier.

… and accordingly drop PrivateNetwork.

refs: #11971

Revision dcdd11d6 (diff)
Added by intrigeri over 1 year ago

Pass -no-exports to mksquashfs, to same ~1MB on the ISO size.

We don't need to make our SquashFS exportable over NFS.

refs: #11971

Revision 0e1b399d (diff)
Added by intrigeri over 1 year ago

Enable the bugfix-11971-fontconfig-cache-in-iso APT overlay (refs: #11971).

Revision d7dae731 (diff)
Added by intrigeri over 1 year ago

Ship the fontconfig cache in the ISO again, after making it reproducible (refs: #11971).

See https://labs.riseup.net/code/issues/11971#note-22 for details.

Revision a829b391
Added by anonym over 1 year ago

Merge remote-tracking branch 'origin/bugfix/11971-fontconfig-cache-in-iso' into feature/stretch

Fix-committed: #11971, #12348

History

#1 Updated by intrigeri over 1 year ago

  • Related to Feature #12318: Have our test suite track detailed boot-up performance added

#2 Updated by intrigeri over 1 year ago

In a VM on a quite loaded system:

  • dpkg-reconfigure fontconfig took 27 seconds
  • Updating certificates in /etc/ssl/certs took 13 seconds

In practice, I think this added ~20 seconds to the total boot time after apparmor.service stopped blocking.

#3 Updated by u over 1 year ago

concerning config/2050-fontconfig

A solution might be to not have font cache at all. But I wonder if that would result in longer loading times for libreoffice for example.

#4 Updated by u over 1 year ago

I booted a test ISO and everything seems to work fine. I opened LibreOffice, a PDF, Inkscape and could successfully change and read different fonts as well as have a font list. However, I'm in no position to judge if booting is faster than usual nor if running a program using system fonts is slower than usual.

This has been done in tails:tails/feature/11971+fontconfig.
ISO is here: https://nightly.tails.boum.org/build_Tails_ISO_feature-11971-fontconfig/

#5 Updated by anonym over 1 year ago

u wrote:

This has been done in tails:tails/feature/11971+fontconfig.
ISO is here: https://nightly.tails.boum.org/build_Tails_ISO_feature-11971-fontconfig/

I see only unrelated failures in the automated test suite run: https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-11971-fontconfig/1/

#6 Updated by intrigeri over 1 year ago

I'll look into the CA cert updates, while waiting for mksquashfs to crunch numbers. Likely it's a good candidate to move to a dedicated systemd unit file.

#7 Updated by u over 1 year ago

Ok, i've tested manually in a VM, comparing the latest feature/56300 with the fontconfig ISO mentioned in my previous comment. The VM had 2GB RAM and used 3 cores of an i5.

with fontconfig cache generated at boot time
  • boot until Greeter started = 78 secs
  • login from greeter = 10 secs
  • start libreoffice once = 6.5 secs
  • start libreoffice second time = 2 secs
  • start inkscape = 6.5 secs
  • memory used = 596MB
  • memory used while running libreoffice = 675MB
  • top (after tor has started and tails-iuk and tails-upgrader were run:)
    KiB Mem : 2052372 total, 297868 free, 595472 used, 1159032 buff/cache
without fontconfig cache
  • boot until Greeter started = 68 secs
  • login from greeter = 21 secs
  • start libreoffice once = 6.5 secs
  • start libreoffice second time = 1.8 secs
  • start inkscape = 6.5 secs
  • memory used = 523MB
  • memory used while running libreoffice = 594MB
  • top (after tor has started and tails-iuk and tails-upgrader were run:)
    KiB Mem : 2052372 total, 379516 free, 513800 used, 1159056 buff/cache
tails 3.0 beta2 fontconfig cache in the ISO
  • boot until Greeter started = 67 secs
  • login from greeter = 16 secs
  • start libreoffice once = 15 secs
  • start libreoffice second time = 1.8 secs
  • start inkscape = 9 secs
  • memory used = 580MB
  • memory used while running libreoffice = 652MB
  • top (after tor has started and tails-iuk and tails-upgrader were run:)
    KiB Mem : 2052372 total, 579476 free, 580344 used, 892552 buff/cache

#8 Updated by intrigeri over 1 year ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

#9 Updated by intrigeri over 1 year ago

  • Assignee changed from intrigeri to u
  • % Done changed from 10 to 20

Done (and pushed) my bits about update-ca-certificates, sending to u's plate wrt. the fontconfig part :)

#10 Updated by u over 1 year ago

Same test again on bare metal (i3 processor, 16 Gigs of RAM):

with fontconfig cache generated at boot time
  • boot until Greeter started = 84 secs
  • login from greeter = 10 secs
  • start libreoffice once = 7 secs
  • memory used = 551MB
  • memory used while running libreoffice = 617MB
  • top: KiB Mem : 16349932 total, 14860230 free 551400 used, 1195460 buff/cache
without fontconfig cache
  • boot until Greeter started = 73 secs
  • login from greeter = 19 secs
  • start libreoffice once = 6.5 secs
  • memory used = 591MB
  • memory used while running libreoffice = 653MB
  • top: KiB Mem : 16349932 total, 14563116 free 590912 used, 1195840 buff/cache
    After running LibreOffice the last value changes to 1359180 buff/cache.
tails 3.0 beta2 fontconfig cache in the ISO
  • boot until Greeter started = 85 secs
  • login from greeter = 18 secs
  • start libreoffice once = 20 secs
  • memory used = 566MB
  • memory used while running libreoffice = 615MB
  • top: KiB Mem : 16349936 total, 14876420 free 549528 used, 925292 buff/cache

#11 Updated by u over 1 year ago

Basically, it looks to me like boot until the greeter is faster with fontconfig not generated at build time. However, the login to Xserver is longer.
Starting up libreoffice seems unaffected between fontcache and nofontcache.
What seems weird is that more memory is used on baremetal when fontcache is not generated at build and less memory is used on a VM.

#12 Updated by u over 1 year ago

  • Assignee changed from u to intrigeri
  • QA Check set to Ready for QA
  • Feature Branch set to 451f:config/chroot_local-includes/lib/live/config/

We should build this branch and see if it works out well please.

#13 Updated by u over 1 year ago

  • Feature Branch changed from 451f:config/chroot_local-includes/lib/live/config/ to 451f:tails/config/chroot_local-includes/lib/live/config/

#14 Updated by intrigeri over 1 year ago

  • Assignee changed from intrigeri to u
  • % Done changed from 20 to 50
  • Feature Branch changed from 451f:tails/config/chroot_local-includes/lib/live/config/ to 451f:feature/11971+fontconfig_to_systemd

Code review passes! I'd like to do some systemd magics to verify it is ordered as expected, so please reassign to me (unless you prefer we do it together!) once you've successfully tested an ISO built from this branch.

#15 Updated by u over 1 year ago

Using a build from the latest branch, which does invoke fontconfig over systemd, I think we've solved our problem:

Test on bare metal (i3 processor, 16 Gigs of RAM):

  • boot until Greeter started = 68 secs
  • login from greeter = 11 secs
  • start libreoffice once = 7 secs
  • memory used = 568MB
  • memory used while running libreoffice = 638MB
  • top: KiB Mem : 16349936 total, 14584664 free 568484 used, 1195472 buff/cache

#16 Updated by u over 1 year ago

  • Assignee changed from u to intrigeri

#17 Updated by intrigeri over 1 year ago

Yay!

I'll fetch an ISO and will give it a try.

#18 Updated by intrigeri over 1 year ago

  • Assignee changed from intrigeri to u
  • % Done changed from 50 to 70

Merged and then we've improved it. Last step is to benchmark current feature/5630-deterministic-builds vs. 3.0~beta2 on bare metal, to ensure there's no severe regression in the end.

#19 Updated by intrigeri over 1 year ago

  • QA Check deleted (Ready for QA)

Hold on, we might be able to move the fontconfig cache generation back to ISO build time: https://bugs.debian.org/857892. I'm in favour of trying this first, i.e.:

  1. fork a topic branch dedicated to test this, based on feature/5630-deterministic-builds
  2. build a custom package based on the one from Stretch, with Chris' patch applied, and upload it to the suite corresponding to the new topic branch in our custom APT repo
  3. enable this suite in APT_overlays.d
  4. drop the fontconfig systemd unit file and drop the fontconfig cache deletion at ISO build time, i.e. go back to how we did things in 2.x
  5. build two ISOs and compare
  6. if the build is reproducible, then benchmark this new ISO against feature/5630-deterministic-builds (that has the fontconfig cache generation done at boot time) and decide what's best; and if we want to keep generating that cache at boot time, then consider applying https://cgit.freedesktop.org/fontconfig/commit/?id=7a6622f25cdfab5ab775324bef1833b67109801b to make it faster

What do you think? Wanna try this?

#20 Updated by intrigeri over 1 year ago

  • Target version set to Tails_2.12
  • Feature Branch deleted (451f:feature/11971+fontconfig_to_systemd)
  • Type of work changed from Test to Code

Ulrike agreed to do it. Real deadline: have this finalized for the 3.0~rcN we'll publish mid-May. Please reassign to me (with a target version = 3.0 and priority = Elevated) if this is not done by then :)

#21 Updated by u over 1 year ago

  • Target version changed from Tails_2.12 to Tails_3.0

#22 Updated by intrigeri over 1 year ago

  • Priority changed from Normal to Elevated

I suspect (#11983#note-14) that the current state of our fontconfig tweaks breaks stuff, so marking as a blocker for 3.0. The deadline set above still holds, but since this somewhat blocks #11983 now, I certainly wouldn't mind if it was done earlier :)

#23 Updated by intrigeri over 1 year ago

  • Blocks Feature #11983: Check if the test suite has more failures on the reproducible ISO added

#24 Updated by intrigeri over 1 year ago

  • Target version changed from Tails_3.0 to Tails_3.0~rc1

#25 Updated by u over 1 year ago

intrigeri wrote:

  1. build a custom package based on the one from Stretch, with Chris' patch applied, and upload it to the suite corresponding to the new topic branch in our custom APT repo

Why wouldn't we use the package from experimental instead? seems that this one contains the patch already.

#26 Updated by intrigeri over 1 year ago

why wouldn't we use the package from experimental instead? seems that this one contains the patch already.

  • If we do that, then we have to track fontconfig 2.12+ for the entire Tails 3.x series; this will probably work smoothly for a while, but at some point I bet the packages won't be installable on Stretch anymore (e.g. they'll get a dependency on a newer libc6 or something) and then we'll have to maintain backports until Tails 4.x, which will likely require much more work than maintaining a tiny delta against the Stretch package.
  • fontconfig has been poorly maintained in Debian for ages; the last maintainer upload is 3 years old; the upload to experimental was a NMU done by someone who had never uploaded the package before, and I have no idea what's their plan to transition unstable to 2.12 / to maintain the package in experimental; I'd rather not rely on this upload and its future maintenance.
  • libfontconfig1-dev has many reverse-build-dependencies, so a proper transition (with binNMUs) may be required to have other packages work fine with libfontconfig1 2.12+.
  • It's very unlikely that fontconfig 2.12 has been tested much on Debian systems; even Ubuntu doesn't ship 2.12 yet (they've diverged from Debian many years ago, and apparently they nowadays ship some snapshot taken somewhere between fontconfig 2.11 and 2.12). So the impact of upgrading from 2.11 to this new upstream release is unknown: it may very well break stuff. I'd rather not Tails users be the ones who discover such breakage first.
  • There's no security support for packages in experimental, so if we use them we (for some value of "we" :) needs to take responsibility for tracking their security status. One security upload was done during the Jessie lifetime already, so this is not a theoretical concern.

⇒ all in all, pulling these packages from experimental saves a little bit of work up-front, but feels risky (QA-wise) and adds lots more work on our shoulders for the next 2 years. I don't think the tiny short-term benefits are worth the short and long-term drawbacks.

#27 Updated by u over 1 year ago

intrigeri wrote:

Hold on, we might be able to move the fontconfig cache generation back to ISO build time: https://bugs.debian.org/857892. I'm in favour of trying this first, i.e.:

  1. fork a topic branch dedicated to test this, based on feature/5630-deterministic-builds
  2. build a custom package based on the one from Stretch, with Chris' patch applied, and upload it to the suite corresponding to the new topic branch in our custom APT repo

Interestingly, the line deleted by Chris's patch is not present in the version he talks about?
I double checked:

wget http://httpredir.debian.org/debian/pool/main/f/fontconfig/fontconfig_2.11.0-6.7.dsc
then see in src/fpat.c
search for "memset (p, 0, sizeof (FcPattern));"
this should be located right at the beginning of the file, line 6.
But it's not.

#28 Updated by intrigeri over 1 year ago

Interestingly, the line deleted by Chris's patch is not present in the version he talks about?

Chris' patch adds a line, it doesn't delete any.

#29 Updated by u over 1 year ago

intrigeri wrote:

Interestingly, the line deleted by Chris's patch is not present in the version he talks about?

Chris' patch adds a line, it doesn't delete any.

Haha, I had no idea how tired my eyes are sometimes.

This is probably due to the fact that the fontconfig package does not use version control (!§$%@!) and so this got mixed up in quilt, making me think that I removed the line, but it was simply the other way round.

#30 Updated by intrigeri over 1 year ago

What's your ETA? If later than Wednesday night, I'll probably reassign to me (as said a couple months ago) as this blocks merging the reproducible builds branch, which I want to merge in time for 3.0~rc1.

#31 Updated by intrigeri over 1 year ago

  • Assignee changed from u to intrigeri

intrigeri wrote:

What's your ETA? If later than Wednesday night, I'll probably reassign to me (as said a couple months ago) as this blocks merging the reproducible builds branch, which I want to merge in time for 3.0~rc1.

Taking over, hoping to complete this fast enough to fix this problem and then work on #11983 and #12348 in time for 3.0~rc1 that we'll freeze at some point tomorrow. I didn't find any published WIP on this front (I've looked in the official Git repo, in u's own one, and in our custom APT repo), so I'll start from scratch. Still, u: if you read this and want to share what you already did or share some of the next steps, let's coordinate e.g. on XMPP :)

#32 Updated by intrigeri over 1 year ago

  • Feature Branch set to bugfix/11971-fontconfig-cache-in-iso

#33 Updated by intrigeri over 1 year ago

Patched fontconfig package uploaded to the bugfix-11971-fontconfig-cache-in-iso APT suite.

#34 Updated by intrigeri over 1 year ago

Prepared branch that includes the fontconfig cache, will built & test locally before pushing.

#35 Updated by intrigeri over 1 year ago

Fixed some reproducibility issues (initrd differs due to initramfs-tools in testing having superseded our patched package + forgot about forcecleanup). Building twice again to make sure the fontconfig cache itself doesn't introduce reproducibility problems.

#36 Updated by intrigeri over 1 year ago

  • Target version changed from Tails_3.0~rc1 to Tails_3.0

intrigeri wrote:

Hold on, we might be able to move the fontconfig cache generation back to ISO build time: https://bugs.debian.org/857892. I'm in favour of trying this first, i.e.:

  1. fork a topic branch dedicated to test this, based on feature/5630-deterministic-builds
  2. build a custom package based on the one from Stretch, with Chris' patch applied, and upload it to the suite corresponding to the new topic branch in our custom APT repo
  3. enable this suite in APT_overlays.d
  4. drop the fontconfig systemd unit file and drop the fontconfig cache deletion at ISO build time, i.e. go back to how we did things in 2.x
  5. build two ISOs and compare

Ouch, I see differences in /var/cache/fontconfig/ even with Chris' patch applied. Too late to investigate this further, so we'll have to do #11983 without this change.

#37 Updated by intrigeri over 1 year ago

  • Blocks deleted (Feature #11983: Check if the test suite has more failures on the reproducible ISO)

#38 Updated by intrigeri over 1 year ago

  • Related to Feature #11983: Check if the test suite has more failures on the reproducible ISO added

#39 Updated by intrigeri over 1 year ago

Actually, anonym and I want this branch in 3.0~rc1, even if it means that release won't be reproducible: robustness feels more important than reproducibility at this point. So we'll evaluate this branch with the test suite, merge it, and I'll file another ticket about making the fontconfig cache reproducible, that anonym or I will try to get help from Chris about.

#40 Updated by intrigeri over 1 year ago

  • Target version changed from Tails_3.0 to Tails_3.0~rc1

#41 Updated by intrigeri over 1 year ago

  • Related to Bug #12567: fontconfig cache is not generated reproducibly even with patch from Debian#857892 added

#42 Updated by u over 1 year ago

intrigeri wrote:

What's your ETA? If later than Wednesday night, I'll probably reassign to me (as said a couple months ago) as this blocks merging the reproducible builds branch, which I want to merge in time for 3.0~rc1.

I had only had time to start building the package, but I did not pursue the initial plan to build and compare the ISOs. Fine with me that you took over and sorry to see that the patch does not function as intended.

#43 Updated by intrigeri over 1 year ago

  • Assignee changed from intrigeri to anonym
  • QA Check set to Ready for QA

I've run locally a large part of the test suite (including fragile tests) on this branch today and am killing it now during the Thunderbird tests, as I need to context switch and test other branches now. I've seen only fragile tests fail so far. Please review'n'merge :) #12567 tracks the next steps on this topic, so let's close this ticket that we have already used and reused for follow-ups to the initial issue it was about.

#44 Updated by anonym over 1 year ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 70 to 100
  • QA Check changed from Ready for QA to Pass

#45 Updated by intrigeri over 1 year ago

Merged into the branch for #5630 and reverted "Ship the fontconfig cache in the ISO again", to try and have that branch build ISOs reproducibly until #12567 is solved.

#46 Updated by intrigeri over 1 year ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF