Project

General

Profile

Bug #12567

Feature #5630: Reproducible builds

fontconfig cache is not generated reproducibly even with patch from Debian#857892

Added by intrigeri 4 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Build system
Target version:
Start date:
05/19/2017
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
bugfix/12567-fontconfig-fixup
Type of work:
Code
Blueprint:
Easy:
Affected tool:

Description

Noticed on #11971. I'll attach the actual files and diffoscope output and will check with lamby if he can handle this with the agreed upon budget before June 5.

The package we use is https://deb.tails.boum.org/pool/main/f/fontconfig/fontconfig_2.11.0-6.7.0tails1.dsc

12567.tar.bz2 (303 KB) intrigeri, 05/19/2017 11:20 AM

diff.txt View (7.3 KB) lamby, 05/26/2017 05:46 PM

12567-2.tar.gz (1.25 KB) anonym, 05/31/2017 10:40 PM

diff.txt View (7.31 KB) lamby, 06/02/2017 05:48 PM

12567-3.tar.bz2 (3.13 MB) anonym, 06/02/2017 10:57 PM

diff.txt View (7.29 KB) lamby, 06/03/2017 07:59 AM


Related issues

Related to Tails - Feature #11971: Consider migrating some of /lib/live/config/* to systemd unit files Resolved 11/20/2016

Associated revisions

Revision bcd61b18 (diff)
Added by anonym 4 months ago

Enable the bugfix-12567-fontconfig-fixup APT overlay.

bugfix-12567-fontconfig-fixup|main|amd64: fontconfig 2.11.0-6.7.0tails2
bugfix-12567-fontconfig-fixup|main|amd64: fontconfig-config 2.11.0-6.7.0tails2
bugfix-12567-fontconfig-fixup|main|amd64: libfontconfig1 2.11.0-6.7.0tails2
bugfix-12567-fontconfig-fixup|main|amd64: libfontconfig1-dbg 2.11.0-6.7.0tails2
bugfix-12567-fontconfig-fixup|main|amd64: libfontconfig1-dev 2.11.0-6.7.0tails2
bugfix-12567-fontconfig-fixup|main|i386: fontconfig-config 2.11.0-6.7.0tails2
bugfix-12567-fontconfig-fixup|main|source: fontconfig 2.11.0-6.7.0tails2

Will-fix: #12567

Revision 9ea44ce4
Added by anonym 4 months ago

Merge remote-tracking branch 'origin/bugfix/12567-fontconfig-fixup' into testing

Fix-committed: #12567

History

#1 Updated by intrigeri 4 months ago

  • Related to Feature #11971: Consider migrating some of /lib/live/config/* to systemd unit files added

#2 Updated by intrigeri 4 months ago

  • Description updated (diff)

#3 Updated by intrigeri 4 months ago

  • File 12567.tar.bz2 added
  • Assignee changed from intrigeri to lamby

Here are the 2 sets of /var/cache/fontconfig files + the diffoscope output.

Chris:

  • would you have time to work on this by June 5, and ideally earlier (in my dreamworld this would be solved by the end of May);
  • how are you doing wrt. the time budget we agreed on in March? if there's nothing left, how much time do you think we should allocate for this issue?

Sorry for the timing, I've just learnt about this issue today :(

#4 Updated by lamby 4 months ago

intrigeri wrote:

  • would you have time to work on this by June 5, and ideally earlier
    (in my dreamworld this would be solved by the end of May);

Yes of course. Almost certainly can do this before end of May. :)

  • how are you doing wrt. the time budget we agreed on in March?

Hm? We didn't agree on anything; were you expecting me to work
on stuff? :) Very happy to, you were just going to check budgets
and other internal stuff before ACKing that with me IIRC. Let
me know if that's wrong.

#5 Updated by lamby 4 months ago

Friendly ping on this re. May deadline? :)

#6 Updated by intrigeri 4 months ago

  • would you have time to work on this by June 5, and ideally earlier (in my dreamworld this would be solved by the end of May);

Yes of course. Almost certainly can do this before end of May. :)

Amazing!

  • how are you doing wrt. the time budget we agreed on in March?

Hm? We didn't agree on anything; were you expecting me to work on stuff? :) Very happy to, you were just going to check budgets and other internal stuff before ACKing that with me IIRC. Let me know if that's wrong.

Ouch, sorry! So, I've now checked and I've good news: we can have you spent quite some more time on this project if needed :)

So please go ahead, and get back to me if at any time you feel that this is going to take more than 3 full days of work (I don't expect it will, but who knows). Rationale: if it is that hard a problem, I want to reconsider the option (not shipping the fontconfig cache at all) we had picked initially.

Thanks :)

#7 Updated by lamby 4 months ago

intrigeri wrote:

Ouch, sorry! So, I've now checked and I've good news: we can have you
spent quite some more time on this project if needed :)

Cool, let's chat on this after I get this fixed for you.

> So please go ahead, and get back to me if at any time you feel that this

is going to take more than 3 full days of work

ACK, will do.

#8 Updated by lamby 4 months ago

The root cause is that fontconfig embeds the mtime of each font directory in a "checksum" member of a "_FcCache" struct. This is so that it can identify which cache files remain valid and/or require regeneration.

Unfortunately, we can't just replace the checksum value with SOURCE_DATE_EPOCH as it will mean the cache files will not be valid from fontconfig's PoV at runtime, defeating the entire point of generating them.

Took me a little while to track that down and then work through a number of solutions. I've attached something approximating the cleanest one with the smallest diff. Enjoy :)

FYI this also resulted in the following bug being filed: https://bugs.debian.org/863427

#9 Updated by intrigeri 4 months ago

  • Assignee changed from lamby to anonym
  • QA Check set to Ready for QA

Thanks!

anonym, I expect you'll evaluate this with diffoscope locally, and then once it's merged into testing https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_testing/ should help (although these jobs still have issues — #12579 — I've seen the one for the testing branch work fine once yesterday).

#10 Updated by lamby 4 months ago

If upstream don't like the —list-dirs patch, we could also propose a new "fc-list-dirs" command so it doesn't piggyback on existing functionality. The diff would be rather large, however.

#11 Updated by anonym 4 months ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from anonym to lamby
  • % Done changed from 0 to 20
  • QA Check changed from Ready for QA to Info Needed

I fail to build a package with your patch applied. I tried:

apt source fontconfig=2.11.0-6.7.0tails1
cd fontconfig-2.11.0
quilt import /tmp/10-fc-cache-list-dirs.patch
dch
pdebuild

but then I get:

[...]
patching file debian/fontconfig.postinst
Reversed (or previously applied) patch detected!  Skipping patch.
1 out of 1 hunk ignored
patching file fc-cache/fc-cache.1
patching file fc-cache/fc-cache.c
patching file fc-cache/fc-cache.sgml
dpkg-source: info: the patch has fuzz which is not allowed, or is malformed
dpkg-source: info: if patch '10-fc-cache-list-dirs.patch' is correctly applied by quilt, use 'quilt refresh' to update it
dpkg-source: error: LC_ALL=C patch -t -F 0 -N -p1 -u -V never -E -b -B .pc/10-fc-cache-list-dirs.patch/ --reject-file=- < fontconfig-2.11.0.orig.ydiTR1/debian/patches/10-fc-cache-list-dirs.patch gave error exit status 1

This is not a reversed patch AFAICT, so I really don't get what is going on. Any ideas?

#12 Updated by anonym 4 months ago

  • Assignee changed from lamby to intrigeri

IMHO this (i.e. me being a Debian packaging n00b) is a crappy thing to be blocked by lamby on, so if you can think of what I should look closer at, perhaps this can be resolved quickly without disturbing lamby. :)

#13 Updated by anonym 4 months ago

  • Assignee changed from intrigeri to lamby

#14 Updated by lamby 4 months ago

  • Assignee changed from lamby to anonym

Perhaps quilt import does not work when we are patching files under debian/? Try with regular "patch -p1 < diff.txt"?

#15 Updated by anonym 4 months ago

  • QA Check deleted (Info Needed)

lamby wrote:

Perhaps quilt import does not work when we are patching files under debian/? Try with regular "patch -p1 < diff.txt"?

Ah, now I get it, thanks a lot! So I extracted the debian/ parts of that patch and applied it directly. I've got a package built now that I will test.

#16 Updated by anonym 4 months ago

  • File 12567-2.tar.gz added
  • Assignee changed from anonym to lamby
  • QA Check set to Dev Needed

Sadly it seems your patch is not enough, but it greatly reduces the number of differences: now I get 5; I used to get 75. :)

See attached tarball for the offending files + diffoscope report.

#17 Updated by anonym 4 months ago

Also, what is your best ETA for working on this? Please provide this ASAP so we can evaluate our prognosis for getting this into the 3.0 release that could happen as early as the June 13.

#18 Updated by lamby 4 months ago

  • Assignee changed from lamby to anonym

Thanks fo the update! Two things:

a) Can you let me know how you performed the test, exactly?

b) I can probably look at this this evening, otherwise tomorrow.

#19 Updated by intrigeri 4 months ago

  • Feature Branch set to bugfix/12567-fontconfig-fixup

I'm adding a CI reproducibility test for the topic branch. anonym, please point lamby to the artifacts (on nightly.t.b.o) once it has run once :)

#20 Updated by intrigeri 4 months ago

  • Assignee changed from anonym to lamby

lamby wrote:

a) Can you let me know how you performed the test, exactly?

https://nightly.tails.boum.org/reproducibly_build_Tails_ISO_bugfix-12567-fontconfig-fixup/builds/2017-06-01_06-48-13/archive/build-artifacts/ has the build log and diffoscope output. Don't hesitate (re-)asking anonym if that's not enough info :)

#21 Updated by lamby 4 months ago

  • File diff.txt View added
  • Assignee changed from lamby to anonym

#22 Updated by intrigeri 4 months ago

  • QA Check changed from Dev Needed to Ready for QA

Excellent, thanks :)

#23 Updated by anonym 4 months ago

  • File 12567-3.tar.bz2 added
  • Assignee changed from anonym to lamby
  • QA Check changed from Ready for QA to Dev Needed

I still see the "same" 5 differences after building new packages with that one-line fixup to debian/fontconfig.postinst applied. :/

See the attached tarball for the full fontconfig cache + diffoscope report.

An ETA would be appreciated again!

For the record, I started a job for this on Jenkins which should be done within two hours and hopefully reproduces (hah!) the same differences: https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_bugfix-12567-fontconfig-fixup/3/

#24 Updated by lamby 4 months ago

  • File diff.txt View added
  • Assignee changed from lamby to anonym

anonym wrote:

I still see the "same" 5 differences after building new packages with that one-line fixup to debian/fontconfig.postinst applied. :/

Hah, so I woke up in the night thinking that I missed something about deferencing the symlinks...

Alas, no time to test this this morning (but can probably find time later) but I thiiiink the attached should fix it.

#25 Updated by lamby 4 months ago

It may also need an "-L" argument to find if that does not work. I'm only rushing here as you mention an ETA and there is a release... - this is not how I usually like to send over work!

Either way let me know and I will look later. :)

#26 Updated by anonym 4 months ago

lamby wrote:

anonym wrote:

I still see the "same" 5 differences after building new packages with that one-line fixup to debian/fontconfig.postinst applied. :/

Hah, so I woke up in the night thinking that I missed something about deferencing the symlinks...

A common source of nightmares indeed! :)

Alas, no time to test this this morning (but can probably find time later) but I thiiiink the attached should fix it.

I'm perfectly happy testing whatever crazy, unconfirmed stuff you can come up with. :)

It may also need an "-L" argument to find if that does not work.

Ack, I'll try that if the last patch isn't enough.

I'm only rushing here as you mention an ETA and there is a release... - this is not how I usually like to send over work!

This is perfect for me at this point, and the rush is my fault for not picking up this thread in the reproducibility effort sooner.

#27 Updated by anonym 4 months ago

  • Status changed from In Progress to Fix committed
  • % Done changed from 20 to 100

#28 Updated by anonym 4 months ago

  • Assignee changed from anonym to lamby
  • QA Check changed from Dev Needed to Info Needed

Jenkins managed to reproduce a pair of ISOs! https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_bugfix-12567-fontconfig-fixup/4/

I've also done two reproductions locally, so I think we're good! Merged!

Lamby, are you willing to, and will your time budget allow you to do the upstreaming part (reassign back to me when answering)? If so, all that's needed is your latest patch (so forget about the find -L idea).

#29 Updated by lamby 4 months ago

  • Assignee changed from lamby to anonym

I've also done two reproductions locally, so I think we're good! Merged!

Wooo! Gotta love the power of stepping away from the computer to come up with the solution, even if it's a "oh crap!".

are you willing to, and will your time budget allow you to do the upstreaming part

Indeed I am. Will be a pleasure...

the rush is my fault for not picking up this thread in the reproducibility
effort sooner.

No problem. Is there anything I could have done better mind you? I mean, apart from do a full test Tails build (where the offending poppler-data (!) package would have been installed and flagged..).

[Assigning back as requested]

#31 Updated by anonym 4 months ago

  • Assignee changed from anonym to lamby

For the record, we're currently having a Tails reprodicible builds (remote-)party (#12608#note-10) and it is looking really good! We've reproducibly built the same Tails on 5 different (real, hardware-wise) systems! And for extra robustness we've faked the system time so it appears we built it a month in the future, and forced different QEMU CPU models and machine types. It Just Works™! Congrats to us! :)

lamby wrote:

Forwarded upstream here: https://lists.freedesktop.org/archives/fontconfig/2017-June/thread.html (when it regenerates anyway...)

I still don't see it. Did it bounce?

Debian bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082

Thanks!

#32 Updated by lamby 4 months ago

anonym wrote:

Forwarded upstream here: https://lists.freedesktop.org/archives/fontconfig/2017-June/thread.html (when it regenerates anyway...)

I still don't see it. Did it bounce?

Odd, resent. Will fix and followup etc. until it hits the list.

#33 Updated by lamby 4 months ago

lamby wrote:

Odd, resent. Will fix and followup etc. until it hits the list.

Done: https://lists.freedesktop.org/archives/fontconfig/2017-June/005948.html

#34 Updated by anonym 3 months ago

  • Assignee deleted (lamby)
  • QA Check changed from Info Needed to Pass

Excellent! Then there are no more loose ends here!

#35 Updated by intrigeri 3 months ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF