Project

General

Profile

Bug #15400

Python files installed in /usr/local/lib break reproducibility

Added by bertagaz 7 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
-
Target version:
Start date:
03/13/2018
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
bugfix/15400-python-lib-breaks-reproducibility
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

Since #14680 and the installation of the Tails pythonlib in the ISO, it's not possible to reproduce the ISO anymore.

See https://nightly.tails.boum.org/reproducibly_build_Tails_ISO_stable/builds/193/archive/build-artifacts/diffoscope.html for more details.


Related issues

Related to Tails - Bug #15421: Improve the implementation of the #15400 bugfix Resolved 03/16/2018
Blocks Tails - Bug #15417: Upgrade Tor Browser to 7.5.2 Resolved 03/16/2018

Associated revisions

Revision f9cf01b9 (diff)
Added by bertagaz 7 months ago

Add reproducibility issue to the 3.6 release note.

Refs: #15400

Revision 6dbd9f55 (diff)
Added by intrigeri 7 months ago

Release Manager: make some expectations explicit.

The goal is to avoid issues such as #15400: Jenkins had been telling us for
48 hours that the testing branch did not build reproducibly anymore,
but the release manager missed that, built a tentative 3.6 ISO and decided
to release it as-is.

refs: #15400

Revision 85d42412 (diff)
Added by segfault 7 months ago

Strip dates from Tailslib Python package

Will-fix: #15400

Revision 158b83c2
Added by intrigeri 7 months ago

Merge branch 'bugfix/15400-python-lib-breaks-reproducibility' into stable (Fix-committed: #15400)

Revision d5fd1ef5
Added by bertagaz 5 months ago

Merge remote-tracking branch 'origin/bug/15400-python-lib-breaks-reproducibility' into stable

Fix-committed: #15400

History

#1 Updated by bertagaz 7 months ago

  • Status changed from Confirmed to In Progress

#2 Updated by intrigeri 7 months ago

bertagaz wrote:

Since #14680 and the installation of the Tails pythonlib in the ISO, it's not possible to reproduce the ISO anymore.

I suspect you mean something else than #14680.

It would be nice to have some kind of blame-free post-mortem to understand why this problem was not identified early enough in this dev cycle, as in: best case, before the faulty branch was merged; or worst case, before the 3.6 ISO was built. Quite possibly it's not clear who is responsible to notice such problems and ensure they're addressed. I would first like to hear bertagaz' take on it because I guess he's the one who has followed the closest what happened in the 48 hours before the release, after this problem has been diagnosed by our CI. I'll try to start a discussion about this on a more suitable place.

#3 Updated by bertagaz 7 months ago

intrigeri wrote:

I suspect you mean something else than #14680.

right, sorry, I meant #15370 probably, or #11753 (if testing how python apps behave was part of this ticket).

It would be nice to have some kind of blame-free post-mortem to understand why this problem was not identified early enough in this dev cycle, as in: best case, before the faulty branch was merged; or worst case, before the 3.6 ISO was built. Quite possibly it's not clear who is responsible to notice such problems and ensure they're addressed. I would first like to hear bertagaz' take on it because I guess he's the one who has followed the closest what happened in the 48 hours before the release, after this problem has been diagnosed by our CI. I'll try to start a discussion about this on a more suitable place.

I clearly missed this when I reviewed #15370, and it was part of the job. This issue was raised late in the release cycle, and I merged it while having also other hotter branches to review in a timely manner, so I did not pay enough attention to check if the change broke reproducibility. I fixed other issues brought by it though, but did not think reproducibility would be that hit.

#4 Updated by lamby 7 months ago

Could you update the ticket with a diffoscope HTML URI? The linked one 404s and I can't seem to find the now-relevant job/build. :)

#5 Updated by segfault 7 months ago

  • Description updated (diff)

Could you update the ticket with a diffoscope HTML URI?

Done.

#6 Updated by segfault 7 months ago

  • Assignee changed from segfault to bertagaz
  • QA Check set to Ready for QA
  • Feature Branch set to bug/15400-python-lib-breaks-reproducibility

Should be fixed by d55d2c0a08d5f5eacf654d7486dfe7a8adc0c695 in branch bug/15400-python-lib-breaks-reproducibility in my repo. Assigning to the 3.7 RM, which is bertagaz according our calendar :)

#7 Updated by segfault 7 months ago

segfault wrote:

Should be fixed by d55d2c0a08d5f5eacf654d7486dfe7a8adc0c695 in branch bug/15400-python-lib-breaks-reproducibility in my repo

I had to fix a path, the commit is now ccdcf81352617406099ea9a5b00511ae68a41bb7.

#8 Updated by intrigeri 7 months ago

It would be nice to have some kind of blame-free post-mortem […]

I clearly missed this when I reviewed #15370, and it was part of the job. This issue was raised late in the release cycle, and I merged it while having also other hotter branches to review in a timely manner, so I did not pay enough attention to check if the change broke reproducibility. I fixed other issues brought by it though, but did not think reproducibility would be that hit.

Thanks for explaining! I think I've understood the root cause and I hope 6dbd9f55bbcc7d69a013976fb0e7000b43c2efff will avoid such issues in the future :) Case closed then.

#9 Updated by intrigeri 7 months ago

  • Target version changed from Tails_3.7 to Tails_3.6.1

#10 Updated by intrigeri 7 months ago

  • Assignee changed from bertagaz to intrigeri

#11 Updated by intrigeri 7 months ago

segfault, in the future please fork bugfix branches off stable. I'll rebase it this time.

#12 Updated by intrigeri 7 months ago

  • Subject changed from python files installed in /usr/local/lib break reproducibility to Python files installed in /usr/local/lib break reproducibility
  • Priority changed from Normal to Elevated
  • % Done changed from 0 to 50
  • Feature Branch changed from bug/15400-python-lib-breaks-reproducibility to bugfix/15400-python-lib-breaks-reproducibility

#13 Updated by segfault 7 months ago

intrigeri wrote:

segfault, in the future please fork bugfix branches off stable.

Ok, didn't know this, but will try to keep it in mind.

#14 Updated by intrigeri 7 months ago

  • % Done changed from 50 to 70

Thanks!

  • Code review:
    • wrt. a867dfd898275d60c935c739a676e6ddb99e2847 I don't really get it: why not drop the quotes instead? You're not quoting the same variable a few lines above so it has to be safe for unquoted usage anyway.
    • I'm not super happy with the fact the implementation will break if the glob ever matches more than one file. Something based on ls or find and loops over the matches would be more robust.
    • segfault, if you agree with the above please give yourself a follow-up ticket for 3.7 or later. But I'm not going to block on this for our 3.6.1 security release.
  • Reproducibility testing: passes on my local Jenkins, congrats!
  • Automated test suite: I'll have results in ~1.5 hours and if nothing is wrong I'll merge.

#15 Updated by intrigeri 7 months ago

segfault, in the future please fork bugfix branches off stable.

Ok, didn't know this

Then I suggest you read https://tails.boum.org/contribute/git/ one of these days :)

#16 Updated by intrigeri 7 months ago

  • Blocks Bug #15417: Upgrade Tor Browser to 7.5.2 added

#17 Updated by intrigeri 7 months ago

  • QA Check changed from Ready for QA to Pass

intrigeri wrote:

  • Automated test suite: I'll have results in ~1.5 hours and if nothing is wrong I'll merge.

Passes (minus the fragile tests that I did not run) locally except one scenario that is always broken here => merging!

#18 Updated by intrigeri 7 months ago

  • Status changed from In Progress to Fix committed
  • % Done changed from 70 to 100

#19 Updated by intrigeri 7 months ago

  • Assignee deleted (intrigeri)

#20 Updated by segfault 7 months ago

intrigeri wrote:

Thanks!

  • Code review:
    • wrt. a867dfd898275d60c935c739a676e6ddb99e2847 I don't really get it: why not drop the quotes instead? You're not quoting the same variable a few lines above so it has to be safe for unquoted usage anyway.

You are right. I thought the quotes in strip_nondeterminism_wrapper() in the following line were the cause:

strip-nondeterminism "${@}"

But indeed in a quick test (I didn't run the build yet) it seems to work if I just remove the quotes in 00-install-tailslib. I don't understand why, but I'm used to not understanding why bash does weird things.

  • I'm not super happy with the fact the implementation will break if the glob ever matches more than one file. Something based on ls or find and loops over the matches would be more robust.

It should not break. I already tested this before, but I see that I should have noted it in the commit message: Both realpath and strip-nondeterminism can take multiple filenames, and then do its work on all of them. So if the glob matches multiple files, they will simply all be stripped.

  • segfault, if you agree with the above please give yourself a follow-up ticket for 3.7 or later. But I'm not going to block on this for our 3.6.1 security release.

I just pushed b3429630495744cb9d88471c6389d340db399df2, if you find the time to review it in time for 3.6.1, you could cherry-pick on your branch. Else I can create a follow-up ticket for 3.7.

#21 Updated by segfault 7 months ago

  • Assignee set to intrigeri

#22 Updated by intrigeri 7 months ago

  • Assignee deleted (intrigeri)

Let's discuss this on #15421: I'm not going to do the QA of this branch again today.

#23 Updated by intrigeri 7 months ago

  • Related to Bug #15421: Improve the implementation of the #15400 bugfix added

#24 Updated by bertagaz 7 months ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF