Project

General

Profile

Feature #10748

Feature #5926: Freezable APT repository

Feature #9489: Implement packages importing and freezing

Generate a manifest of packages used at build time

Added by intrigeri over 2 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
Build system
Target version:
Start date:
12/13/2015
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
feature/10748-gen-packages-manifest
Type of work:
Code

Description

At ISO build time, generate a list of used packages and version, including packages used at build time but not shipped in the ISO.

Output: a machine-readable file that sums up all the information we'll later need to create a tagged, partial snapshot of the APT repositories we use, that contains only packages that were used at ISO build time.


Related issues

Related to Tails - Feature #5548: Research ways to distribute source Resolved
Duplicated by Tails - Feature #6297: Save list of packages used at ISO build time Duplicate
Blocks Tails - Feature #11412: Drop support for packages-from-acng-log in our Puppet manifests Resolved 05/11/2016

Associated revisions

Revision 3175d7a5
Added by intrigeri over 2 years ago

Merge remote-tracking branch 'kibi/feature/5926-freezable-APT-repository' into feature/10748-gen-packages-manifest

Refs: #10748

Revision 655fda5e (diff)
Added by intrigeri over 2 years ago

Move gen-manifests.pl to a better place, and give it a less ambiguous name.

This is where other auxiliary build scripts run before or after
live-build are stored.

Refs: #10748

Revision 452d94a0 (diff)
Added by intrigeri over 2 years ago

Generate build manifest once the ISO has been created.

... and adjust manual build doc to include new dependencies.

Refs: #10748

Revision b6eeb6f9 (diff)
Added by intrigeri over 2 years ago

Extract list of (origin, reference) from the build configuration.

For now, we hard-code arbitrary snapshot serial numbers in the build
configuration. They at least allow us to write and test the code that
reads them from the configuration tree.

refs: #10748

Revision f7f07e73 (diff)
Added by intrigeri over 2 years ago

generate-build-manifest: also work if using {time-based,tagged} snapshot while building.

The initial PoC did not take into account that most of the time,
and particularly when we're interested in this script's output,
we'll be using our own time-based snapshots for building ISO images.

refs: #10748

Revision 05dabbb5
Added by intrigeri over 2 years ago

Merge branch 'feature/10748-gen-packages-manifest' into feature/5926-freezable-APT-repository

Refs: #10748

Revision 3380121d (diff)
Added by intrigeri over 2 years ago

TODO--

I doubt that de-duplication would save enough resources to be worth
the effort and added complexity.

refs: #10748

Revision 6ec84076 (diff)
Added by intrigeri over 2 years ago

apt-get wrapper: break command-line parsing as soon as we've guessed a valid operation mode.

refs: #10748

Revision 1719105c (diff)
Added by intrigeri over 2 years ago

Move gen-manifests.pl to a better place, and give it a less ambiguous name.

This is where other auxiliary build scripts run before or after
live-build are stored.

Refs: #10748

Revision 427792a6 (diff)
Added by intrigeri over 2 years ago

Generate build manifest once the ISO has been created.

... and adjust manual build doc to include new dependencies.

Refs: #10748

Revision 111c10c1 (diff)
Added by intrigeri over 2 years ago

Extract list of (origin, reference) from the build configuration.

For now, we hard-code arbitrary snapshot serial numbers in the build
configuration. They at least allow us to write and test the code that
reads them from the configuration tree.

refs: #10748

Revision 7025f1ce (diff)
Added by intrigeri over 2 years ago

generate-build-manifest: also work if using {time-based,tagged} snapshot while building.

The initial PoC did not take into account that most of the time,
and particularly when we're interested in this script's output,
we'll be using our own time-based snapshots for building ISO images.

refs: #10748

Revision be6e6bfe (diff)
Added by intrigeri over 2 years ago

TODO--

I doubt that de-duplication would save enough resources to be worth
the effort and added complexity.

refs: #10748

Revision e7bfac6a (diff)
Added by Cyril Brulebois over 2 years ago

Build a global (package,version) list for each type.

Keeping separate (per-origin) lists is misleading:
- Information is lost: we would need to keep track of all origins where
a given (package,version) is available from.
- That's actually not needed since we have references to the time-based
snapshots, and the server-side manifest consumer has everything to
decide which packages to keep for a tagged snapshot.

Therefore, drop per-origin lists; and keep a single list for each type
(binary, source).

Refs: #10748

Signed-off-by: Cyril Brulebois <>

Revision 9e2df212
Added by anonym about 2 years ago

Merge remote-tracking branch 'origin/feature/10748-gen-packages-manifest' into devel

Fix-committed: #10748

History

#1 Updated by intrigeri over 2 years ago

  • Blueprint set to https://tails.boum.org/blueprint/freezable_APT_repository/

#2 Updated by intrigeri over 2 years ago

  • Blocked by Feature #6297: Save list of packages used at ISO build time added

#3 Updated by intrigeri over 2 years ago

  • % Done changed from 10 to 20
  • Feature Branch changed from kibi:feature/5926-freezable-APT-repository to feature/5926-freezable-APT-repository

Merged kibi's work, tested basic functionality, integrated into our build system. I'll track what's left to check and test to the blueprint.

#4 Updated by intrigeri over 2 years ago

OK, so everything left to check depends on me looking into tails-prepare-tagged-apt-snapshot-import first.

#5 Updated by intrigeri over 2 years ago

  • Parent task changed from #9489 to #10749

#6 Updated by intrigeri over 2 years ago

  • Blocked by deleted (Feature #6297: Save list of packages used at ISO build time)

#7 Updated by intrigeri over 2 years ago

  • Parent task deleted (#10749)

#8 Updated by intrigeri over 2 years ago

  • Parent task set to #9489

#9 Updated by intrigeri over 2 years ago

  • Blocked by Feature #6297: Save list of packages used at ISO build time added

#10 Updated by intrigeri over 2 years ago

  • Blocks Feature #10749: Create partial APT snapshot from a build manifest and a set of time-based snapshots added

#11 Updated by intrigeri over 2 years ago

  • Assignee changed from intrigeri to CyrilBrulebois
  • QA Check changed from Ready for QA to Info Needed

The main potential issues I've discovered are being discussed over email:

  • "Tails#10748: build manifest vs. multiple origins"
  • "Tails#10748, Tails#10749: architectures"

#12 Updated by intrigeri over 2 years ago

  • Assignee changed from CyrilBrulebois to intrigeri
  • Priority changed from Normal to Elevated
  • Target version changed from Tails_2.0 to Tails_2.2
  • QA Check deleted (Info Needed)

Re-assigning to me so I have it in mind, and will schedule time with Cyril to address the identified problems in the delivered code.

#13 Updated by intrigeri over 2 years ago

  • Target version changed from Tails_2.2 to Tails_2.3

#14 Updated by intrigeri over 2 years ago

intrigeri wrote:

The main potential issues I've discovered are being discussed over email:

  • "Tails#10748: build manifest vs. multiple origins"

It would be good to check if this change in APT will impact that problem:

apt (1.1~exp9) experimental; urgency                    =medium

  A new algorithm for pinning has been implemented, it now assigns a
  pin priority to a version instead of assigning a pin to a package.

  This might break existing corner cases of pinning, if they use multiple
  pins involving the same package name or patterns matching the same
  package name, but should overall lead to pinning that actually works
  as intended and documented.

 -- Julian Andres Klode <jak@debian.org>  Mon, 17 Aug 2015 14:45:17 +0200

#15 Updated by intrigeri over 2 years ago

intrigeri wrote:

The main potential issues I've discovered are being discussed over email:

  • "Tails#10748: build manifest vs. multiple origins"

We won't handle that when generating the manifest, but on #10749.

#16 Updated by intrigeri over 2 years ago

  • Feature Branch changed from feature/5926-freezable-APT-repository to feature/10748-gen-packages-manifest

#17 Updated by intrigeri over 2 years ago

  • Blocks deleted (Feature #10749: Create partial APT snapshot from a build manifest and a set of time-based snapshots)

#18 Updated by intrigeri over 2 years ago

  • Subject changed from Generate a manifest of packages used at build time per-origin to Generate a manifest of packages used at build time

#19 Updated by intrigeri over 2 years ago

  • Blocked by deleted (Feature #6297: Save list of packages used at ISO build time)

#20 Updated by intrigeri over 2 years ago

  • Duplicated by Feature #6297: Save list of packages used at ISO build time added

#21 Updated by intrigeri over 2 years ago

  • Related to Feature #5548: Research ways to distribute source added

#22 Updated by intrigeri over 2 years ago

  • Description updated (diff)

#23 Updated by intrigeri over 2 years ago

  • Feature Branch changed from feature/10748-gen-packages-manifest to feature/10748-gen-packages-manifest, puppet-tails:feature/10748-gen-packages-manifest

#24 Updated by intrigeri about 2 years ago

  • Target version changed from Tails_2.3 to Tails_2.4

#25 Updated by intrigeri about 2 years ago

  • Feature Branch changed from feature/10748-gen-packages-manifest, puppet-tails:feature/10748-gen-packages-manifest to feature/10748-gen-packages-manifest, puppet-tails

#26 Updated by intrigeri about 2 years ago

  • Feature Branch changed from feature/10748-gen-packages-manifest, puppet-tails to feature/10748-gen-packages-manifest

#27 Updated by intrigeri about 2 years ago

  • Blocks Feature #11412: Drop support for packages-from-acng-log in our Puppet manifests added

#28 Updated by intrigeri about 2 years ago

  • Blocks deleted (Feature #11412: Drop support for packages-from-acng-log in our Puppet manifests)

#29 Updated by intrigeri about 2 years ago

  • Blocks Feature #11412: Drop support for packages-from-acng-log in our Puppet manifests added

#30 Updated by intrigeri about 2 years ago

  • Blocks Feature #10749: Create partial APT snapshot from a build manifest and a set of time-based snapshots added

#31 Updated by intrigeri about 2 years ago

  • Assignee changed from intrigeri to anonym
  • % Done changed from 20 to 50
  • QA Check set to Ready for QA

Please review and merge :)

#32 Updated by anonym about 2 years ago

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Ready for QA to Info Needed

Code looks good, and makes sense. Great job! I admittedly didn't look at Cyril's generate-build-manifest, but I assume you have, and I know you are in a much better position to do so. :)

However, I thought I should at least make a low-effort sanity check of the generated .build-manifest compared to the packages downloaded according to the .buildlog. Try running the below, but set ISO to some of your own builds.

ISO=tails-i386-devel-2.4-20160512T1527Z-7137ac2.iso
BUILD_MANIFEST="${ISO}.build-manifest" 
BUILDLOG="${ISO}.buildlog" 
sed -nE 's/^  ( |-) package: //p' "${BUILD_MANIFEST}" | sort -u > /tmp/pkgs-from-manifest
sed -nE \
  -e 's/^I: Retrieving (\S+) .*$/\1/p' \
  -e 's/^Get:[0-9]+\s+\S+\s+\S+\s+(\S+)\s+.*$/\1/p' \
  "${BUILDLOG}" | sort -u > /tmp/pkgs-from-buildlog
diff -Naur /tmp/pkgs-from-manifest /tmp/pkgs-from-buildlog

The diff contains some crap you'll have to filter out manually, but shows two packages that's omitted but definitely should not be:

  • lockfile-progs: this one is even shipped in Tails, so I do not see why it should not be in the manifest.
  • squashfs-tools: this one is not in Tails, but used during build to squash the filesystem so it should be in the manifest.

So, either explain why I am confused and reassign the ticket back to me, or, if you confirm that this is wrong, keep the ticket and change QA Check to Dev Needed.

#33 Updated by intrigeri about 2 years ago

  • QA Check changed from Info Needed to Dev Needed

#34 Updated by intrigeri about 2 years ago

However, I thought I should at least make a low-effort sanity check of the generated .build-manifest compared to the packages downloaded according to the .buildlog.

Thanks for trying that! I would have noticed at least one of these two issues later on, when trying to build from a (partial) tagged snapshot that has only the packages listed in the build manifest, but still it's good to detect problems early :)

  • lockfile-progs: this one is even shipped in Tails, so I do not see why it should not be in the manifest.

Fixed in 4b4e5b8. tl;dr: yes, apt-get purge can install packages.

  • squashfs-tools: this one is not in Tails, but used during build to squash the filesystem so it should be in the manifest.

OK, so this one is a weird one.

It is pulled by the build system outside of the chroot that we customize (that includes using our apt-get wrapper), compress and ship, and it affects the ISO just as much as anything else in the build VM, so arguably it's part of the environment that we do not try to capture. OTOH, in practice it'll be pulled from http://time-based.snapshots.deb.tails.boum.org/debian/ most of the time, so when we build from a tagged snapshot it'll be pulled from http://tagged.snapshots.deb.tails.boum.org/debian/, and then the build will fail because that package will be missing. So, even if I could have a good reason to dismiss this one, I have to deal with it... somehow. I'll see if I can handle it nicely (not too hopeful), and worst case we'll have a list of "extra" packages that should always be added to the build manifest, regardless of whether we have detected that they have been pulled during the build process.

#35 Updated by intrigeri about 2 years ago

  • Assignee changed from intrigeri to anonym
  • % Done changed from 50 to 60
  • QA Check changed from Dev Needed to Ready for QA

Both fixed.

#36 Updated by anonym about 2 years ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 60 to 100
  • QA Check changed from Ready for QA to Pass

Yup, now the two problematic packages are present in the build-manifest. While reviewing your recent addition to generate-build-manifest I had a look at the full script, not the Perl particularities, but the general approach, and it looks straightforward and good.

No more complaints => merged!

#37 Updated by intrigeri about 2 years ago

  • Blocks deleted (Feature #10749: Create partial APT snapshot from a build manifest and a set of time-based snapshots)

#38 Updated by anonym about 2 years ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF