Feature #11979

Feature #5630: Reproducible builds

Move Vagrant's apt-cacher-ng data to a dedicated disk

Added by intrigeri 7 months ago. Updated about 1 month ago.

Status:ResolvedStart date:11/21/2016
Priority:ElevatedDue date:
Assignee:-% Done:

100%

Category:Build system
Target version:Tails_3.0~rc1
QA Check:Pass Blueprint:
Feature Branch:wip/feature/11979-additional-disk-for-apt-cacher-ng Easy:
Type of work:Code Affected tool:

Description

... so that it can be shared between various Vagrant VMs.


Related issues

Blocks Tails - Feature #11980: Create and provision a new Vagrant VM for every ISO build Resolved 11/21/2016
Blocked by Tails - Feature #11972: Switch our Jenkins ISO build system to vagrant-libvirt Resolved 12/04/2016

Associated revisions

Revision f498eb97
Added by bertagaz 7 months ago

Add a second disk to handle the apt-cacher-ng cache.

Will-fix: #11979

Revision bcbb5306
Added by bertagaz 7 months ago

Make sure that the apt-cacher-ng cache disk is formated ad mounted.

Will-fix: #11979

Revision b5be680f
Added by bertagaz 7 months ago

Fix typo.

Will-fix: #11979

Revision 11abd24a
Added by bertagaz 7 months ago

Fix permissions on the acng cache mounted filesystem.

Will-fix: #11979

Revision df5292fc
Added by bertagaz 7 months ago

Use generic name for acng cache disk.

Otherwise it is guessed from the basebox name, and thus use a new acng
disk each time the basebox name changes. We want to reuse this same uniq
cache disk no matter the basebox.

Will-fix: #11979

Revision b4a61666
Added by bertagaz 7 months ago

Remove buggy code.

As /var/cache/apt-cacher-ng is not mounted automatically when an already
existing vagrant builder is started,created, this part of the code was
triggered without necessity. Better just remove it than trying to work
around.

Will-fix: #11979

Revision afb31ea8
Added by bertagaz 7 months ago

Fix typo.

Refs: #11979

Revision e257e9ff
Added by bertagaz 7 months ago

Don't use absolute path for acng data disk.

Seems not implemented "yet", says the error message.

Refs: #11979

Revision 79b7f9ca
Added by bertagaz about 1 month ago

Merge remote-tracking branch 'origin/wip/11972-use-vagrant-in-jenkins' into stable

Fix-committed: #11972, #11979, #11980, #11981, #11006

History

#1 Updated by intrigeri 7 months ago

  • Blocks Feature #11980: Create and provision a new Vagrant VM for every ISO build added

#2 Updated by bertagaz 7 months ago

  • Status changed from Confirmed to In Progress

#3 Updated by bertagaz 7 months ago

  • Assignee set to anonym
  • Target version set to Tails_2.9.1
  • % Done changed from 0 to 50
  • QA Check set to Ready for QA
  • Feature Branch set to wip/feature/11979-additional-disk-for-apt-cacher-ng

Pushed a branch that works locally and in Jenkins (see the first Jenkins build mentionning vdb and the following build reusing it). Bonus point it migrates previous cache if apt-cacher-ng has already been used in the VM, even if that's not really part of the use case we want to support.

What do you say about reviewing that anonym?

#4 Updated by bertagaz 7 months ago

bertagaz wrote:

Bonus point it migrates previous cache if apt-cacher-ng has already been used in the VM, even if that's not really part of the use case we want to support.

While working on #11980 I removed this part which was buggy and felt too much optimization. The diff is now pretty short and simple and it's quite robust (thanks to vagrant supporting the handling of such additional disks).

My previous note points to a job in Jenkins that is running this ticket branch and #11972 branch in Jenkins since then, without problem. Still #11972 has to be reviewed first, as this branch depends on it.

ame review policy than for this other ticket: if happy, please don't merge it, I'll take care of it, to get this deployed smoothly in Jenkins.

#5 Updated by bertagaz 7 months ago

  • Blocked by Feature #11972: Switch our Jenkins ISO build system to vagrant-libvirt added

#6 Updated by anonym 7 months ago

  • Assignee changed from anonym to bertagaz
  • QA Check changed from Ready for QA to Dev Needed

#7 Updated by anonym 7 months ago

  • % Done changed from 50 to 70
  • QA Check changed from Dev Needed to Info Needed

This branch looks good to me and I'm happy you removed the acng migration code!

I have an idea: from my "spoiler" in #11972#note-14 I think it'd be a great idea to make this disk into tails-builder-common.qcow2, and also store the Git checkout there (what currently is called WORKSPACE -- that name is probably made sense in the beginning, but it's quite wrong these days so feel free to rename it). That way a full git clone is rarely needed, and the builds are speed up as much. Of course, then you shouldn't mount it on /var/cache/apt-cacher-ng any more. :)

What do you think?

#8 Updated by anonym 6 months ago

  • Target version changed from Tails_2.9.1 to Tails 2.10

#9 Updated by intrigeri 5 months ago

  • Target version changed from Tails 2.10 to Tails_2.12

#10 Updated by anonym 3 months ago

I've pushed fixes so that the disk is ignored unless vmproxy is used, which is what we'll need when deploying this on Jenkins.

#11 Updated by anonym 3 months ago

anonym wrote:

This branch looks good to me and I'm happy you removed the acng migration code!

I have an idea: from my "spoiler" in #11972#note-14 I think it'd be a great idea to make this disk into tails-builder-common.qcow2, and also store the Git checkout there (what currently is called WORKSPACE -- that name is probably made sense in the beginning, but it's quite wrong these days so feel free to rename it). That way a full git clone is rarely needed, and the builds are speed up as much. Of course, then you shouldn't mount it on /var/cache/apt-cacher-ng any more. :)

Let's skip this; we cannot do the same on Jenkins since it won't use this disk, so let's not complicate things.

#12 Updated by anonym 3 months ago

  • % Done changed from 70 to 80
  • QA Check changed from Info Needed to Ready for QA

#13 Updated by anonym 3 months ago

At least I am affected by a nasty bug: https://github.com/vagrant-libvirt/vagrant-libvirt/issues/746

I think this will complicate #11980 since we probably will want to do a vagrant destroy when the build finishes. OTOH, we also need to be able to clean up existing old VMs even if vagrant/.vagrant is lost, and vagrant destroy cannot do that, so perhaps we need to use something else any way.

#14 Updated by anonym 3 months ago

Note that I've merged this ticket's feature branch into wip/feature/11980-per-branch-vagrant-build-vm.

anonym wrote:

At least I am affected by a nasty bug: https://github.com/vagrant-libvirt/vagrant-libvirt/issues/746

I think this will complicate #11980 since we probably will want to do a vagrant destroy when the build finishes. OTOH, we also need to be able to clean up existing old VMs even if vagrant/.vagrant is lost, and vagrant destroy cannot do that, so perhaps we need to use something else any way.

In the end I I don't think a fixed vagrant destroy would have been good enough for us, and I implemented something different for #11981. Just to be clear: we are not blocked (and won't benefit) from this upstream vagrant-libvirt bug being solved.

#15 Updated by bertagaz 3 months ago

  • Assignee changed from bertagaz to anonym
  • QA Check changed from Ready for QA to Dev Needed

anonym wrote:

I've pushed fixes so that the disk is ignored unless vmproxy is used, which is what we'll need when deploying this on Jenkins.

Made a review of the code. Seems ready to be merged for me, except:
  • '$TAILS_OFFLINE_BUILD' in Rakefile does not exist.
  • merging the base branch raises conflicts. Looks like it relates to the former remark. Care to fix that? :)

Meanwhile I'll run it locally. If it goes well, I'll merge it with your fix from my review.

#16 Updated by intrigeri 3 months ago

  • '$TAILS_OFFLINE_BUILD' in Rakefile does not exist.

I've fixed that on stable, devel, etc. earlier today.

#17 Updated by anonym 3 months ago

  • Assignee changed from anonym to bertagaz
  • QA Check changed from Dev Needed to Ready for QA

#18 Updated by bertagaz 3 months ago

  • % Done changed from 80 to 100
  • QA Check changed from Ready for QA to Pass

intrigeri wrote:

  • '$TAILS_OFFLINE_BUILD' in Rakefile does not exist.

I've fixed that on stable, devel, etc. earlier today.

Ok, nothing more raised by testing it. Ready to be merged, congrats! I'll do that together with #11980.

#19 Updated by intrigeri 2 months ago

  • Priority changed from Normal to Elevated
  • Target version changed from Tails_2.12 to Tails_3.0~rc1

(The plan is to complete this early in the 3.0 cycle, so it doesn't get in the way of 3.0~rc1 and 3.0 final.)

#20 Updated by bertagaz about 1 month ago

  • Status changed from In Progress to Fix committed

#21 Updated by bertagaz about 1 month ago

  • Assignee deleted (bertagaz)

#22 Updated by intrigeri about 1 month ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF