Project

General

Profile

Feature #12576

Feature #5630: Reproducible builds

Have Jenkins use basebox:clean_old instead of basebox:clean_all

Added by intrigeri 7 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Continuous Integration
Target version:
Start date:
05/22/2017
Due date:
% Done:

100%

QA Check:
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

See parent ticket for details.


Related issues

Related to Tails - Feature #12002: Estimate hardware cost of reproducible builds in Jenkins Resolved 11/28/2016
Related to Tails - Feature #14811: Check if old vagrant baseboxes are cleaned up in Jenkins Confirmed 10/07/2017
Blocked by Tails - Bug #12595: Not enough space in /var/lib/jenkins on isobuilders Resolved 05/25/2017
Blocked by Tails - Bug #12575: Fix basebox:clean_old Resolved 05/22/2017

Associated revisions

Revision fc1bba10 (diff)
Added by bertagaz 7 months ago

Rakefile: lower to 4 months the age to which baseboxes gets deleted by basebox:clean_old.

Refs: #12575, #12531, #12576

History

#1 Updated by intrigeri 7 months ago

  • Blocked by Bug #12575: Fix basebox:clean_old added

#2 Updated by intrigeri 7 months ago

  • Blocks Feature #12002: Estimate hardware cost of reproducible builds in Jenkins added

#3 Updated by bertagaz 7 months ago

  • Blocked by Bug #12595: Not enough space in /var/lib/jenkins on isobuilders added

#4 Updated by bertagaz 7 months ago

  • Blocked by Bug #12599: /var/lib/libvirt/images gets filled on isobuilders added

#5 Updated by intrigeri 7 months ago

  • Priority changed from High to Elevated

See notes 14 to 25 on #12531, where there's a discussion about how exactly we can make use of clean_old on Jenkins.

Also, downgrading severity a little bit: it's important to address this build performance regression, but please first focus on what actually breaks builds (#12531 and subtasks), and if this one has to be postponed to shortly after the 3.0 release, so be it (I'd rather not see this change be deployed to late in this cycle unless it's been heavily tested locally first: early June won't be a good time to break our CI, at all ;)

#6 Updated by intrigeri 7 months ago

  • Blocked by deleted (Bug #12575: Fix basebox:clean_old)

#7 Updated by intrigeri 7 months ago

  • Parent task deleted (#12531)

#8 Updated by intrigeri 7 months ago

  • Blocks deleted (Feature #12002: Estimate hardware cost of reproducible builds in Jenkins)

#9 Updated by intrigeri 7 months ago

  • Blocked by deleted (Bug #12595: Not enough space in /var/lib/jenkins on isobuilders)

#10 Updated by intrigeri 7 months ago

  • Blocked by deleted (Bug #12599: /var/lib/libvirt/images gets filled on isobuilders)

#11 Updated by intrigeri 7 months ago

  • Parent task set to #5630

#12 Updated by intrigeri 7 months ago

  • Blocks Feature #12002: Estimate hardware cost of reproducible builds in Jenkins added

#13 Updated by intrigeri 7 months ago

  • Blocked by Bug #12595: Not enough space in /var/lib/jenkins on isobuilders added

#14 Updated by intrigeri 7 months ago

  • Blocked by Bug #12599: /var/lib/libvirt/images gets filled on isobuilders added

#15 Updated by intrigeri 7 months ago

  • Blocked by Bug #12575: Fix basebox:clean_old added

#16 Updated by bertagaz 7 months ago

  • Status changed from Confirmed to In Progress

#17 Updated by intrigeri 7 months ago

Note: this should only be deployed once all active branches have the fix for clean_old. Perhaps the easiest way at this point is to wait for the 3.0 release, unless it's still reasonably doable to apply all recent build system fixes on the stable branch (I think I've seen a bunch of changes pushed to testing/devel only).

#18 Updated by anonym 6 months ago

intrigeri wrote:

Note: this should only be deployed once all active branches have the fix for clean_old. Perhaps the easiest way at this point is to wait for the 3.0 release, unless it's still reasonably doable to apply all recent build system fixes on the stable branch (I think I've seen a bunch of changes pushed to testing/devel only).

Perhaps we instead should just consider Tails 2.x, and temporarily disable CI for the stable branch (and make sure no other active branch has it as base branch), until 3.0 is released? That way we this blocker is gone.

Also, in case it helps (I don't have the full picture of what the problem on Jenkins is), note that I could easily decrease the disk space needed for generating the basebox by reducing the basebox' image size from 20GB to 15GB, or whatever that is enough to do a disk build (or even down to 5 GB or so if we decide to not support disk builds until we have a real fix...).

#19 Updated by intrigeri 6 months ago

Perhaps we instead should just consider Tails 2.x, and temporarily disable CI for the stable branch (and make sure no other active branch has it as base branch), until 3.0 is released? That way we this blocker is gone.

Right, this would work, but IMO improving performance of our CI environment 2 weeks earlier is not worth taking the risk of having no CI in case we want or have to release another 2.x. See #12576#note-5 for my thoughts about the priority and timeline of this ticket.

Also, in case it helps (I don't have the full picture of what the problem on Jenkins is), note that I could easily decrease the disk space needed for generating the basebox […]

Thanks for the offer. I don't think this would unblock anything fundamentally (it likely won't be enough to allow us to store N baseboxes) but let's keep it in mind in case we're a little bit short on storage, and still lack data to purchase additional storage, when we address #12595 which blocks this ticket anyway: I hope not, but saving 4 * a few GB might be just what we need to drop a blocker once we're there.

#20 Updated by intrigeri 6 months ago

  • Blocks deleted (Feature #12002: Estimate hardware cost of reproducible builds in Jenkins)

#21 Updated by intrigeri 6 months ago

  • Related to Feature #12002: Estimate hardware cost of reproducible builds in Jenkins added

#22 Updated by bertagaz 6 months ago

intrigeri wrote:

Note: this should only be deployed once all active branches have the fix for clean_old. Perhaps the easiest way at this point is to wait for the 3.0 release, unless it's still reasonably doable to apply all recent build system fixes on the stable branch (I think I've seen a bunch of changes pushed to testing/devel only).

Indeed. I've investigated a bit what was not merged into stable, by looking at the diff between it and the testing branch in auto/, vagrant/ and Rakefile. There's not so much: seems #12556, #12531, #12525 and 505b31e28732f621795ac5aca4e0b9c65d5e4f02 are the only missing pieces (if I got it right, another look is welcome).

There are other differences (like the ntp/date setup in the build VM) which are related to reproducibility and can be skipped for the stable branch. The Vagrant build related differences I've listed above seem to be easy to merge or cherry-pick in stable given their simplicity.

Now I'm a bit undecided whether doing that now or right after 3.0. As you stated in #note-5, we're not at the best moment to do that.

#23 Updated by intrigeri 6 months ago

Now I'm a bit undecided whether doing that now or right after 3.0. As you stated in #note-5, we're not at the best moment to do that.

I'm in favour of having everything well tested and ready ASAP so it can be deployed first thing after the 3.0 release.

#24 Updated by bertagaz 6 months ago

  • Blocked by deleted (Bug #12599: /var/lib/libvirt/images gets filled on isobuilders)

#25 Updated by intrigeri 6 months ago

  • Target version changed from Tails_3.0 to Tails_3.1

I'll build the 3.0 ISO in two days, so let's not do potentially disruptive changes on our infra at this point.

#26 Updated by intrigeri 6 months ago

  • Priority changed from Elevated to Normal

This will be nice but you have quite a few things with much higher prio to deal with.

#27 Updated by intrigeri 5 months ago

  • Target version changed from Tails_3.1 to Tails_3.2

(Please focus on making builds robust again first, and postpone the performance improvements. Sorry we could not discuss this at the CI team meeting today.)

#29 Updated by intrigeri 3 months ago

  • Target version changed from Tails_3.2 to Tails_3.3

I doesn't feel realistic that all the dependency chain (#12595 and #13425) is resolved in time for 3.2.

#30 Updated by intrigeri 2 months ago

FWIW on my local Jenkins I have done this yesterday night:

--- a/macros/builders.yaml
+++ b/macros/builders.yaml
@@ -39,8 +39,7 @@
 - builder:
     name: clean_old_baseboxes
     builders:
-      - shell: "rake basebox:clean_all" 
-      - shell: "rm -rf /var/lib/jenkins/.vagrant.d" 
+      - shell: "rake basebox:clean_old" 

 - builder:
     name: move_artifacts_to_dir_1

If I notice anything weird or broken, I'll report back here.

#31 Updated by bertagaz 2 months ago

  • Related to Feature #14811: Check if old vagrant baseboxes are cleaned up in Jenkins added

#32 Updated by bertagaz 2 months ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (bertagaz)
  • % Done changed from 0 to 100

intrigeri wrote:

If I notice anything weird or broken, I'll report back here.

Yes, I've just enabled the basebox:clean_old rake task in Jenkins. I've done basic testing, and the basebox is no longer deleted, build is not broken. I've created #14811 to check if baseboxes older than 4 monthes are deleted or not. Closing this ticket.

#33 Updated by intrigeri 2 months ago

Woohoo!

Also available in: Atom PDF