Project

General

Profile

Bug #13302

Feature #5630: Reproducible builds

Bug #12531: ISO builds on Jenkins are fragile since the migration to vagrant-libvirt

/var/lib/libvirt/images sometimes gets filled on isobuilders, take 2

Added by intrigeri 5 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
Continuous Integration
Target version:
Start date:
06/30/2017
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Easy:
Affected tool:

Description

icinga2 just told me that this problem is not 100% solved, and just happened again on isobuilder1 during https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_devel/27/:

$ ls -l /var/lib/libvirt/images
total 2.8G
-rw------- 1 libvirt-qemu libvirt-qemu 1.5G Jun 30 16:46 tails-builder-amd64-jessie-20170609-daf786c121_default.img
-rwxr--r-- 1 libvirt-qemu libvirt-qemu 1.3G Jun 30 15:02 tails-builder-amd64-jessie-20170609-daf786c121_vagrant_box_image_0.img*
$ df /var/lib/libvirt/images
/dev/vdc    ext4      2.9G  2.8G     0 100% /var/lib/libvirt/images

Related issues

Related to Tails - Bug #12599: /var/lib/libvirt/images gets filled on isobuilders Resolved 05/25/2017
Blocks Tails - Bug #12618: Retrieving ISO build artifacts sometimes fails on Jenkins Resolved 05/31/2017

History

#1 Updated by intrigeri 5 months ago

  • Related to Bug #12599: /var/lib/libvirt/images gets filled on isobuilders added

#2 Updated by intrigeri 5 months ago

  • Subject changed from /var/lib/libvirt/images gets filled on isobuilders, take 2 to /var/lib/libvirt/images sometimes gets filled on isobuilders, take 2

#3 Updated by bertagaz 5 months ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 50

intrigeri wrote:

icinga2 just told me that this problem is not 100% solved, and just happened again on isobuilder1 during https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_devel/27/:

[...]

I've growned the libvirt partitions of the isobuilders of 1G (to 4G), so we should be good by now. Let's wait a few days to see.

#4 Updated by bertagaz 5 months ago

  • Blocks Bug #12618: Retrieving ISO build artifacts sometimes fails on Jenkins added

#5 Updated by intrigeri 4 months ago

I've growned the libvirt partitions of the isobuilders of 1G (to 4G), so we should be good by now. Let's wait a few days to see.

In passing, I wonder why the previous size was enough most of the time, but sometimes not. I somewhat suspect the "sometimes not" means there's a bug somewhere, and perhaps growing the partition won't be enough. Anyway, I agree with what you did, and we'll see :)

#6 Updated by bertagaz 4 months ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Ready for QA

intrigeri wrote:

I've growned the libvirt partitions of the isobuilders of 1G (to 4G), so we should be good by now. Let's wait a few days to see.

In passing, I wonder why the previous size was enough most of the time, but sometimes not. I somewhat suspect the "sometimes not" means there's a bug somewhere, and perhaps growing the partition won't be enough. Anyway, I agree with what you did, and we'll see :)

There is maybe several reasons other than a bug: e.g building Stretch makes the snapshot grow a bit more than to build Jessie. Some changes in the basebox creation or provisioning happened too. We'll see.

Anyway, icinga2 is happy since a week, we seems to be good now.

#7 Updated by intrigeri 4 months ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 50 to 100
  • QA Check changed from Ready for QA to Pass

bertagaz wrote:

In passing, I wonder why the previous size was enough most of the time, but sometimes not. I somewhat suspect the "sometimes not" means there's a bug somewhere, and perhaps growing the partition won't be enough. Anyway, I agree with what you did, and we'll see :)

There is maybe several reasons other than a bug: e.g building Stretch makes the snapshot grow a bit more than to build Jessie. Some changes in the basebox creation or provisioning happened too. We'll see.

None of this explains why this problem was happening seemingly randomly, but whatever, we'll see.

Anyway, icinga2 is happy since a week, we seems to be good now.

Great :)

Also available in: Atom PDF