Bug #12618

Feature #5630: Reproducible builds

Bug #12531: ISO builds on Jenkins are fragile since the migration to vagrant-libvirt

Retrieving ISO build artifacts sometimes fails on Jenkins

Added by intrigeri about 2 months ago. Updated 18 days ago.

Status:ConfirmedStart date:05/31/2017
Priority:ElevatedDue date:
Assignee:bertagaz% Done:

0%

Category:Continuous Integration
Target version:Tails_3.1
QA Check: Blueprint:
Feature Branch: Easy:
Type of work:Sysadmin Affected tool:

Description

E.g. https://jenkins.tails.boum.org/job/build_Tails_ISO_feature-12599/17/console, https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_feature-5630-deterministic-builds/4/console and https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_testing/10/consoleFull expose this problem after successfully building an ISO image:

15:05:41 Retrieving artifacts from Vagrant build box.
15:05:42 Warning: Permanently added '192.168.121.131' (ECDSA) to the list of known hosts.
15:05:42 Warning: Permanently added '192.168.121.131' (ECDSA) to the list of known hosts.
15:21:47 packet_write_wait: Connection to 192.168.121.131 port 22: Broken pipe
15:21:47 lost connection
15:21:49 ==> default: Domain is not running. Please run `vagrant up` or `vagrant resume` first.
15:21:49 ==> default: Domain is not running. Please run `vagrant up` or `vagrant resume` first.
15:21:50 ==> default: Domain is not running. Please run `vagrant up` or `vagrant resume` first.
15:21:51 ==> default: Domain is not running. Please run `vagrant up` or `vagrant resume` first.
15:21:53 rake aborted!
15:21:53 CommandError: command ["scp", "-i", "/var/lib/jenkins/workspace/build_Tails_ISO_feature-12599/vagrant/.vagrant/machines/default/libvirt/private_key", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null", "vagrant@192.168.121.131:/home/vagrant/amnesia/tails-amd64-feature_12599-3.0-20170530T1428Z-4e95409+testing@60c7405.iso.apt-sources", "vagrant@192.168.121.131:/home/vagrant/amnesia/tails-amd64-feature_12599-3.0-20170530T1428Z-4e95409+testing@60c7405.iso", "vagrant@192.168.121.131:/home/vagrant/amnesia/tails-amd64-feature_12599-3.0-20170530T1428Z-4e95409+testing@60c7405.iso.buildlog", "vagrant@192.168.121.131:/home/vagrant/amnesia/tails-amd64-feature_12599-3.0-20170530T1428Z-4e95409+testing@60c7405.iso.packages", "vagrant@192.168.121.131:/home/vagrant/amnesia/tails-amd64-feature_12599-3.0-20170530T1428Z-4e95409+testing@60c7405.iso.build-manifest", "build-artifacts/"] failed with exit status 1
15:21:53 /var/lib/jenkins/workspace/build_Tails_ISO_feature-12599/Rakefile:71:in `run_command'
15:21:53 /var/lib/jenkins/workspace/build_Tails_ISO_feature-12599/Rakefile:434:in `block in <top (required)>'
15:21:53 Tasks: TOP => build
15:21:53 (See full trace by running task with --trace)
15:21:53 Build step 'Execute shell' marked build as failure
15:21:53 [PostBuildScript] - Execution post build scripts.

One way to list such problems is to look for unusually small ISO images in the artifacts directory:
ssh jenkins.lizard ls -l --sort=size /var/lib/jenkins/jobs/*/builds/*/archive/build-artifacts/*.iso.

I don't know if it's mere coincidence, but these 3 builds were run on isobuilder2.


Related issues

Blocked by Tails - Bug #13302: /var/lib/libvirt/images sometimes gets filled on isobuilders, take 2 Resolved 06/30/2017

History

#1 Updated by intrigeri about 2 months ago

This might be caused by #12599#note-15 (unlikely, but who knows). I'm logging date + disk/memory usage info in /tmp/log on all isobuilders so we can check what their status was next time we see this failure.

#2 Updated by bertagaz about 2 months ago

intrigeri wrote:

This might be caused by #12599#note-15 (unlikely, but who knows). I'm logging date + disk/memory usage info in /tmp/log on all isobuilders so we can check what their status was next time we see this failure.

I think it is, at least for the 1st build you're mentioning in the description that was the case. /var/lib/libvirt/images was filled, and the build VM was paused because there was no space left on this partition. I've workaround it by hand at that moment for this one.

#3 Updated by bertagaz about 2 months ago

Last failure of that kind I can see in the logs is job reproducibly_build_Tails_ISO_testing, build 2017-05-30_15-54-29. Since then we've deployed #12999 and #12577. They should help making this failure disappear, and #12595 will fix this definitely anyway.

#4 Updated by intrigeri about 2 months ago

  • Target version changed from Tails_3.0 to Tails_3.1

Let's consider our CI infra as frozen until 3.0 is out.

#5 Updated by bertagaz 18 days ago

  • Blocked by Bug #13302: /var/lib/libvirt/images sometimes gets filled on isobuilders, take 2 added

#6 Updated by intrigeri 18 days ago

Blocked by "Bug #13302: /var/lib/libvirt/images sometimes gets filled on isobuilders, take 2" added

I don't get it. Your previous comment said "#12595 will fix this definitely anyway", and #12595 is about another partition. Can you please clarify?

#7 Updated by bertagaz 18 days ago

intrigeri wrote:

Blocked by "Bug #13302: /var/lib/libvirt/images sometimes gets filled on isobuilders, take 2" added

I don't get it. Your previous comment said "#12595 will fix this definitely anyway", and #12595 is about another partition. Can you please clarify?

Yes, I made a mistake in the first commit link, it's the libvirt partition that is problematic here, as explained in #12618#note-2. So #13302 should fix this for the time being.

#8 Updated by intrigeri 18 days ago

Great, thanks for clarifying :)

Also available in: Atom PDF