Project

General

Profile

Feature #5288

Run the test suite automatically on autobuilt ISOs

Added by Tails over 4 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
Continuous Integration
Target version:
Start date:
01/03/2014
Due date:
10/15/2015
% Done:

100%

QA Check:
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
No
Affected tool:

Description

Running the test suite automatically on the ISO built by autobuild would improve the quality of our development branches and overall robustness.


Subtasks

Feature #6564: Deploy a platform for automated testingResolved

Feature #6565: Run the test suite automatically on Git pushDuplicate

Feature #7631: Get a server able to run our automated test suiteResolved

Bug #8506: Fix virtual network problems on lizardResolved

Feature #8667: Lead a discussion to specify what automatically build ISOs needs to be tested, when and howResolved

Feature #9486: Support running multiple instances of the test suite in parallelResolved

Feature #9646: Upgrade jenkins-job-builder to 1.3.0+Resolved

Bug #10215: Suboptimal advance booking of Jenkins slaves for testing ISOsResolved

Feature #9597: Set up a way to share artifacts between jobsResolved

Bug #9645: Create at least a second VM for testing ISO imagesResolved

Feature #10117: Design how to run our test suite in JenkinsResolved

Feature #10118: Write library code that maps Jenkins jobs from building to testingResolved

Feature #10119: Adapt the job generation script to also output corresponding test jobsResolved

Feature #10158: Mount /dev/vdc in /var/lib/jenkins on isotester1Resolved

Bug #10229: ISO testing jobs seem to lack a timeoutResolved

Feature #10287: Set up limited email notification on automatic test failure for the initial deploymentResolved

Feature #10296: Reevaluate how the email notifications for failed automatic tests scaleResolved

Feature #10382: Implement the specified notification system for test suite failures on JenkinsResolved

Bug #10396: Sort out overallocated storage situation on isotesterN.lizardResolved

Bug #10409: Reintroduce the old_iso logic for release branchesResolvedintrigeri

Feature #10445: Pass --artifacts-base-uri in Jenkins test suite wrapperResolved

Feature #10507: Document how to get the statistics we need out of JenkinsResolved

Bug #10601: isotesterN:s sometimes disappearResolved

Bug #10717: Concerning amount of test suite runs aborted on Jenkins due to timeoutRejected

Bug #10725: Test jobs sometimes hang at cucumber report parsing stepRejected

Feature #10863: Collect & analyze false positives in Jenkins test suite runs (late December 2015 to late January 2016)Resolved

Bug #10993: Define and bootstrap a process to collect & analyze false positives in Jenkins test suite runsResolved


Related issues

Related to Tails - Feature #6090: Automated builds Resolved 07/26/2013 02/28/2015
Blocked by Tails - Bug #7060: Fix automated test suite's "I run ..." step Resolved 04/10/2014
Blocked by Tails - Feature #8539: Make the test suite robust enough to be run as part of a CI setup Resolved 10/05/2014 03/06/2015

History

#1 Updated by intrigeri over 4 years ago

  • Category set to Continuous Integration
  • Starter set to No

#2 Updated by bertagaz over 4 years ago

  • Assignee set to bertagaz

#3 Updated by intrigeri over 4 years ago

  • Subject changed from run the test suite on autobuild ISOs to run the test suite on autobuilt ISOs

#4 Updated by intrigeri about 4 years ago

  • Subject changed from run the test suite on autobuilt ISOs to Run the test suite on autobuilt ISOs

#5 Updated by intrigeri about 4 years ago

  • Subject changed from Run the test suite on autobuilt ISOs to Run the test suite automatically on autobuilt ISOs

#6 Updated by intrigeri over 3 years ago

#7 Updated by intrigeri over 3 years ago

  • Blocked by Bug #6092: Tails Installer sometimes hangs late in the installation process added

#8 Updated by intrigeri over 3 years ago

  • Blocks Bug #7060: Fix automated test suite's "I run ..." step added

#9 Updated by intrigeri over 3 years ago

  • Blocks deleted (Bug #7060: Fix automated test suite's "I run ..." step)

#10 Updated by intrigeri over 3 years ago

  • Blocked by Bug #7060: Fix automated test suite's "I run ..." step added

#11 Updated by intrigeri over 3 years ago

  • Blocked by Bug #8009: The Unsafe Browser tests sometimes fail due to too fast keyboard interaction added

#12 Updated by intrigeri over 3 years ago

  • Blocked by Bug #8059: Windows Camouflage automated tests sometimes failed due to differently ordered icons in the notification area added

#13 Updated by intrigeri over 3 years ago

  • Blocked by Bug #8140: Running applications via the GNOME menu is fragile added

#14 Updated by intrigeri over 3 years ago

  • Blocked by Feature #5330: Test suite: identify and document race conditions added

#15 Updated by intrigeri about 3 years ago

  • Blocked by Feature #8539: Make the test suite robust enough to be run as part of a CI setup added

#16 Updated by intrigeri about 3 years ago

  • Blocked by deleted (Bug #6092: Tails Installer sometimes hangs late in the installation process)

#17 Updated by intrigeri about 3 years ago

  • Blocked by deleted (Bug #8009: The Unsafe Browser tests sometimes fail due to too fast keyboard interaction)

#18 Updated by intrigeri about 3 years ago

  • Blocked by deleted (Bug #8059: Windows Camouflage automated tests sometimes failed due to differently ordered icons in the notification area)

#19 Updated by intrigeri about 3 years ago

  • Blocked by deleted (Bug #8140: Running applications via the GNOME menu is fragile)

#20 Updated by intrigeri about 3 years ago

  • Blocked by deleted (Feature #5330: Test suite: identify and document race conditions)

#21 Updated by bertagaz about 3 years ago

  • Target version changed from Hardening_M1 to Tails_1.6

#23 Updated by intrigeri over 2 years ago

  • Blocks Feature #9264: Consider buying more server hardware to run our automated test suite added

#30 Updated by bertagaz over 2 years ago

  • Target version changed from Tails_1.6 to Tails_1.7

#31 Updated by sajolida over 2 years ago

Note that this is due on October 15 which is actually before Tails 1.7.

#32 Updated by bertagaz about 2 years ago

  • Target version changed from Tails_1.7 to Tails_1.8

Postponing hopefully for the last time.

#33 Updated by intrigeri about 2 years ago

  • Blocks deleted (Feature #9264: Consider buying more server hardware to run our automated test suite)

#34 Updated by bertagaz about 2 years ago

  • Target version changed from Tails_1.8 to Tails_2.0

Postponing

#35 Updated by intrigeri about 2 years ago

  • Target version changed from Tails_2.0 to Tails_2.3

Please set a target version that means "all subtasks are fixed". That won't be 2.0 so postponing quite a bit already.

#36 Updated by intrigeri about 2 years ago

  • Status changed from Confirmed to In Progress

#37 Updated by intrigeri about 2 years ago

  • Blocked by Bug #10601: isotesterN:s sometimes disappear added

#38 Updated by intrigeri about 2 years ago

(#10601 should be a subtask, but Redmine doesn't want to hear about it.)

#39 Updated by intrigeri almost 2 years ago

  • Blocked by deleted (Bug #10601: isotesterN:s sometimes disappear)

#40 Updated by intrigeri almost 2 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (bertagaz)

All subtasks have been resolved.

#41 Updated by sajolida almost 2 years ago

High five!

Also available in: Atom PDF