Project

General

Profile

Feature #12391

Run full test suite when building from a release tag

Added by anonym about 1 year ago. Updated 8 days ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Start date:
03/20/2017
Due date:
% Done:

50%

QA Check:
Ready for QA
Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

When jenkins notices a tag has been pushed or modified (from git push -f --tags) it should check if the changelog at the tagged commit has its first entry != UNRELEASED. Then we know that that is a commit/tag the we are releasing, so it should add a build from that exact tag. And in the test suite it should run all tests (i.e. drop do not do --tag ~@fragile).

This will be a step towards involving Jenkins in our release process for real, and the full test suite run will gather some interesting data about how well the @fragile tagged tests are running on Jenkins. We could save these (ideally Jenkins should be told to keep builds of release tags longer than other builds) and look at them once or twice a year to see if it seems some tests are not fragile on Jenkins any more.

History

#1 Updated by anonym about 1 year ago

  • Description updated (diff)

#2 Updated by anonym about 1 year ago

  • Assignee set to bertagaz

What do you think? :)

#3 Updated by u 12 months ago

ping @bertagaz :)

#4 Updated by intrigeri 12 months ago

  • Status changed from New to Confirmed
  • QA Check set to Info Needed

#5 Updated by intrigeri 11 months ago

  • Target version set to Tails_3.2

#6 Updated by intrigeri 11 months ago

(Just so the initial question is on bertagaz' radar. The goal is not to implement this for 3.2.)

#7 Updated by bertagaz 9 months ago

  • Target version changed from Tails_3.2 to Tails_3.3

#8 Updated by bertagaz 8 months ago

  • Target version changed from Tails_3.3 to Tails_3.5

Realistically reschedule for 3.4.

#9 Updated by u 5 months ago

  • Target version changed from Tails_3.5 to Tails_3.6

Postpone this again.

#10 Updated by bertagaz 3 months ago

  • Target version changed from Tails_3.6 to Tails_3.7

#11 Updated by bertagaz about 1 month ago

  • Target version changed from Tails_3.7 to Tails_3.8

#12 Updated by intrigeri 8 days ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check deleted (Info Needed)

anonym wrote:

When jenkins notices a tag has been pushed or modified (from git push -f --tags) it should check if the changelog at the tagged commit has its first entry != UNRELEASED. Then we know that that is a commit/tag the we are releasing, so it should add a build from that exact tag.

We already build from the tag in that case (IIRC that was done for reproducible builds) so the ISO build part is covered.

And in the test suite it should run all tests (i.e. drop do not do --tag ~@fragile).

I'm on it (because I have no other simple way to run the full test suite as part of the 3.7.1 release process due to #15460).

#13 Updated by intrigeri 8 days ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

#14 Updated by intrigeri 8 days ago

  • Assignee changed from intrigeri to bertagaz
  • % Done changed from 10 to 50
  • QA Check set to Ready for QA

Seems to work (https://jenkins.tails.boum.org/job/manual_test_Tails_ISO_stable/18/):

08:38:13 + grep -q -E \+force-all-tests$
08:38:13 + git describe --tags --exact-match 0314a6c499dce0f36a317fb11e455d47e177f30a
08:38:13 3.7.1
08:38:13 + TAGS_ARGS=
08:38:13 + as_root_do ./run_test_suite --old-iso tmp/tails-amd64-3.7.1.iso --iso tmp/tails-amd64-3.7.1.iso --artifacts-base-uri https://jenkins.tails.boum.org/job/manual_test_Tails_ISO_stable/18/artifact/build-artifacts/ --capture --
08:38:13 + sudo -n WORKSPACE=/var/lib/jenkins/workspace/manual_test_Tails_ISO_stable TMP_DIR=/tmp/TailsToaster UPSTREAMJOB_ISO=tmp/tails-amd64-3.7.1.iso PREVIOUS_ISO=tmp/tails-amd64-3.7.1.iso ./run_test_suite --old-iso tmp/tails-amd64-3.7.1.iso --iso tmp/tails-amd64-3.7.1.iso --artifacts-base-uri https://jenkins.tails.boum.org/job/manual_test_Tails_ISO_stable/18/artifact/build-artifacts/ --capture --

#15 Updated by intrigeri 8 days ago

anonym wrote:

We could save these (ideally Jenkins should be told to keep builds of release tags longer than other builds) and look at them once or twice a year to see if it seems some tests are not fragile on Jenkins any more.

One cheap-ish way would be to have a script running regularly, that looks for jobs that were run from a tag (clone tails.git and run git describe --tags --exact-match on GIT_COMMIT for each jenkins.lizard:/var/lib/jenkins/jobs/test_Tails_ISO_stable/builds/*-*-*_*-*-*/injectedEnvVars.txt), and that copies their cucumber.json to some place that's not affecting by Jenkins' GC. Let's not block on this here. I assume this topic will resurface once development on our test suite resumes.

Also available in: Atom PDF