Project

General

Profile

Feature #14596

Feature #14568: Additional Software Packages

Write automated tests for Additional Software GUI

Added by u about 1 year ago. Updated 2 days ago.

Status:
In Progress
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
09/04/2017
Due date:
03/06/2018
% Done:

50%

QA Check:
Dev Needed
Feature Branch:
feature/14596-automated-tests-for-ASP-gui-on-stable
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Additional Software Packages

Description

This needs to happen before March 25th 2018 (B3)

We need:

  1. fixing the "all persistence configuration files have safe access rights" step: live-additional-software.conf now has 644 permissions (on purpose), which prevents any scenario that uses persistence to run, so it was temporarily disabled in 19c6e71f0ea99698536b2c3281ee2852ee538001
  2. fixing the Synaptic scenario, that's broken since b34e8a511733a9dbd8718f93f4f28b05e486cbad
  3. a scenario (described in #14572#note-34 and in the offline ASP blueprint) to test the case where an ASP online package upgrade failed, and how if next boot is offline ASP do cope with it; this will validate the work that's been merged already and shipped in Tails 3.6
  4. tests for the main scenarios described on https://tails.boum.org/blueprint/additional_software_packages/gui/

Related issues

Related to Tails - Bug #15799: "Additional software packages are installed even without network" test always fail in my environment Resolved 08/17/2018
Blocks Tails - Feature #14597: Review automated tests for Additional Software GUI Confirmed 09/04/2017 04/01/2018
Blocked by Tails - Feature #14576: Write automated tests for Additional Software GUI (Gherkin) In Progress 08/30/2017 02/28/2018

Associated revisions

Revision 19c6e71f (diff)
Added by intrigeri 7 months ago

Test suite: temporarily disable the "all persistence configuration files have safe access rights" check.

This step will need to be adjusted: live-additional-software.conf now has 644
permissions (refs: #14596)

Revision f2407deb (diff)
Added by intrigeri 7 months ago

Test suite: hopefully fix the temporarily disabling of the "all persistence configuration files have safe access rights" step (refs: #14596)

Revision 377181c3 (diff)
Added by bertagaz 6 months ago

Test suite: add step definitions for ASP feature.

refs: #14596

Revision 76bea076 (diff)
Added by bertagaz 6 months ago

Fatorize ASP services starting detection step.

This rely on the ASP upgrade service creating a state file once started,
which is not the case yet, and should be added to ASP code.

Refs: #14596

Revision abb33e20 (diff)
Added by bertagaz 6 months ago

Test suite: Temoorary fix bug in APT hooks introduced by ASP.

At the moment, APT waits for the DPKG post install hook notification to
be clicked on before returning, which is a bug that should be fixed with
ticket #15382. Meanwhile we have to run APT install in the background
and check when the package is installed to go on with the feature.

Refs: #14596, #15382

Revision d846d6ff (diff)
Added by bertagaz 6 months ago

Test suite: add step definitions for ASP feature.

refs: #14596

Revision 6bb26dbb (diff)
Added by bertagaz 6 months ago

Fatorize ASP services starting detection step.

This rely on the ASP upgrade service creating a state file once started,
which is not the case yet, and should be added to ASP code.

Refs: #14596

Revision 26f89d52 (diff)
Added by bertagaz 6 months ago

Test suite: Temoorary fix bug in APT hooks introduced by ASP.

At the moment, APT waits for the DPKG post install hook notification to
be clicked on before returning, which is a bug that should be fixed with
ticket #15382. Meanwhile we have to run APT install in the background
and check when the package is installed to go on with the feature.

Refs: #14596, #15382

Revision 2c5a0ef8 (diff)
Added by bertagaz 6 months ago

Test suite: Refine a ASP step. (refs: #14596)

Revision 997cab05 (diff)
Added by bertagaz 6 months ago

Test suite: ASP notifications on install or upgrade are the same.

Refs: #14596

Revision aecc789e (diff)
Added by bertagaz 6 months ago

Test suite: Add a step to the second scenario and fill some pending ones.

Refs: #14596

Revision b32661fe (diff)
Added by bertagaz 6 months ago

Test suite: adapt a step shared by apt and ASP features.

When using ASP to create the persistence, it is first mounted in another
directory for initial configuration.

Refs: #14596

Revision af3f79ee (diff)
Added by bertagaz 6 months ago

Make ASP upgrade service create a state file after bootup.

Same than the install service. Is usefull to have a robust way to test
if the service started, and to chain things after.

Refs: #14594, #14596

Revision c089f27c (diff)
Added by bertagaz 6 months ago

Remove unicode character from ASP notification message.

It hits #12185.

Refs: #14594, #14596

Revision dcef9167 (diff)
Added by bertagaz 6 months ago

Test suite: Add a step to the second scenario and fill some pending ones.

Refs: #14596

Revision 4c0e23c6 (diff)
Added by bertagaz 6 months ago

Test suite: adapt a step shared by apt and ASP features.

When using ASP to create the persistence, it is first mounted in another
directory for initial configuration.

Refs: #14596

Revision c3ebc4e9 (diff)
Added by bertagaz 6 months ago

Test suite: Fix name and expand ASP service startup step.

Refs: #14596

Revision 6948fd9c (diff)
Added by bertagaz 6 months ago

Test suite: Add ASP service status step. (refs: #14596)

Revision 1f33a182 (diff)
Added by bertagaz 6 months ago

Test suite: Write, factorize and move apt steps in the correct place.

Refs: #14596

Revision 6c5cddaa (diff)
Added by bertagaz 6 months ago

Test suite: Add step to interact with ASP notifications for package addition/removal.

Refs: #14596

Revision ec702ef7 (diff)
Added by bertagaz 6 months ago

Test suite: Deactivate ASP documentation step for now.

Refs: #14596

Revision e18aba9a (diff)
Added by bertagaz 6 months ago

Test suite: Add step for ASP install/upgrade services. (refs: #14596)

Plug them into the Tails VM boot process. Will be usefull to test other
things, e.g. Tails server.

Revision 730af809 (diff)
Added by bertagaz 6 months ago

Test suite: Let's set the VM non-onion APT sources early in the process.

This is required because of chutney, so let's move that there so that
the APT sources are correctly set everytime.

Refs: #14596

Revision 74dcec3e (diff)
Added by bertagaz 6 months ago

Test suite: Rework ASP notification handling.

The "Installing..." one is almost imposssible to catch in the test
suite, as it disappear fast. So for now its detection is disabled.

Refs: #14596

Revision 4e6306d7 (diff)
Added by bertagaz 6 months ago

Test suite: Drop obsolete ASP step. (refs: #14596)

Revision d6c210ef (diff)
Added by bertagaz 6 months ago

Test suite: untag most ASP scenarios from fragile. (refs: #14596)

Revision 0ca3692a (diff)
Added by bertagaz 6 months ago

Test suite: fix "persistent filesystems have safe access rights" step.

Refs: #14596

Revision 6273bbb6 (diff)
Added by bertagaz 6 months ago

Test suite: Rework ASP notification handling.

The "Installing..." one is almost imposssible to catch in the test
suite, as it disappear fast. So for now its detection is disabled.

Refs: #14596

Revision 67dc4473 (diff)
Added by bertagaz 6 months ago

Test suite: Drop obsolete ASP step. (refs: #14596)

Revision e3714b36 (diff)
Added by bertagaz 6 months ago

Test suite: Add a scenario for ASP, refine package hanling in the feature.

Refs: #14596

Revision 3d588395 (diff)
Added by bertagaz 6 months ago

Test suite: remove workaround for #15430.

Fixed in ASP.

Refs: #14596, #15430

Revision 64906a62 (diff)
Added by bertagaz 6 months ago

Test suite: untag most ASP scenarios from fragile. (refs: #14596)

Revision 02af0301 (diff)
Added by bertagaz 6 months ago

Test suite: Fix ASP scenario. (refs: #14596)

Revision e5522991 (diff)
Added by bertagaz 6 months ago

Test suite: really test how packages using debconf behaves with ASP.

Refs: #14596

Revision 807fa8c0 (diff)
Added by bertagaz 6 months ago

Test suite: Add missing ASP step. (refs: #14596)

Revision 0eaf4a3a (diff)
Added by bertagaz 6 months ago

Test suite: fix bad paste leftovers. (refs: #14596)

Revision a1a369cd (diff)
Added by bertagaz 6 months ago

Test suite: Fix APT feature vs. ASP. (refs: #14596)

  • Remove duplicated scenarios
  • Fix APT sources check scenario
  • Tag ASP scenarios with @check_tor_leaks where necessary

Revision daf6c668 (diff)
Added by bertagaz 6 months ago

Test suite: Grow delay to wait for ASP notifications. (refs: #14596)

They seem a bit slower to show up when the test suite is run in Jenkins.

Revision 9c8f9e6d (diff)
Added by bertagaz 6 months ago

Test suite: Fix APT scenario. (refs: #14596)

Revision 2105c305 (diff)
Added by bertagaz 6 months ago

Test suite: Mark APT step as fragile for now. (refs: #14596)

Revision 41a16eea (diff)
Added by bertagaz 6 months ago

Test suite: Untag ASP Synaptic step as fragile. (refs: #14596)

In fact we need it to install a package that will be removed in a
subsequent scenario. If it's really too fragile, we can still
s/synaptic/apt/ for a while.

Revision 95cbaa93 (diff)
Added by bertagaz 6 months ago

Test suite: Move Chutney APT sources modification step.

Plug it after the chutney Tor configuration step (an not inside), so
that it is possible to disable it. This should fix the APT feature.

Refs: #14596

Revision c517e0fd (diff)
Added by bertagaz 6 months ago

Test suite: Try re-enable ASP notifications step. (refs: #14596)

It was disabled with 6273bbb as lacking robutness. Let see now that the
ASP feature run smoothly how this step behaves.

Revision 768935b3 (diff)
Added by bertagaz 6 months ago

Test suite: Fill missing ASP upgrade scenario steps. (refs: #14596)

Revision 9ac32e3e (diff)
Added by bertagaz 6 months ago

Test suite: remove @fragile tag for ASP upgrade scenario. (refs: #14596)

Revision 18c624c1 (diff)
Added by bertagaz 6 months ago

Test suite: Fill scenario removing package from ASP through its GUI.

Refs: #14596

Revision 076d52e2 (diff)
Added by bertagaz 6 months ago

Test suite: Add missing documentation step in ASP feature. (refs: #14596)

Catching the doucmentation window in the browser did not work for some
reason, so it uses a screenshot to detect the right page has been
opened.

Revision 9f99bfbc (diff)
Added by bertagaz 6 months ago

Test suite: Add scenario testing ASP when persistence is locked.

Refs: #14596

Revision 8fbecd68 (diff)
Added by bertagaz 4 months ago

Fix typo. (refs: #14596)

Revision 0fdec7a7 (diff)
Added by bertagaz 4 months ago

Fix bug introduced in commit 9f99bfb. (refs: #14596)

Revision 3761ed0a (diff)
Added by bertagaz about 1 month ago

Test suite: take into account that apt(8) won't return when run in the remote shell with the ASP hooks enabled.

This cherry-picks 26f89d52ad5a1a9c937b0b0682116dd5f04c6cea aka. "Test
suite: Temoorary fix bug in APT hooks introduced by ASP" whose original commit
message was:

At the moment, APT waits for the DPKG post install hook notification to
be clicked on before returning, which is a bug that should be fixed with
ticket #15382. Meanwhile we have to run APT install in the background
and check when the package is installed to go on with the feature.

As bertagaz clarified later (https://labs.riseup.net/code/issues/15382#note-12),
for some reason that remains to be investigated, we still need this workaround
despite the ASP's apt-post hook now being non-blocking.

Refs: #14596

Revision 9e1be302 (diff)
Added by bertagaz 16 days ago

Test suite: Remove unecessary step for ASP and factor out a bit.

Refs: #14596

Revision f4463e6e (diff)
Added by bertagaz 16 days ago

Test suite: Fill last pending ASP scenario.

Refs: #14596

Revision 5c418670 (diff)
Added by bertagaz 13 days ago

Test suite: add step definitions for ASP feature.

refs: #14596

Revision 0d01241c (diff)
Added by bertagaz 13 days ago

Fatorize ASP services starting detection step.

This rely on the ASP upgrade service creating a state file once started,
which is not the case yet, and should be added to ASP code.

Refs: #14596

Revision 021eb311 (diff)
Added by bertagaz 13 days ago

Test suite: Refine a ASP step. (refs: #14596)

Revision 60cf315b (diff)
Added by bertagaz 13 days ago

Test suite: ASP notifications on install or upgrade are the same.

Refs: #14596

Revision 520d2a08 (diff)
Added by bertagaz 13 days ago

Test suite: Add a step to the second scenario and fill some pending ones.

Refs: #14596

Revision 24dbd382 (diff)
Added by bertagaz 13 days ago

Test suite: Add a step to the second scenario and fill some pending ones.

Refs: #14596

Revision c79f6585 (diff)
Added by bertagaz 13 days ago

Test suite: adapt a step shared by apt and ASP features.

When using ASP to create the persistence, it is first mounted in another
directory for initial configuration.

Refs: #14596

Revision 885a61a6 (diff)
Added by bertagaz 13 days ago

Test suite: Fix name and expand ASP service startup step.

Refs: #14596

Revision 82c56a94 (diff)
Added by bertagaz 13 days ago

Test suite: Add ASP service status step. (refs: #14596)

Revision 7db56022 (diff)
Added by bertagaz 13 days ago

Test suite: Write, factorize and move apt steps in the correct place.

Refs: #14596

Revision e3375f01 (diff)
Added by bertagaz 13 days ago

Test suite: Add step to interact with ASP notifications for package addition/removal.

Refs: #14596

Revision 2a785ea5 (diff)
Added by bertagaz 13 days ago

Test suite: Deactivate ASP documentation step for now.

Refs: #14596

Revision a4d94e90 (diff)
Added by bertagaz 13 days ago

Test suite: Add step for ASP install/upgrade services. (refs: #14596)

Plug them into the Tails VM boot process. Will be usefull to test other
things, e.g. Tails server.

Revision 49c3535a (diff)
Added by bertagaz 13 days ago

Test suite: Let's set the VM non-onion APT sources early in the process.

This is required because of chutney, so let's move that there so that
the APT sources are correctly set everytime.

Refs: #14596

Revision 98ea1504 (diff)
Added by bertagaz 13 days ago

Test suite: fix "persistent filesystems have safe access rights" step.

Refs: #14596

Revision 05a07555 (diff)
Added by bertagaz 13 days ago

Test suite: Rework ASP notification handling.

The "Installing..." one is almost imposssible to catch in the test
suite, as it disappear fast. So for now its detection is disabled.

Refs: #14596

Revision fe74fb98 (diff)
Added by bertagaz 13 days ago

Test suite: Drop obsolete ASP step. (refs: #14596)

Revision 880bbd07 (diff)
Added by bertagaz 13 days ago

Test suite: Add a scenario for ASP, refine package hanling in the feature.

Refs: #14596

Revision 4eb23799 (diff)
Added by bertagaz 13 days ago

Test suite: remove workaround for #15430.

Fixed in ASP.

Refs: #14596, #15430

Revision fd5deb96 (diff)
Added by bertagaz 13 days ago

Test suite: untag most ASP scenarios from fragile. (refs: #14596)

Revision 2eb8fc12 (diff)
Added by bertagaz 13 days ago

Test suite: Fix ASP scenario. (refs: #14596)

Revision cbb3c91d (diff)
Added by bertagaz 13 days ago

Test suite: really test how packages using debconf behaves with ASP.

Refs: #14596

Revision 9ccccffe (diff)
Added by bertagaz 13 days ago

Test suite: Add missing ASP step. (refs: #14596)

Revision 691aa9e5 (diff)
Added by bertagaz 13 days ago

Test suite: fix bad paste leftovers. (refs: #14596)

Revision 1021c2f1 (diff)
Added by bertagaz 13 days ago

Test suite: Fix APT feature vs. ASP. (refs: #14596)

  • Remove duplicated scenarios
  • Fix APT sources check scenario
  • Tag ASP scenarios with @check_tor_leaks where necessary

Revision e13a48f3 (diff)
Added by bertagaz 13 days ago

Test suite: Grow delay to wait for ASP notifications. (refs: #14596)

They seem a bit slower to show up when the test suite is run in Jenkins.

Revision d04c36b9 (diff)
Added by bertagaz 13 days ago

Test suite: Fix APT scenario. (refs: #14596)

Revision 32f996dd (diff)
Added by bertagaz 13 days ago

Test suite: Mark APT step as fragile for now. (refs: #14596)

Revision 8ec1fae8 (diff)
Added by bertagaz 13 days ago

Test suite: Untag ASP Synaptic step as fragile. (refs: #14596)

In fact we need it to install a package that will be removed in a
subsequent scenario. If it's really too fragile, we can still
s/synaptic/apt/ for a while.

Revision bc7b07f2 (diff)
Added by bertagaz 13 days ago

Test suite: Move Chutney APT sources modification step.

Plug it after the chutney Tor configuration step (an not inside), so
that it is possible to disable it. This should fix the APT feature.

Refs: #14596

Revision 7c3d08f4 (diff)
Added by bertagaz 13 days ago

Test suite: Fill missing ASP upgrade scenario steps. (refs: #14596)

Revision ee49ae96 (diff)
Added by bertagaz 13 days ago

Test suite: remove @fragile tag for ASP upgrade scenario. (refs: #14596)

Revision a5c4bef3 (diff)
Added by bertagaz 13 days ago

Test suite: Fill scenario removing package from ASP through its GUI.

Refs: #14596

Revision d2a67e7f (diff)
Added by bertagaz 13 days ago

Test suite: Add missing documentation step in ASP feature. (refs: #14596)

Catching the doucmentation window in the browser did not work for some
reason, so it uses a screenshot to detect the right page has been
opened.

Revision 408e6d96 (diff)
Added by bertagaz 13 days ago

Test suite: Add scenario testing ASP when persistence is locked.

Refs: #14596

Revision ad863afe (diff)
Added by bertagaz 13 days ago

Fix typo. (refs: #14596)

Revision 004f3952 (diff)
Added by bertagaz 13 days ago

Fix bug introduced in commit 9f99bfb. (refs: #14596)

Revision 12fc3185 (diff)
Added by bertagaz 13 days ago

Test suite: Remove unecessary step for ASP and factor out a bit.

Refs: #14596

Revision a627c1eb (diff)
Added by bertagaz 13 days ago

Test suite: Fill last pending ASP scenario.

Refs: #14596

Revision f5e0a3bb (diff)
Added by bertagaz 12 days ago

Test suite: Fix ASP config application name in dogtail step.

Refs: #14596

Revision bf98379f (diff)
Added by bertagaz 10 days ago

Test suite: Raise waiting time for the ASP installation service startup.

It seems it takes a bit more time in Jenkins.

Refs: #14596

History

#1 Updated by u about 1 year ago

  • Target version set to 2018

#3 Updated by u about 1 year ago

  • Blocks Feature #14597: Review automated tests for Additional Software GUI added

#4 Updated by u about 1 year ago

  • Blocks Feature #14598: Code review for Additional Software packages GUI added

#5 Updated by u about 1 year ago

  • Description updated (diff)
  • Due date changed from 04/01/2018 to 03/06/2018
  • Target version changed from 2018 to Tails_3.6

#6 Updated by u about 1 year ago

  • Target version changed from Tails_3.6 to Tails_3.7

#7 Updated by bertagaz 7 months ago

Moved to the ticket description as in #14572#note-36

#8 Updated by bertagaz 7 months ago

  • Description updated (diff)

#9 Updated by intrigeri 7 months ago

  • Description updated (diff)

#10 Updated by intrigeri 7 months ago

  • Blocked by Feature #14576: Write automated tests for Additional Software GUI (Gherkin) added

#11 Updated by u 7 months ago

We need

  1. these tests to be written (and working plausibly well enough) by March 25;
  2. tests reviewed and refixed by April 15.

#12 Updated by bertagaz 7 months ago

Thanks to remind me the "collective" pressure during the meeting, that was so much fun. I think I had the point already.

#13 Updated by intrigeri 7 months ago

  • Description updated (diff)

#14 Updated by intrigeri 7 months ago

  • Description updated (diff)

#15 Updated by intrigeri 7 months ago

  • Status changed from Confirmed to In Progress

#16 Updated by bertagaz 6 months ago

  • % Done changed from 0 to 10
  • Feature Branch set to wip/feature/14596-automated-tests-for-ASP-gui

I will not wait for the full review of #14576, so I've started working on this ticket. I've added the missing steps, andd will start replacing the pending ones with actual code.

I had to workaround #15382, but this is a revertable change.

#17 Updated by bertagaz 6 months ago

  • % Done changed from 10 to 30
  • Feature Branch changed from wip/feature/14596-automated-tests-for-ASP-gui to feature/14596-automated-tests-for-ASP-gui

bertagaz wrote:

I will not wait for the full review of #14576, so I've started working on this ticket. I've added the missing steps, andd will start replacing the pending ones with actual code.

Pushed a lot more commits. I had to hit hard to get most of the feature to work, hence the delay. Only the ASP upgrading service scenarios are left for now (as well as the ones from #14576#note-13), but that's already usable on #14594 and tests a good extend of the ASP features.

I had to workaround #15382, but this is a revertable change.

The way the test suite execute the APT commands still colapse with the ASP hooks, so I had to leave this change.

Let see what Jenkins thinks about it.

#18 Updated by bertagaz 6 months ago

  • % Done changed from 30 to 40

bertagaz wrote:

Let see what Jenkins thinks about it.

Ok, after pushing a few fixes the branch runs well in Jenkins too.

I have locally added the upgrade scenario that i'll push later after a bit of cleaning.

Leftovers:

  • Regression test against the fixed bug of failed upgrades
  • Removing a package from ASP through its GUI
  • Opening the documentation from the notification link
  • The first 'Installing...' notification that is hard to catch because it disappear (sometimes fast). I'll add a note on #14594 to discuss about that (should it not disappear, or should I ignore this one?).
  • Scenarios from #14576#note-13

Other than that it's running well actually and already covers a good extent of ASP. I've largely exploded the time available for this ticket on the budget, so I'll unfocus a bit on this task.

#19 Updated by bertagaz 5 months ago

  • Target version changed from Tails_3.7 to Tails_3.8

#20 Updated by intrigeri 3 months ago

  • Target version changed from Tails_3.8 to Tails_3.9

#21 Updated by u 3 months ago

bertagaz, what should we do with this ticket? Are there still tests that need to be written? Did you push your latest modifications? Does this need review? Please assign the ticket to a reviewer (possibly Alan) if that's the case.

#22 Updated by u about 2 months ago

Ping? We plan to release this feature in 10 days… Please let me know what remains to be done here and when you plan to do it. Thanks!

#23 Updated by intrigeri about 2 months ago

u wrote:

Does this need review? Please assign the ticket to a reviewer (possibly Alan) if that's the case.

It does need to be reviewed, see #14597 for details (who and when).

#24 Updated by intrigeri about 2 months ago

  • Priority changed from Normal to High

(As per #14597#note-13.)

#25 Updated by intrigeri about 2 months ago

  • Blocks deleted (Feature #14598: Code review for Additional Software packages GUI)

#26 Updated by intrigeri about 1 month ago

FYI, now that the main ASP branch was merged, I had to cherry-pick 26f89d52ad5a1a9c937b0b0682116dd5f04c6cea into devel (now known as 3761ed0a85f209da0551ff02c8566b83744d03e0) because otherwise the I install "cowsay" using apt test step fails. Will push once I've confirmed that's enough to fix the problem I see.

#27 Updated by intrigeri about 1 month ago

Will push once I've confirmed that's enough to fix the problem I see.

Done.

Now entering meta territory. In the future, please handle these separately:

  • Changes requires to ensure the pre-existing test suite keeps passing, such as the one I've cherry-picked: these should go into the main topic branch for the feature, so regardless of the timing of the rest of the test suite work, even if that other work is not completed in time for the merge as planned, merging the main topic branch for the feature does not break the test suite.
  • New tests specific to the new features:
    • These should go into their dedicated branch, as you did :)
    • That dedicated branch should follow closely the main topic branch for the feature, to ensure it's actually testing what we want to merge. In the last two months this only happened once, because I did it myself.

#28 Updated by intrigeri about 1 month ago

  • Related to Bug #15799: "Additional software packages are installed even without network" test always fail in my environment added

#29 Updated by intrigeri about 1 month ago

I've merged the most recent state of the testing branch I could (anything newer would make it FTBFS) and pushed, so that your topic branch is built & tested on Jenkins again.

#30 Updated by intrigeri 18 days ago

  • Target version changed from Tails_3.9 to Tails_3.10

#31 Updated by bertagaz 17 days ago

intrigeri wrote:

Will push once I've confirmed that's enough to fix the problem I see.

Done.

Thanks!

Now entering meta territory. In the future, please handle these separately:

  • Changes requires to ensure the pre-existing test suite keeps passing, such as the one I've cherry-picked: these should go into the main topic branch for the feature, so regardless of the timing of the rest of the test suite work, even if that other work is not completed in time for the merge as planned, merging the main topic branch for the feature does not break the test suite.

Right, given both ASP code and test branches were supposed to be merged together I did not pay much attention to that, my bad.

  • New tests specific to the new features:
    • These should go into their dedicated branch, as you did :)
    • That dedicated branch should follow closely the main topic branch for the feature, to ensure it's actually testing what we want to merge. In the last two months this only happened once, because I did it myself.

That's not true, I've updated my branch myself, at least tried to at every Tails release (and sometimes #14594 branch too), but the merge was a bif mess. Then I was MIA for some time.

I've merged the most recent state of the testing branch I could (anything newer would make it FTBFS) and pushed, so that your topic branch is built & tested on Jenkins again.

Thanks again. I guess now that 3.9 is out I can update my branch.

u wrote:

bertagaz, what should we do with this ticket? Are there still tests that need to be written? Did you push your latest modifications? Does this need review? Please assign the ticket to a reviewer (possibly Alan) if that's the case.

There's one last test that I'm currently finishing to test (recovering from upgrade failure). It should be pushed tomorrow. From what I remember from our last meeting with Alan, we agreed that other scenarios discussed in #14576#note-15 were not really necessary. So per #14596#note-18 that means once this last scenario is pushed, the branch will be ready for review. But beware that without a workaround it will hit the bug in #14598#note-62.

#32 Updated by intrigeri 17 days ago

  • That dedicated branch should follow closely the main topic branch for the feature, to ensure it's actually testing what we want to merge. In the last two months this only happened once, because I did it myself.

That's not true, I've updated my branch myself, at least tried to at every Tails release (and sometimes #14594 branch too), but the merge was a bif mess. Then I was MIA for some time.

I suspect you've missed "in the last two months" in my comment. Once you take it into account, then either my claim is correct or I got my Git command wrong, in which case I'm sorry. Anyway, what's done is done and I was explicitly requesting a workflow change for the future :)

#33 Updated by bertagaz 15 days ago

  • Assignee changed from bertagaz to intrigeri
  • % Done changed from 40 to 50
  • QA Check set to Ready for QA

bertagaz wrote:

There's one last test that I'm currently finishing to test (recovering from upgrade failure). It should be pushed tomorrow. From what I remember from our last meeting with Alan, we agreed that other scenarios discussed in #14576#note-15 were not really necessary. So per #14596#note-18 that means once this last scenario is pushed, the branch will be ready for review. But beware that without a workaround it will hit the bug in #14598#note-62.

Done, so I think it's ready for review now.

#34 Updated by intrigeri 13 days ago

  • Assignee changed from intrigeri to bertagaz
  • QA Check deleted (Ready for QA)

The branch FTBFS on Jenkins, presumably because it's based on the devel branch. Please base your branch on stable. Thanks in advance!

#35 Updated by bertagaz 13 days ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Ready for QA
  • Feature Branch changed from feature/14596-automated-tests-for-ASP-gui to feature/14596-automated-tests-for-ASP-gui-on-stable

intrigeri wrote:

The branch FTBFS on Jenkins, presumably because it's based on the devel branch. Please base your branch on stable. Thanks in advance!

Ok pushed a new branch rebased on stable. It has one scenario failing due to the bug I reported on #14598#note-62. If you want to run the feature sucessfuly, just replace "The upgrade of your additional.." by "The installation of your additional..." on line 145 until it's fixed.

#36 Updated by intrigeri 12 days ago

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Ready for QA to Info Needed

Ok pushed a new branch rebased on stable.

Thanks! Any reason to keep the old one?

It has one scenario failing due to the bug I reported on #14598#note-62.

Actually, the same two scenarios failed during the first two runs on Jenkins. Is the second one caused by a bug in the code (as opposed to a test suite bug) too?

#37 Updated by bertagaz 11 days ago

  • Assignee changed from bertagaz to intrigeri

intrigeri wrote:

Ok pushed a new branch rebased on stable.

Thanks! Any reason to keep the old one?

Yes, what I reported on #14598#note-62 did not show up in https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-14596-automated-tests-for-asp-gui-on-stable/3/ so I'd like to see if it depends on running on a branch based on the #14594 branch which is based on devel and has newer commits.

It has one scenario failing due to the bug I reported on #14598#note-62.

Actually, the same two scenarios failed during the first two runs on Jenkins. Is the second one caused by a bug in the code (as opposed to a test suite bug) too?

I've pushed a fix for one of this failure (which I've never seen with the branch based on the newer #14594) and the other one did not show up in the build mentioned above. However, a new one happened in the next build on Jenkins. I'm a bit surprised, first time I see it, and first time it appears in any build in Jenkins. Seems gnome-shell took more time than usual to start. Will have a look.

#38 Updated by intrigeri 11 days ago

  • Assignee changed from intrigeri to bertagaz

However, a new one happened in the next build on Jenkins. I'm a bit surprised, first time I see it, and first time it appears in any build in Jenkins. Seems gnome-shell took more time than usual to start. Will have a look.

Seems that this should remains on your plate for now, then. Please reassign to me for review once Jenkins is happy :)

#39 Updated by bertagaz 11 days ago

intrigeri wrote:

Seems that this should remains on your plate for now, then. Please reassign to me for review once Jenkins is happy :)

Oops sorry, I did not mean to do that.

#40 Updated by bertagaz 2 days ago

Importing here part of #14576#note-29 that were said to better fit in this ticket's comments.

intrigeri wrote:

But wait, I suggest you don't jump on fixing them all immediately: I've just noticed that this branch increases the (non-fragile) test suite runtime by 40% (the ASP feature takes longer than the next 3 features that take most time combined).

Indeed, as I have tried to cover most of the use cases and code base, this feature is now quite long to run, specially since it is not optimized.

I see that you've mentioned this problem — which was smaller back then — on this ticket a few months ago but I don't recall any follow-up discussion on this topic. I'd rather have discussed this collectively months ago during a team meeting, because this would have informed your work (what should be implemented and how) instead of now that you've invested so much time into implementing all the tests already. But what's done is done, here we are today, so let's have this conversation now.

Yeah, it has not been discussed during meetings, and no one took a look at the work in progress. I don't remember there was a meeting at the time this feature has grown so much. Might be that I missed some too.

It's clear to me that this feature cannot justifiably add such a cost (resources, longer feedback loop): IMO 30 minutes would be great; 40 minutes would be good; 50-60 minutes might be acceptable if every test has a well justified cost/benefit; but 1h40 is definitely too much.

Sounds like a more reasonnable timing proposal.

I think we should do this: first, look for what makes it take so much time and for cheap ways to optimize the implementation, without affecting the nature of the tests at this point (currently there's no cheating in place, which is a very good starting point, we'll see if we can stick to it).
If these optimizations cut the cost down to something acceptable, great, we can stop here. Otherwise, for each test: is it worth its current cost? If not, what would be the drawback of taking shortcuts, e.g. merely checking whether the config was updated as expected instead of rebooting? Or shall we remove it entirely or mark it as @too_slow and skip it on Jenkins by default? If you want I can help a bit, e.g. to brainstorm candidate cheap optimizations (once we know what steps take so much time) and to suggest shortcuts (cheating) we could use.

Yes, I think avoiding rebooting in every scenario to check if the package is really installed or removed is a good way to go. Checking that once is probably enough, and then for other scenarios we can just check for the presence or absence of the package in the ASP conf. But from calculations I've made, it only lowers the whole feature run time from 20-25 minutes in Jenkins, which is not enough.

The two scenarios that take the most time to run are the upgrade ones. Now that I think about it, we can probably merge them (after the upgrade failure, check after reboot the old package is still correctly installed, then plug the network and check it's upgraded). That would remove something like 10 minutes from the run time, which would get closer to 60 minutes.

As an indication, here are the (gross) run time numbers I've collected from Jenkins:

  Scenario: I am warned I can not use ASP when I boot Tails from a DVD and install a package
    -> 4mn (no reboot)
  Scenario: I can set up and use ASP when I install a package in a Tails that has no persistent partition
    -> 9mn (reboot needed)
  Scenario: I can install packages in a Tails session with locked down persistence without being annoyed by ASP 
    -> 4mn (no reboot)
  Scenario: Packages I install with Synaptic and add to ASP are automatically installed
    -> 10mn (without rebooting: 6mn)
  Scenario: Packages I uninstall and accept to remove from ASP are not installed anymore
    -> 9mn (without rebooting: 5mn)
  Scenario: Packages I uninstall but don't want to remove from ASP are automatically installed
    -> 9mn (without rebooting: 6mn)
  Scenario: Packages I install but not do not add to ASP are not automatically installed
    -> 9mn (without rebooting: 5mn)
  Scenario: Packages I have installed and added to ASP are upgraded when a network is available
    -> 11mn (reboot needed)
  Scenario: Packages I uninstall through ASP GUI are not installed anymore
    -> 9mn (without rebooting: 5mn)
  Scenario: Recovering in offline mode after ASP previously failed to upgrade a package
    -> 14mn (reboot needed)
  Scenario: I am notified when ASP fails to install a package
    -> 8mn (reboot needed)

-> Actual run time : 96mn
-> Without reboots: 77mn

I'll try with the above proposals to see how it goes reagrding the feature run time.

#41 Updated by bertagaz 2 days ago

  • QA Check changed from Info Needed to Dev Needed

Also available in: Atom PDF