Project

General

Profile

Feature #14596

Feature #14568: Additional Software Packages

Write automated tests for Additional Software GUI

Added by u 10 months ago. Updated about 1 month ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
09/04/2017
Due date:
03/06/2018
% Done:

40%

QA Check:
Feature Branch:
feature/14596-automated-tests-for-ASP-gui
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Additional Software Packages

Description

This needs to happen before March 25th 2018 (B3)

We need:

  1. fixing the "all persistence configuration files have safe access rights" step: live-additional-software.conf now has 644 permissions (on purpose), which prevents any scenario that uses persistence to run, so it was temporarily disabled in 19c6e71f0ea99698536b2c3281ee2852ee538001
  2. fixing the Synaptic scenario, that's broken since b34e8a511733a9dbd8718f93f4f28b05e486cbad
  3. a scenario (described in #14572#note-34 and in the offline ASP blueprint) to test the case where an ASP online package upgrade failed, and how if next boot is offline ASP do cope with it; this will validate the work that's been merged already and shipped in Tails 3.6
  4. tests for the main scenarios described on https://tails.boum.org/blueprint/additional_software_packages/gui/

Related issues

Blocks Tails - Feature #14597: Review automated tests for Additional Software GUI Confirmed 09/04/2017 04/01/2018
Blocks Tails - Feature #14598: Code review for Additional Software packages GUI In Progress 09/04/2017 04/15/2018
Blocked by Tails - Feature #14576: Write automated tests for Additional Software GUI (Gherkin) In Progress 08/30/2017 02/28/2018

Associated revisions

Revision 19c6e71f (diff)
Added by intrigeri 4 months ago

Test suite: temporarily disable the "all persistence configuration files have safe access rights" check.

This step will need to be adjusted: live-additional-software.conf now has 644
permissions (refs: #14596)

Revision f2407deb (diff)
Added by intrigeri 4 months ago

Test suite: hopefully fix the temporarily disabling of the "all persistence configuration files have safe access rights" step (refs: #14596)

Revision 377181c3 (diff)
Added by bertagaz 3 months ago

Test suite: add step definitions for ASP feature.

refs: #14596

Revision 76bea076 (diff)
Added by bertagaz 3 months ago

Fatorize ASP services starting detection step.

This rely on the ASP upgrade service creating a state file once started,
which is not the case yet, and should be added to ASP code.

Refs: #14596

Revision abb33e20 (diff)
Added by bertagaz 3 months ago

Test suite: Temoorary fix bug in APT hooks introduced by ASP.

At the moment, APT waits for the DPKG post install hook notification to
be clicked on before returning, which is a bug that should be fixed with
ticket #15382. Meanwhile we have to run APT install in the background
and check when the package is installed to go on with the feature.

Refs: #14596, #15382

Revision d846d6ff (diff)
Added by bertagaz 3 months ago

Test suite: add step definitions for ASP feature.

refs: #14596

Revision 6bb26dbb (diff)
Added by bertagaz 3 months ago

Fatorize ASP services starting detection step.

This rely on the ASP upgrade service creating a state file once started,
which is not the case yet, and should be added to ASP code.

Refs: #14596

Revision 26f89d52 (diff)
Added by bertagaz 3 months ago

Test suite: Temoorary fix bug in APT hooks introduced by ASP.

At the moment, APT waits for the DPKG post install hook notification to
be clicked on before returning, which is a bug that should be fixed with
ticket #15382. Meanwhile we have to run APT install in the background
and check when the package is installed to go on with the feature.

Refs: #14596, #15382

Revision 2c5a0ef8 (diff)
Added by bertagaz 3 months ago

Test suite: Refine a ASP step. (refs: #14596)

Revision 997cab05 (diff)
Added by bertagaz 3 months ago

Test suite: ASP notifications on install or upgrade are the same.

Refs: #14596

Revision aecc789e (diff)
Added by bertagaz 3 months ago

Test suite: Add a step to the second scenario and fill some pending ones.

Refs: #14596

Revision b32661fe (diff)
Added by bertagaz 3 months ago

Test suite: adapt a step shared by apt and ASP features.

When using ASP to create the persistence, it is first mounted in another
directory for initial configuration.

Refs: #14596

Revision af3f79ee (diff)
Added by bertagaz 3 months ago

Make ASP upgrade service create a state file after bootup.

Same than the install service. Is usefull to have a robust way to test
if the service started, and to chain things after.

Refs: #14594, #14596

Revision c089f27c (diff)
Added by bertagaz 3 months ago

Remove unicode character from ASP notification message.

It hits #12185.

Refs: #14594, #14596

Revision dcef9167 (diff)
Added by bertagaz 3 months ago

Test suite: Add a step to the second scenario and fill some pending ones.

Refs: #14596

Revision 4c0e23c6 (diff)
Added by bertagaz 3 months ago

Test suite: adapt a step shared by apt and ASP features.

When using ASP to create the persistence, it is first mounted in another
directory for initial configuration.

Refs: #14596

Revision c3ebc4e9 (diff)
Added by bertagaz 3 months ago

Test suite: Fix name and expand ASP service startup step.

Refs: #14596

Revision 6948fd9c (diff)
Added by bertagaz 3 months ago

Test suite: Add ASP service status step. (refs: #14596)

Revision 1f33a182 (diff)
Added by bertagaz 3 months ago

Test suite: Write, factorize and move apt steps in the correct place.

Refs: #14596

Revision 6c5cddaa (diff)
Added by bertagaz 3 months ago

Test suite: Add step to interact with ASP notifications for package addition/removal.

Refs: #14596

Revision ec702ef7 (diff)
Added by bertagaz 3 months ago

Test suite: Deactivate ASP documentation step for now.

Refs: #14596

Revision e18aba9a (diff)
Added by bertagaz 3 months ago

Test suite: Add step for ASP install/upgrade services. (refs: #14596)

Plug them into the Tails VM boot process. Will be usefull to test other
things, e.g. Tails server.

Revision 730af809 (diff)
Added by bertagaz 3 months ago

Test suite: Let's set the VM non-onion APT sources early in the process.

This is required because of chutney, so let's move that there so that
the APT sources are correctly set everytime.

Refs: #14596

Revision 74dcec3e (diff)
Added by bertagaz 3 months ago

Test suite: Rework ASP notification handling.

The "Installing..." one is almost imposssible to catch in the test
suite, as it disappear fast. So for now its detection is disabled.

Refs: #14596

Revision 4e6306d7 (diff)
Added by bertagaz 3 months ago

Test suite: Drop obsolete ASP step. (refs: #14596)

Revision d6c210ef (diff)
Added by bertagaz 3 months ago

Test suite: untag most ASP scenarios from fragile. (refs: #14596)

Revision 0ca3692a (diff)
Added by bertagaz 3 months ago

Test suite: fix "persistent filesystems have safe access rights" step.

Refs: #14596

Revision 6273bbb6 (diff)
Added by bertagaz 3 months ago

Test suite: Rework ASP notification handling.

The "Installing..." one is almost imposssible to catch in the test
suite, as it disappear fast. So for now its detection is disabled.

Refs: #14596

Revision 67dc4473 (diff)
Added by bertagaz 3 months ago

Test suite: Drop obsolete ASP step. (refs: #14596)

Revision e3714b36 (diff)
Added by bertagaz 3 months ago

Test suite: Add a scenario for ASP, refine package hanling in the feature.

Refs: #14596

Revision 3d588395 (diff)
Added by bertagaz 3 months ago

Test suite: remove workaround for #15430.

Fixed in ASP.

Refs: #14596, #15430

Revision 64906a62 (diff)
Added by bertagaz 3 months ago

Test suite: untag most ASP scenarios from fragile. (refs: #14596)

Revision 02af0301 (diff)
Added by bertagaz 3 months ago

Test suite: Fix ASP scenario. (refs: #14596)

Revision e5522991 (diff)
Added by bertagaz 3 months ago

Test suite: really test how packages using debconf behaves with ASP.

Refs: #14596

Revision 807fa8c0 (diff)
Added by bertagaz 3 months ago

Test suite: Add missing ASP step. (refs: #14596)

Revision 0eaf4a3a (diff)
Added by bertagaz 3 months ago

Test suite: fix bad paste leftovers. (refs: #14596)

Revision a1a369cd (diff)
Added by bertagaz 3 months ago

Test suite: Fix APT feature vs. ASP. (refs: #14596)

  • Remove duplicated scenarios
  • Fix APT sources check scenario
  • Tag ASP scenarios with @check_tor_leaks where necessary

Revision daf6c668 (diff)
Added by bertagaz 3 months ago

Test suite: Grow delay to wait for ASP notifications. (refs: #14596)

They seem a bit slower to show up when the test suite is run in Jenkins.

Revision 9c8f9e6d (diff)
Added by bertagaz 3 months ago

Test suite: Fix APT scenario. (refs: #14596)

Revision 2105c305 (diff)
Added by bertagaz 3 months ago

Test suite: Mark APT step as fragile for now. (refs: #14596)

Revision 41a16eea (diff)
Added by bertagaz 3 months ago

Test suite: Untag ASP Synaptic step as fragile. (refs: #14596)

In fact we need it to install a package that will be removed in a
subsequent scenario. If it's really too fragile, we can still
s/synaptic/apt/ for a while.

Revision 95cbaa93 (diff)
Added by bertagaz 3 months ago

Test suite: Move Chutney APT sources modification step.

Plug it after the chutney Tor configuration step (an not inside), so
that it is possible to disable it. This should fix the APT feature.

Refs: #14596

Revision c517e0fd (diff)
Added by bertagaz 3 months ago

Test suite: Try re-enable ASP notifications step. (refs: #14596)

It was disabled with 6273bbb as lacking robutness. Let see now that the
ASP feature run smoothly how this step behaves.

Revision 768935b3 (diff)
Added by bertagaz 3 months ago

Test suite: Fill missing ASP upgrade scenario steps. (refs: #14596)

Revision 9ac32e3e (diff)
Added by bertagaz 3 months ago

Test suite: remove @fragile tag for ASP upgrade scenario. (refs: #14596)

Revision 18c624c1 (diff)
Added by bertagaz 3 months ago

Test suite: Fill scenario removing package from ASP through its GUI.

Refs: #14596

Revision 076d52e2 (diff)
Added by bertagaz 3 months ago

Test suite: Add missing documentation step in ASP feature. (refs: #14596)

Catching the doucmentation window in the browser did not work for some
reason, so it uses a screenshot to detect the right page has been
opened.

Revision 9f99bfbc (diff)
Added by bertagaz 3 months ago

Test suite: Add scenario testing ASP when persistence is locked.

Refs: #14596

Revision 8fbecd68 (diff)
Added by bertagaz 16 days ago

Fix typo. (refs: #14596)

Revision 0fdec7a7 (diff)
Added by bertagaz 16 days ago

Fix bug introduced in commit 9f99bfb. (refs: #14596)

History

#1 Updated by u 10 months ago

  • Target version set to 2018

#3 Updated by u 10 months ago

  • Blocks Feature #14597: Review automated tests for Additional Software GUI added

#4 Updated by u 10 months ago

  • Blocks Feature #14598: Code review for Additional Software packages GUI added

#5 Updated by u 10 months ago

  • Description updated (diff)
  • Due date changed from 04/01/2018 to 03/06/2018
  • Target version changed from 2018 to Tails_3.6

#6 Updated by u 10 months ago

  • Target version changed from Tails_3.6 to Tails_3.7

#7 Updated by bertagaz 4 months ago

Moved to the ticket description as in #14572#note-36

#8 Updated by bertagaz 4 months ago

  • Description updated (diff)

#9 Updated by intrigeri 4 months ago

  • Description updated (diff)

#10 Updated by intrigeri 4 months ago

  • Blocked by Feature #14576: Write automated tests for Additional Software GUI (Gherkin) added

#11 Updated by u 4 months ago

We need

  1. these tests to be written (and working plausibly well enough) by March 25;
  2. tests reviewed and refixed by April 15.

#12 Updated by bertagaz 4 months ago

Thanks to remind me the "collective" pressure during the meeting, that was so much fun. I think I had the point already.

#13 Updated by intrigeri 4 months ago

  • Description updated (diff)

#14 Updated by intrigeri 4 months ago

  • Description updated (diff)

#15 Updated by intrigeri 4 months ago

  • Status changed from Confirmed to In Progress

#16 Updated by bertagaz 3 months ago

  • % Done changed from 0 to 10
  • Feature Branch set to wip/feature/14596-automated-tests-for-ASP-gui

I will not wait for the full review of #14576, so I've started working on this ticket. I've added the missing steps, andd will start replacing the pending ones with actual code.

I had to workaround #15382, but this is a revertable change.

#17 Updated by bertagaz 3 months ago

  • % Done changed from 10 to 30
  • Feature Branch changed from wip/feature/14596-automated-tests-for-ASP-gui to feature/14596-automated-tests-for-ASP-gui

bertagaz wrote:

I will not wait for the full review of #14576, so I've started working on this ticket. I've added the missing steps, andd will start replacing the pending ones with actual code.

Pushed a lot more commits. I had to hit hard to get most of the feature to work, hence the delay. Only the ASP upgrading service scenarios are left for now (as well as the ones from #14576#note-13), but that's already usable on #14594 and tests a good extend of the ASP features.

I had to workaround #15382, but this is a revertable change.

The way the test suite execute the APT commands still colapse with the ASP hooks, so I had to leave this change.

Let see what Jenkins thinks about it.

#18 Updated by bertagaz 3 months ago

  • % Done changed from 30 to 40

bertagaz wrote:

Let see what Jenkins thinks about it.

Ok, after pushing a few fixes the branch runs well in Jenkins too.

I have locally added the upgrade scenario that i'll push later after a bit of cleaning.

Leftovers:

  • Regression test against the fixed bug of failed upgrades
  • Removing a package from ASP through its GUI
  • Opening the documentation from the notification link
  • The first 'Installing...' notification that is hard to catch because it disappear (sometimes fast). I'll add a note on #14594 to discuss about that (should it not disappear, or should I ignore this one?).
  • Scenarios from #14576#note-13

Other than that it's running well actually and already covers a good extent of ASP. I've largely exploded the time available for this ticket on the budget, so I'll unfocus a bit on this task.

#19 Updated by bertagaz about 1 month ago

  • Target version changed from Tails_3.7 to Tails_3.8

Also available in: Atom PDF