Project

General

Profile

Feature #14576

Feature #14568: Additional Software Packages

Write automated tests for Additional Software GUI (Gherkin)

Added by u 11 months ago. Updated 14 days ago.

Status:
In Progress
Priority:
Elevated
Assignee:
Category:
Test suite
Target version:
Start date:
08/30/2017
Due date:
02/28/2018
% Done:

50%

QA Check:
Info Needed
Feature Branch:
wip/feature/14576-automated-tests-for-ASP-gui
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Additional Software Packages

Description

Please write the Gherkin part of these tests until February 28th 2018. (B3)


Related issues

Blocked by Tails - Feature #14574: Design GUI for Additional Software packages Resolved 01/29/2018
Blocks Tails - Feature #14596: Write automated tests for Additional Software GUI In Progress 09/04/2017 03/06/2018

Associated revisions

Revision 1067fa05 (diff)
Added by bertagaz 5 months ago

Start writing gherkin scenarios for ASP.

Refs: #14576

Revision e76ef1b3 (diff)
Added by bertagaz 5 months ago

Mark ASP feature as fragile.

So that it won't be run in Jenkins.

Refs: #14576

Revision a96bf302 (diff)
Added by bertagaz 5 months ago

ASP test suite: Move Background into a Scenario.

In the same vein than USB upgrade: use a scenario to bootstrap the Tails
environment with ASP persistence configured rather than a Background
step.

Refs: #14576

Revision 4c6cc3cc (diff)
Added by bertagaz 5 months ago

Typo. (refs: #14576)

Revision 6a078ac8 (diff)
Added by bertagaz 5 months ago

ASP test suite: rework and simplify failed upgrade scenario.

Refs: #14576

Revision e6920ed3 (diff)
Added by bertagaz 5 months ago

ASP test suite: Write 2 more scenarios, delay the ASP GUI one for later.

Refs: #14576

Revision f1bdff68 (diff)
Added by bertagaz 5 months ago

ASP test suite: Fix missing step forgotten while rewriting scenario.

Refs: #14576

Revision 1e5a3276 (diff)
Added by bertagaz 5 months ago

ASP test suite: Add scenario to test ASP GUI.

Refs: #14576

Revision 4aeb9e8c (diff)
Added by bertagaz 5 months ago

Test suite: Rework the ASP feature. (refs: #14576)

Revision 7c78ce17 (diff)
Added by bertagaz 5 months ago

Test suite: reword an ASP scenario name. (refs: #14576)

Revision f2c0aa54 (diff)
Added by bertagaz 5 months ago

Test suite: fix some misnamed ASP steps. (refs: #14576)

Revision a3b1c395 (diff)
Added by bertagaz 5 months ago

ASP test suite: typos. (refs #14576)

Revision b40063a4 (diff)
Added by bertagaz 5 months ago

Rephrase step coherently. (refs: #14576)

Revision 31d6aa8f (diff)
Added by bertagaz 5 months ago

ASP test suite: typo. (refs: #14576)

Revision 9e4905f1 (diff)
Added by bertagaz 4 months ago

Test suite: add step to open documentation through ASP notification.

Refs: #14576

History

#1 Updated by u 11 months ago

  • Subject changed from Write automated tests for Additional Software GUI to Write automated tests for Additional Software GUI (Gherkin)

#2 Updated by u 11 months ago

  • Description updated (diff)

#3 Updated by intrigeri 11 months ago

  • Category set to Test suite

#4 Updated by BitingBird 11 months ago

  • Target version set to 2018

#5 Updated by u 11 months ago

  • Affected tool set to Additional Software Packages

#6 Updated by u 11 months ago

  • Target version changed from 2018 to Tails_3.6

#7 Updated by intrigeri 10 months ago

  • Blocked by Feature #14574: Design GUI for Additional Software packages added

#8 Updated by bertagaz 5 months ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 20
  • QA Check set to Dev Needed
  • Feature Branch set to feature/14576-automated-tests-for-ASP-gui

I've started to redact the scenarios. I've added 9 scenarios I could think of while reading the blueprint. A quick look from someone to see if I did not forget something would be great.

Feature: Additional software packages
  As a Tails user
  I may want to install softwares not shipped in Tails
  And have them installed automatically when I enable persistence in the Greeter

  Scenario: ASP persistence can be set up and used when installing a package in Tails without persistence

  Scenario: Additional software packages are installed even without network

  Scenario: Packages installed with Synaptic and added to ASP are automatically installed

  Scenario: Packages installed with APT but not added to ASP are not automatically installed

  Scenario: Packages installed with ASP are upgraded when a network is available

  # Tests the ASP upgrade process as well as when a previous upgrade failed.
  Scenario: Recovering in offline mode after a previous failed ASP packages upgrade

  Scenario: I am warned I can not use ASP when booting Tails from DVD and installing a package

  # Tests package removal from ASP configuration using ASP GUI
  Scenario: Editing ASP configuration through the GUI and remove a package from the list

  # Tests manual package removal from ASP using APT
  Scenario: Editing ASP configuration through the GUI and remove a package from the list

  Scenario: I am notified when ASP failed to install a package

Of this 9 new scenarios, 6 are already almost written in Gherkin. It brought expected refactoring in the code. More will come when the 4 left will be written. Moving target, so for now no need to look in details the steps wording yet.

#9 Updated by bertagaz 5 months ago

bertagaz wrote:

I've started to redact the scenarios. I've added 9 scenarios I could think of while reading the blueprint. A quick look from someone to see if I did not forget something would be great.

[...]

Here's an updated version

Feature: Additional software packages
  As a Tails user
  I may want to install softwares not shipped in Tails
  And have them installed automatically when I enable persistence in the Greeter

  Scenario: I am warned I can not use ASP when booting Tails from DVD and installing a package

  Scenario: ASP persistence can be set up and used when installing a package in Tails without persistence

  Scenario: Additional software packages are installed even without network

  Scenario: Packages installed with Synaptic and added to ASP are automatically installed

  Scenario: Packages installed with APT but not added to ASP are not automatically installed

  Scenario: Packages installed with ASP are upgraded when a network is available

  Scenario: Recovering in offline mode after a previous failed ASP upgrade

  Scenario: I am notified when ASP fails to install a package

  Scenario: Removing a package from ASP through APT 

  Scenario: Removing a package from ASP through its GUI

#10 Updated by bertagaz 5 months ago

  • % Done changed from 20 to 30
  • Feature Branch changed from feature/14576-automated-tests-for-ASP-gui to wip/feature/14576-automated-tests-for-ASP-gui

Pushed a few commits that finish to do a first write of the steps for all the scenarios described above.

I'll let the dust settle a bit before re-reading them and fixing the last issues I'll find, then it'll be ready for review.

#11 Updated by bertagaz 5 months ago

  • Assignee changed from bertagaz to anonym
  • % Done changed from 30 to 50
  • QA Check changed from Dev Needed to Ready for QA

bertagaz wrote:

I'll let the dust settle a bit before re-reading them and fixing the last issues I'll find, then it'll be ready for review.

I've pushed some commits on the branch, which fix some issues. I think now all the ASP features described in the blueprint are covered, so it's time for the reviewer to have a look at the Gherkin.

I've used the same trick as in the usb upgrade feature: the second scenario bootstraps a Tails with ASP persistence set. I've used scenarios to uninstall packages and remove them from ASP configuration so that we don't have to install X different packages to test ASP features. It's also a good way to simulate a user's Tails ASP persistence life cycle: installing packages and then removing them, and then re-add them.

So what do you think? There may be improvement in the phrasing probably.

#12 Updated by intrigeri 5 months ago

  • Blocks Feature #14596: Write automated tests for Additional Software GUI added

#13 Updated by alant 4 months ago

As asked, I had a look at the scenarios.

I think most of the important things are covered. I suggest adding:

Scenario: I am not notified when I install a package and I have a persistent storage but it is not unlocked

Scenario: I am not notified when I upgrade a package that I upgrade a package that was already installed

Scenario: I am not notified when I uninstall a package that is not configured as additional software

This one currently fails, but should not, so it may be worth testing it:

Scenario: Packages I install with packagekit and add to ASP are automatically installed

#14 Updated by bertagaz 4 months ago

  • Target version changed from Tails_3.6 to Tails_3.7

#15 Updated by bertagaz 4 months ago

alant wrote:

As asked, I had a look at the scenarios.

I think most of the important things are covered.

Great.

I suggest adding:

Scenario: I am not notified when I install a package and I have a persistent storage but it is not unlocked

Sure is missing, good catch.

For the following two, I'm unsure. Mainly because the whole feature is adding almost an hour of run time in Jenkins, and there's still one big scenario to write. So I wonder if we should spend time, energy and resources to test such not so important use cases. Also the 'not notified' concept means the scenario will have to wait a certain amount of time without seeing the notification, which will probably add more run time. What do you think?

Scenario: I am not notified when I upgrade a package that I upgrade a package that was already installed

I'm not sure to get this one. If a user try to upgrade a package (that has probably been upgraded automatically), she should not see any notifications?

Scenario: I am not notified when I uninstall a package that is not configured as additional software

And then for this one:

This one currently fails, but should not, so it may be worth testing it:

Scenario: Packages I install with packagekit and add to ASP are automatically installed

Is it possible at the moment to use packagekit in Tails? Are we shipping a tool to use it to install a package? SOrry, I don't know much about it.

#16 Updated by bertagaz 4 months ago

bertagaz wrote:

alant wrote:

Scenario: I am not notified when I install a package and I have a persistent storage but it is not unlocked

Sure is missing, good catch.

That's 9f99bfbca21415ae8db4482c5be39bbfb50ec357

#17 Updated by alant 3 months ago

The tests are always failing (https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-14596-automated-tests-for-asp-gui/lastCompletedBuild/cucumberTestReport/), so I don't know how I can rely on them for CI of my work. Will this be fixed soon?

#18 Updated by bertagaz 2 months ago

  • Target version changed from Tails_3.7 to Tails_3.8

#19 Updated by intrigeri about 2 months ago

  • Assignee changed from anonym to bertagaz
  • QA Check changed from Ready for QA to Info Needed

(As per Alan's reply.)

#22 Updated by intrigeri 23 days ago

  • Target version changed from Tails_3.8 to Tails_3.9

#23 Updated by u 14 days ago

  • Priority changed from Normal to Elevated

bertagaz: did you push your modifications? Can you please reply to Alan's question in #17?
The release of this feature is supposed to happen in Tails 3.9, the RC is currently scheduled in 4 weeks (it might be delayed for half a week). Let's try to fix these tests during the next 2 weeks so that Alan can finish his work. Thanks!

Also available in: Atom PDF