Project

General

Profile

Feature #9519

Make the test suite more deterministic through network simulation

Added by anonym almost 3 years ago. Updated over 1 year ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Test suite
Target version:
-
Start date:
06/02/2015
Due date:
% Done:

100%

QA Check:
Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

As a Tails contributor
When Jenkins tells me that I broke something
Then I want to be able to trust it
And I won't ignore such notifications

This would eliminate all or parts of the issues we have with false test failures due to transient network errors.


Subtasks

Feature #9520: Investigate the Shadow network simulator for use in our test suiteRejected

Feature #9521: Use the chutney Tor network simulator in our test suiteResolved


Related issues

Related to Tails - Bug #9478: How to deal with transient network errors in the test suite? Resolved 05/27/2015
Related to Tails - Bug #12211: Adapt GnuPG automated tests after switching to an Onion keyserver Resolved 02/03/2017
Related to Tails - Bug #14770: "Fetching OpenPGP keys" scenarios are fragile In Progress 10/04/2017

History

#1 Updated by anonym almost 3 years ago

  • Related to Bug #9478: How to deal with transient network errors in the test suite? added

#2 Updated by anonym almost 3 years ago

Mostly copy-pasted from #9478:

Some problems with this general approach is:

When only simulating the Tor network:

  • We'd have to reconfigure the Tor client to use this fake network. All such reconfigurations are of course bad, since it makes the system under testing deviate from the real situation. However, since we already are using a different Tor network, we've deviated quite a lot already. This would be somewhat alleviated with @release tests using the real Internet + Tor network, that we only run for releases or something like that.

When simulating the Tor network and complete internet, we in addition have:

  • We will have to reconfigure many parts of Tails and run servers for at least:
    • the Tails security/upgrade check
    • incremental upgrades
    • Tor Browser home page (and other pages, since we sometimes need more than this one)
    • APT repos
    • whisperback
    • OpenPGP keyserver
    • (perhaps?) check.torproject.org
    • Gobby server
    • (soon) mail servers
  • We probably cannot do anything for I2P.

#3 Updated by anonym almost 3 years ago

For a part of the performance side of this, I just ran the full test suit (in branch test/wip-improved-snapshots, which should just be a bit better than the situation in stable) and discovered that out of the full 211 minutes it took to run the test suite, 50 minutes (so ~25%) was spent on waiting for Tor to bootstrap (either initially, or after restoring from a snapshot). This was even with some hacks to restart Tor if the bootstrap progress seemed to stall (see #9516#note-1). With a simulated network, where bootstrapping is ~instant, most of this would be eliminated.

#5 Updated by anonym almost 3 years ago

  • Parent task deleted (#8539)

intrigeri wrote:

Do we really want this to block #8539 (as a subtask, it currently does)? If you feel that it will indeed prevent us from usefully run the test suite in a CI setup, fine with me. Just keep in mind that #8539 is a deliverable with a fixed deadline :)

That was definitely a mistake. Removed!

#6 Updated by intrigeri over 2 years ago

  • Description updated (diff)

#7 Updated by BitingBird almost 2 years ago

  • Status changed from Confirmed to In Progress

#8 Updated by intrigeri 5 months ago

  • Related to Bug #12211: Adapt GnuPG automated tests after switching to an Onion keyserver added

#9 Updated by intrigeri 5 months ago

  • Related to Bug #14770: "Fetching OpenPGP keys" scenarios are fragile added

Also available in: Atom PDF