Make the test suite more deterministic through network simulation
As a Tails contributor When Jenkins tells me that I broke something Then I want to be able to trust it And I won't ignore such notifications
This would eliminate all or parts of the issues we have with false test failures due to transient network errors.
#2 Updated by anonym over 2 years ago
Mostly copy-pasted from #9478:
Some problems with this general approach is:
When only simulating the Tor network:
- We'd have to reconfigure the Tor client to use this fake network. All such reconfigurations are of course bad, since it makes the system under testing deviate from the real situation. However, since we already are using a different Tor network, we've deviated quite a lot already. This would be somewhat alleviated with
@releasetests using the real Internet + Tor network, that we only run for releases or something like that.
When simulating the Tor network and complete internet, we in addition have:
- We will have to reconfigure many parts of Tails and run servers for at least:
- the Tails security/upgrade check
- incremental upgrades
- Tor Browser home page (and other pages, since we sometimes need more than this one)
- APT repos
- OpenPGP keyserver
- (perhaps?) check.torproject.org
- Gobby server
- (soon) mail servers
- We probably cannot do anything for I2P.
#3 Updated by anonym over 2 years ago
For a part of the performance side of this, I just ran the full test suit (in branch
test/wip-improved-snapshots, which should just be a bit better than the situation in
stable) and discovered that out of the full 211 minutes it took to run the test suite, 50 minutes (so ~25%) was spent on waiting for Tor to bootstrap (either initially, or after restoring from a snapshot). This was even with some hacks to restart Tor if the bootstrap progress seemed to stall (see #9516#note-1). With a simulated network, where bootstrapping is ~instant, most of this would be eliminated.
#5 Updated by anonym over 2 years ago
- Parent task deleted (
Do we really want this to block #8539 (as a subtask, it currently does)? If you feel that it will indeed prevent us from usefully run the test suite in a CI setup, fine with me. Just keep in mind that #8539 is a deliverable with a fixed deadline :)
That was definitely a mistake. Removed!