Project

General

Profile

Bug #13541

Bug #10288: Fix newly identified issues to make our test suite more robust and faster

Tor still sometimes fails to bootstrap in the test suite

Added by bertagaz about 1 year ago. Updated about 1 month ago.

Status:
In Progress
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
-
Start date:
07/31/2017
Due date:
% Done:

20%

QA Check:
Dev Needed
Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

It's been noticed while reviewing Jenkins test suite runs failure in June (#12289) that the TorBootstrapFailure exception still happens sometimes.

There are 3 possible reasons to it, I think: a) chutney/tor sometimes feels stupid; or b) our test suite sometimes fails to manage chutney properly; or c) some unrelated event on the system prevents them from doing their job.

Next steps to investigate this would be to:

1. look at the test suite debug log and * double-check there's no explanation for the tor bootstrap failure (that's about (a) and (b)) * record the exact times when things went wrong
2. ensure we save the tor log from the Tails system when this happens, so we can see if that tor is stupid (a)
3. make the Journal persistent on isotesters so we can try to correlate such failures with system events (c).


Related issues

Related to Tails - Bug #10238: Investigate why Tor sometimes is slow at or even fails bootstrapping Rejected 09/23/2015
Related to Tails - Feature #12411: Stop restarting tor if bootstrapping stalls Resolved 03/29/2017
Related to Tails - Feature #12289: Deal with June 2017 false positive scenarios Resolved 06/05/2017 07/05/2017
Related to Tails - Bug #15799: "Additional software packages are installed even without network" test always fail in my environment Resolved 08/17/2018

Associated revisions

Revision 38307515 (diff)
Added by bertagaz about 1 year ago

Test suite: factorize Tor and Htpdate artifacts saving method.

  • Use save_failure_artifacts() coherently in the hooks exception
    catching.
  • Use software name as extension to simplify artifacts copying and
    renaming.

Refs: #13541

Revision cd27e054 (diff)
Added by bertagaz about 1 year ago

Also save systemd's journal on failure.

That can be a useful method, so let's do that for evey scenario failure.

Refs: #13541.

Revision fafd7216
Added by anonym 12 months ago

Merge remote-tracking branch 'origin/feature/13541-save-more-data-on-htpdate-or-tor-failures' into testing

Refs: #13541

Revision 2c20f922 (diff)
Added by intrigeri about 1 month ago

Test suite: make the Chutney nodes use a higher V3AuthVotingInterval to make client bootstrap more robust (refs: #13541).

bertagaz reported on https://labs.riseup.net/code/issues/15456#note-11 that this
helped him fix tor bootstrap issues in his local test suite runs, and indeed it
seems to help with mine as well. Let's not keep this local and instead make
everyone, including our Jenkins CI, benefit from this improvement.

History

#1 Updated by bertagaz about 1 year ago

  • Parent task deleted (#10288)

#2 Updated by bertagaz about 1 year ago

  • Related to Bug #10238: Investigate why Tor sometimes is slow at or even fails bootstrapping added

#3 Updated by bertagaz about 1 year ago

  • Related to Feature #12411: Stop restarting tor if bootstrapping stalls added

#4 Updated by bertagaz about 1 year ago

  • Related to Feature #12289: Deal with June 2017 false positive scenarios added

#5 Updated by bertagaz about 1 year ago

I'll paste debug logs and all later when I'll have finished #12290.

#6 Updated by bertagaz about 1 year ago

intrigeri wrote:

2. ensure we save the tor log from the Tails system when this happens, so we can see if that tor is stupid (a)

While doing #13472, I've pushed the feature/13541-save-more-data-on-htpdate-or-tor-failures branch which contains a rough implementation of that. It also saves htpdate.log on 'time has synced failures', so that's why I wanted it pushed and live in Jenkins.

#7 Updated by bertagaz about 1 year ago

  • Status changed from Confirmed to In Progress
  • Assignee set to anonym
  • Target version set to Tails_3.2
  • % Done changed from 0 to 10
  • QA Check set to Ready for QA

intrigeri wrote:

3. make the Journal persistent on isotesters so we can try to correlate such failures with system events (c).

I've pushed another commit in the branch mentioned in my previous note that does that. I've made it so that it does save the journal no matter what the failure. I think that's an interesting information we may want for a lot of reasons/cases. For example, I've done it because I wanted to have the systemd journal for #13461.

I've also pushed another commit in this branch that clean up the Tor and Htpdate logs retrieval.

I think that's something that'd be useful to get into stable and devel so that we get more useful informations from the Jenkins runs, so I'm puthing this branch RfQA. Merging it does not mean this ticket is over though so please set it back to Dev needed and no assignee if/when it's merged.

#8 Updated by bertagaz about 1 year ago

I'm attaching the systemd journal from https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-13541-save-more-data-on-htpdate-or-tor-failures/39/ which exposed this failure. The related saved Tor log is empty, it seems it didn't even started. That's the only builds from this branch that had this problem, so it's not sure it really shows the exact bug we're talking about.

#9 Updated by anonym 12 months ago

  • Assignee changed from anonym to bertagaz
  • Target version changed from Tails_3.2 to Tails_3.3
  • % Done changed from 10 to 20

I merged the feature/13541-save-more-data-on-htpdate-or-tor-failures branch, but had to push some fixups straight into testing:

892150ba5e Be more careful with remote shell usage when a scenario has failed.
0f5b0ea305 Prefer do..end over {} for multi-line blocks.
a44a711aa7 Fix indentation.
dcb30cc995 Simplify.
9f74565b80 Fix typos.
b85dbcb642 Simplify.

Please review (post-merge)!

#10 Updated by bertagaz 12 months ago

  • Target version changed from Tails_3.3 to Tails_3.5

#11 Updated by intrigeri 12 months ago

  • Parent task set to #10288

#12 Updated by bertagaz 11 months ago

  • Assignee deleted (bertagaz)
  • QA Check changed from Ready for QA to Dev Needed

anonym wrote:

I merged the feature/13541-save-more-data-on-htpdate-or-tor-failures branch, but had to push some fixups straight into testing:

Please review (post-merge)!

yay, thanks for the improvements / fixes. It's live, works and changes make sense.

#13 Updated by intrigeri 10 months ago

  • Target version deleted (Tails_3.5)

#14 Updated by intrigeri 10 months ago

While analyzing test failures on #15019 I've just seen 2 problems that I was going to naively classify as "Tor bootstrap failure", thinking "oh crap, the new Tor fails to bootstrap more often!". But then I looked at the Journal and noticed that nm-dispatcher did not react to NM connectivity change, did not start its scripts, and therefore the Tor service was not even started => no chance it ever bootstraps. I'm attaching two example Journals that display this bug.

I don't know if this is caused by the hacks we do around time sync'ing etc. in our test suite (e.g. I see a 2 hours time bump after starting NM + its dispatcher, but before it got a DHCP lease), or if it's a bug that can happen during Tails regular usage.

I suggest we check that the Tor service is actually started next time we're tempted to classify issues as "Tor fails to bootstrap": see if there are entries in the *.tor artifact and possibly look for "Starting Anonymizing overlay network for TCP" in the Journal.

#15 Updated by intrigeri 9 months ago

bertagaz wrote:

anonym wrote:

I merged the feature/13541-save-more-data-on-htpdate-or-tor-failures branch, but had to push some fixups straight into testing:

Please review (post-merge)!

yay, thanks for the improvements / fixes. It's live, works and changes make sense.

For some reason the Journal is not saved consistently, e.g. it's not saved when the "Browsing the web using the Tor Browser ǂ The persistent Tor Browser directory is usable" scenario fails this way:

try_for() timeout expired
Last ignored exception was: RuntimeError: The Dogtail script raised: SearchError: descendent of [application | Firefox]: child with name="Tails - News - Tor Browser" roleName='frame' (Timeout::Error)
./features/support/helpers/misc_helpers.rb:90:in `rescue in try_for'
./features/support/helpers/misc_helpers.rb:36:in `/^"([^"]+)" has loaded in the Tor Browser$/'
features/torified_browsing.feature:139:in `And the Tor Browser loads the startup page'

#16 Updated by intrigeri 6 months ago

On #15456#note-11 bertagaz wrote:

Hint: I see more of this kind of failure with the usual Chutney network used in the test suite. Lately, when setting V3AuthVotingInterval to something like 14400 in submodules/chutney/torrc_templates/authority.i I get close to none of this bootstrap errors. That makes sense: the consensus is not renewed very 10 seconds anymore which gives a much more stable tor network. The Tails VM Tor process gets a valid consensus at the first download attempt and bootstrap very much faster. I wanted to add a note on that on #13541 and/or #15160. It could even help for #15211 a bit (reduce crypto operations).

#17 Updated by intrigeri about 1 month ago

intrigeri wrote:

While analyzing test failures on #15019 I've just seen 2 problems that I was going to naively classify as "Tor bootstrap failure", thinking "oh crap, the new Tor fails to bootstrap more often!". But then I looked at the Journal and noticed that nm-dispatcher did not react to NM connectivity change, did not start its scripts, and therefore the Tor service was not even started => no chance it ever bootstraps.

Similarly, I've just seen one case where NM will get a DHCP lease but did not start its dispatcher scripts, so tor is not started.

#18 Updated by intrigeri about 1 month ago

  • Related to Bug #15799: "Additional software packages are installed even without network" test always fail in my environment added

#19 Updated by intrigeri about 1 month ago

  • Assignee set to intrigeri
  • Target version set to Tails_3.9
  • Feature Branch set to test/13541-more-robust-tor-bootstrap+force-all-tests

intrigeri wrote:

On #15456#note-11 bertagaz wrote:

Hint: I see more of this kind of failure with the usual Chutney network used in the test suite. Lately, when setting V3AuthVotingInterval to something like 14400 in submodules/chutney/torrc_templates/authority.i I get close to none of this bootstrap errors. That makes sense: the consensus is not renewed very 10 seconds anymore which gives a much more stable tor network. The Tails VM Tor process gets a valid consensus at the first download attempt and bootstrap very much faster.

It does seem to help here as well so I've prepared a branch.

#20 Updated by intrigeri about 1 month ago

  • Assignee deleted (intrigeri)
  • Target version deleted (Tails_3.9)
  • Feature Branch deleted (test/13541-more-robust-tor-bootstrap+force-all-tests)

Actually my branch only solves part of the problem, and not the cases when Tor is not even started, so I'll submit it on #15799 instead because that's the immediate problem it fixes.

Also available in: Atom PDF