Project

General

Profile

Bug #15557

Document workaround for partially applied automatic upgrade

Added by intrigeri 6 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
-
Target version:
Start date:
04/30/2018
Due date:
% Done:

100%

QA Check:
Feature Branch:
Type of work:
End-user documentation
Blueprint:
Starter:
Affected tool:
Upgrader

Description

As we see on #14754 sometimes an automated upgrade is partially applied and as a result the upgraded system behaves weirdly. That is occuring a lot recently according to help desk, but I've found no way to fix it yet (there's a tentative fix ready in Git, that will be used first during the 3.9→3.10 upgrade but I'm not counting on it too much).

So IMO we should document a workaround: assuming I've upgraded to 3.6.2 and my Tails does not work anymore, I should add a line that contains "3.6.2.squashfs" (without the quotes) at the end of live/Tails.module (at the root of the Tails system partition). Adjust the version number depending on what version one is upgrading to, e.g. if users have the same problem after upgrading to 3.7, they should add a line that contains "3.7.squashfs" instead. Presumably this must be done from a non-Tails system (or from another, working Tails).


Related issues

Related to Tails - Bug #10347: Troubleshooting for automatic upgrades that fail to restart Confirmed 10/07/2015
Related to Tails - Bug #14754: Keyboard and mouse do not work after upgrading to tails 3.2 Confirmed 10/01/2017
Related to Tails - Bug #11839: After automatic upgrade from Tails 2.5 to 2.6, keyboard and mouse stop working Rejected 09/24/2016
Blocks Tails - Feature #15411: Core work 2018Q2 → 2018Q3: Technical writing Confirmed 03/14/2018

Associated revisions

Revision c7177be6 (diff)
Added by cbrownstein 3 months ago

Add known issue wrt partial upgrade (closes: #15557)

History

#1 Updated by intrigeri 6 months ago

  • Blocks Feature #15411: Core work 2018Q2 → 2018Q3: Technical writing added

#2 Updated by intrigeri 6 months ago

  • Priority changed from Normal to Elevated

(Help desk say lots of users are affected.)

#3 Updated by sajolida 6 months ago

  • Description updated (diff)
  • Assignee changed from sajolida to cbrownstein

Cody: Do you think you can work on it this week?

I'll be super busy with the user testing of additional software :/

#4 Updated by sajolida 6 months ago

Definitely related to #10347. So I'm assigning it to you as well.

This one might be tough, so tell me if you need help!

#5 Updated by sajolida 6 months ago

  • Related to Bug #10347: Troubleshooting for automatic upgrades that fail to restart added

#6 Updated by bertagaz 5 months ago

  • Target version changed from Tails_3.7 to Tails_3.8

#7 Updated by cbrownstein 4 months ago

  • Assignee changed from cbrownstein to intrigeri
  • QA Check set to Info Needed

intrigeri wrote:

As we see on #14574 sometimes an automated upgrade is partially applied and as a result the upgraded system behaves weirdly.

I'm trying to understand the symptoms of a partially-applied automatic upgrade. Can "behaves weirdly" be more definite? #14574 appears to be unrelated to this ticket. Is there a correct ticket I can reference?

So IMO we should document a workaround: assuming I've upgraded to 3.6.2 and my Tails does not work anymore, I should add a line that contains "3.6.2.squashfs" (without the quotes) at the end of live/Tails.module (at the root of the Tails system partition). Adjust the version number depending on what version one is upgrading to, e.g. if users have the same problem after upgrading to 3.7, they should add a line that contains "3.7.squashfs" instead. Presumably this must be done from a non-Tails system (or from another, working Tails).

What can the user expect after this is done?

#8 Updated by intrigeri 4 months ago

  • Description updated (diff)

#9 Updated by intrigeri 4 months ago

  • Related to Bug #14754: Keyboard and mouse do not work after upgrading to tails 3.2 added

#10 Updated by intrigeri 4 months ago

  • Related to Bug #11839: After automatic upgrade from Tails 2.5 to 2.6, keyboard and mouse stop working added

#11 Updated by intrigeri 4 months ago

  • Assignee changed from intrigeri to cbrownstein
  • QA Check changed from Info Needed to Dev Needed

I'm trying to understand the symptoms of a partially-applied automatic upgrade. Can "behaves weirdly" be more definite? #14574 appears to be unrelated to this ticket. Is there a correct ticket I can reference?

Sorry, typo. The correct ticket is #14754.

tl;dr: symptoms are "some hardware that would usually work fine in Tails is not working anymore".

When an automatic upgrade is only partially applied, in some (as we've seen on #14754 and #11839), the kernel and initramfs are upgraded but the SquashFS either has no kernel modules at all for the new kernel version. So drivers that are in the initramfs can be loaded but drivers that should be in the SquashFS cannot. The consequence is that some hardware won't work. For example, user reports teach us that input devices are affected; it is not surprising technically, and it's understandable that it's the first problem users notice because without working input they can't log in and notice how other stuff is broken, but I expect that other areas of hardware support are affected as well, starting with network devices. Furthermore other aspects of Tails, besides hardware support, will be broken, e.g. networking (can't work without the kernel modules needed by our firewall), but well, in practice, who cares given their input devices and network adapters don't work anyway.

If you need more info, don't hesitate to ask :)

So IMO we should document a workaround: assuming I've upgraded to 3.6.2 and my Tails does not work anymore, I should add a line that contains "3.6.2.squashfs" (without the quotes) at the end of live/Tails.module (at the root of the Tails system partition). Adjust the version number depending on what version one is upgrading to, e.g. if users have the same problem after upgrading to 3.7, they should add a line that contains "3.7.squashfs" instead. Presumably this must be done from a non-Tails system (or from another, working Tails).

What can the user expect after this is done?

Their Tails should now be fully upgraded (respectively to 3.6.2 and 3.7 in the above two examples) so any weird/broken behaviour described above should go away.

#12 Updated by intrigeri 4 months ago

  • Target version changed from Tails_3.8 to Tails_3.9

#13 Updated by cbrownstein 3 months ago

  • Assignee changed from cbrownstein to sajolida
  • QA Check changed from Dev Needed to Ready for QA

#14 Updated by sajolida 3 months ago

  • Assignee changed from sajolida to cbrownstein
  • QA Check changed from Ready for QA to Dev Needed

Thanks for the draft!

  • Please split your two first sentences into a separate paragraph. It's your intro. When writing for the web, it's fine (and actually recommended) to do super short paragraph for scanability.
  • The system partition is flagged as hidden so it won't appear in Files in Tails and Debian and will probably be hard to mount on Windows and macOS. I suggest instructing people to do that through GNOME Disks (yeah, it's a pain but that's the best I can think of). We could say that this is possible either from Tails or from Debian/Ubuntu.
  • Regarding "For security reasons, you should not mount your persistent partition if you have one!", if the system on which you apply the workaround is untrusted, the same could apply to the system partition: an untrusted system do insert some malicious code in the Tails system from the system partition. So I wouldn't write anything specific for the persistence partition. I'm not even sure it's worth specifiying anything as people on Debian/Ubuntu probably used their system to install Tails in the first place.
  • Please format the instructions and a numbered list of steps. Once we added GNOME Disks to the dance that will really become necessary.
  • I would try to apply the command line formatting that we have, see #15120, to describe the line that needs to be added to the file. For example, for the placeholder part of the line ("Replace x.x.").

#15 Updated by cbrownstein 3 months ago

  • Assignee changed from cbrownstein to sajolida
  • QA Check changed from Dev Needed to Info Needed
  • Please split your two first sentences into a separate paragraph. It's your intro. When writing for the web, it's fine (and actually recommended) to do super short paragraph for scanability.

Done.

  • The system partition is flagged as hidden so it won't appear in Files in Tails and Debian and will probably be hard to mount on Windows and macOS. I suggest instructing people to do that through GNOME Disks (yeah, it's a pain but that's the best I can think of). We could say that this is possible either from Tails or from Debian/Ubuntu.

Any suggestions as to how detailed the GNOME Disks instructions should be? I'll also have to add instructions for what to do after the partition is mounted.

  • Regarding "For security reasons, you should not mount your persistent partition if you have one!", if the system on which you apply the workaround is untrusted, the same could apply to the system partition: an untrusted system do insert some malicious code in the Tails system from the system partition. So I wouldn't write anything specific for the persistence partition. I'm not even sure it's worth specifiying anything as people on Debian/Ubuntu probably used their system to install Tails in the first place.

I'm deleting the entire sentence for the reasons you've stated.

  • Please format the instructions and a numbered list of steps. Once we added GNOME Disks to the dance that will really become necessary.

Will do.

  • I would try to apply the command line formatting that we have, see #15120, to describe the line that needs to be added to the file. For example, for the placeholder part of the line ("Replace x.x.").

Will do.

#16 Updated by sajolida 3 months ago

  • Assignee changed from sajolida to cbrownstein
  • QA Check changed from Info Needed to Dev Needed

We rediscussed this last week and Cody should have all the info to come up with a new version.

#17 Updated by cbrownstein 3 months ago

  • Assignee changed from cbrownstein to sajolida
  • QA Check changed from Dev Needed to Ready for QA

I've rewritten the instructions to include usage of GNOME Disks.

#18 Updated by sajolida 3 months ago

  • Assignee changed from sajolida to cbrownstein

Cool! I think it's the right level of details and the problem is defined as good as possible. "Behaves weirdly" still sound very vague but that's the best we can do I think.

I pushed a bunch of small fixes on doc/15557-partial-automatic-upgrade. Please have a look and then I can merge.

I didn't test any of this but from some details in your instructions I understand that you did test that :)

#19 Updated by cbrownstein 3 months ago

  • Assignee changed from cbrownstein to sajolida

"Plug your Tails USB stick that behaves weirdly."

Can we change that sentence? The first verb definition of plug that comes to my mind is, "to stop, make tight, or secure by inserting a plug."[1] Obviously that is not the definition of plug that we mean.

I suggest: "Plug in your Tails USB stick that behaves weirdly." I have made that change in changeset eefb8e3afa. That is consistent with existing documentation, e.g., install instructions.

Other than that, my changes are small.

[1] https://www.merriam-webster.com/dictionary/plug

#20 Updated by cbrownstein 2 months ago

  • Status changed from Confirmed to Resolved
  • % Done changed from 0 to 100

#21 Updated by intrigeri 2 months ago

  • Status changed from Resolved to In Progress

(Apparently this was not merged but a buggy "Closes" in a commit message mislead Redmine into closing this ticket.)

#22 Updated by sajolida 2 months ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (sajolida)
  • QA Check deleted (Ready for QA)

I definitely missed 'in' in 'plug in'. Thanks for spotting that!

I merged your branch and this ticket is now solved.

Yeah!

Also available in: Atom PDF