Project

General

Profile

Feature #7496

Make it possible to verify the integrity of a device created by Tails Installer

Added by alant over 3 years ago. Updated about 1 month ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
07/06/2014
Due date:
% Done:

50%

QA Check:
Ready for QA
Feature Branch:
Type of work:
Code
Blueprint:
Easy:
No
Affected tool:

Description

It would be useful to be able to verify the integrity of the system created by Tails Installer.

This is not straightforward because it implies verifying not only the content of the Tails system partition but also the MBR and partition tables.

team: segfault, anonym+intrigeri, jvoisin, sycamoreone

tails-verifier.tar.gz (2.41 KB) segfault, 12/08/2015 12:52 PM


Related issues

Related to Tails - Feature #7269: Explain why it is complicated to propose Tails devices for sale Resolved
Related to Tails - Bug #7552: Firefox extension to automatically verify the ISO checksum Resolved 01/06/2015
Related to Tails - Feature #8510: Distribute a hybrid ISO image Resolved 01/03/2015
Related to Tails - Feature #11679: Rethink the installation process and upgrade process Confirmed 08/20/2016
Blocked by Tails - Feature #8389: Make the full Tails ISO history available to core contributors Resolved 12/04/2014

History

#1 Updated by alant over 3 years ago

  • Related to Feature #7269: Explain why it is complicated to propose Tails devices for sale added

#2 Updated by alant over 3 years ago

From what system do we want to verify the installed device? If it is Tails, then why not to clone and upgrade? So want to verify it from another OS (typically Windows).

There are then several set of data that should be checked:

  • most of them can be generated during build and should be retrived securely somehow
  • there a syslinux autogenerated file in the filesystem (ldlinux.sys) that is not possible to compare to a trusted source

#3 Updated by sajolida over 3 years ago

From what system do we want to verify the installed device? If it is Tails, then why not to clone and upgrade? So want to verify it from another OS (typically Windows).

Other usecases would be:

Verifying that you are using a genuine Tails key:

- You start using Tails by getting a USB key from an organization or
online shop.
- After some time using it, liking it, and getting to know other
Tails users you want to make sure it was genuine.

Checking the where people distributing Tails keys are trustworthy. The
same technique could be used to verify the keys sold or made by other
organization and checking whether they are trustworthy.

#4 Updated by intrigeri about 3 years ago

  • Related to Bug #7552: Firefox extension to automatically verify the ISO checksum added

#5 Updated by ioerror almost 3 years ago

I've created a start to this for a Debian system - this will hopefully, eventually, run on Tails too:

https://github.com/ioerror/tails-verifier/blob/master/README.VERIFICATION

I need help though - I need a collection of every GnuPG signature for every release, as well as a copy of every ISO or every Torrent file.

Could someone please share these with me?

#6 Updated by intrigeri almost 3 years ago

  • Blocked by Feature #8389: Make the full Tails ISO history available to core contributors added

#7 Updated by intrigeri almost 3 years ago

ioerror wrote:

I need help though - I need a collection of every GnuPG signature for every release, as well as a copy of every ISO or every Torrent file.

Could someone please share these with me?

For the ISO and detached signatures, it's WIP: #8389. For the torrent files, you can find them in Git history.

I'm curious how you're dealing with ldlinux.sys, but no time to check the code right now.

#8 Updated by intrigeri almost 3 years ago

  • Status changed from Confirmed to In Progress
  • Assignee set to ioerror

intrigeri wrote:

ioerror wrote:

I need help though - I need a collection of every GnuPG signature for every release, as well as a copy of every ISO or every Torrent file.

Could someone please share these with me?

For the ISO and detached signatures, it's WIP: #8389.

Done, sent credentials in a private email. Enjoy!

#9 Updated by intrigeri almost 3 years ago

#10 Updated by intrigeri almost 3 years ago

FWIW, once we ship a hybrid ISO image again (#8510), it should be much easier to bit-by-bit verify a stick installed manually (as opposed to with Tails Installer).

#11 Updated by intrigeri almost 3 years ago

Next step regarding ldlinux.sys: install Tails twice in a row from the same medium, onto the same (other) medium, and compare that file on these two installations. Then, think.

#12 Updated by intrigeri over 2 years ago

hashrat (https://bugs.debian.org/776499) may be useful.

#13 Updated by sajolida about 2 years ago

  • Target version set to 2017

#14 Updated by jelhan about 2 years ago

As integrity mentioned verifying device integrity seems to be easy since a hybrid ISO image is skipped. I got correct sha256 of a Tails 1.6 which was manually installed on an USB stick.

$ sudo dd if=tails-i386-1.6.iso of=/dev/sdc && sync
$ stat tails-i386-1.6.iso
File: 'tails-i386-1.6.iso'
Size: 987033600 Blocks: 1927824 IO Block: 4096 regular file
$ sudo dd if=/dev/sdc bs=1 count=987033600 | sha256sum
6daba429ec9372af0822d8dd4c3e96407240a31c9373173a5985d8d669ea1d00 -
$ sha256sum tails-i386-1.6.iso
6daba429ec9372af0822d8dd4c3e96407240a31c9373173a5985d8d669ea1d00 tails-i386-1.6.iso

Just reading in the device with a block size of 1 bit really takes it time. But I think there should be a better solution than dd bs=1.

#15 Updated by intrigeri about 2 years ago

As integrity mentioned verifying device integrity seems to be easy since a hybrid ISO image is skipped.

Looks like you missed the "created by Tails Installer" aspect of this ticket.

#16 Updated by anonym almost 2 years ago

I'm wondering if we can rethink incremental upgrades and leverage Grub2's ISOBoot functionality to get this and a lot of other very nice things. Please bear with me and imagine something like this:

  • We kill the IUK concept. We may still be able to support incremental upgrades (see below) but for the time being, think that full upgrades are the only way to upgrade Tails. Note that we definitely sill will support automatic upgrades (full upgrades, #7499).
  • The partition table is made very simple to ease verification. The only variable should be the size (and perhaps some uuid?) of the TailsData partition, and/or whether these is one at all; everything else on the partition table level is static and identical for all Tails installations. I'm no expert at these things, but this is possible, right? Any way, we publish some description of this + sign it.
  • We have a tiny $BOOT_PARTITION for Grub2. We generate a blob that has it installed at build time + sign it, and ship it together with the Installer. It can be installed and extracted (for verification) easily with dd or whatever. Grub2 is configured to boot some specific ISO from...
  • ... another partition, $SYSTEM_PARTITION, storing ISOs (note: plural; see below).
  • On the $SYSTEM_PARTITION we store everything needed to verify all pieces:
    • the description of the partition table + signature
    • the signature of the $BOOT_PARTITION
    • the signatures of all ISOs in $SYSTEM_PARTITION

Given that, each installation includes signatures, descriptions and hashes so all bits of the installation can be verified by an external tool with access to our signing key (and, I guess, the blob for the $BOOT_PARTITION; we could publish these so the verification tool can download them automatically).

Bonus: now we can support upgrading a USB/SD device while it is running; we just need the ISO some how. Alternatives:

  • The Upgrader could download the ISO for us (#7499). It would also download the signature (or UDF) for us and verify it. So we'd still have automatic upgrades, like promised above.
  • Users could manually get hold of the ISO (like our Installer currently supports). I suppose we could offer to download the signature (or UDF) and verify it, like above.
  • The ISO can be extracted from a DVD with Tails burned to it. We just need to know its exact length which we can encode in the ISO metadata, or possibly even check in the UDF, which we can download. We can also offer to verify it like above, or using the UDF.
  • The ISO copied from another USB installation. We would also extract the signatures, descriptors etc so it's verified.

Now we just install the new ISO (and/or IUKs) into $SYSTEM_PARTITION, leaving the old ISO (and/or IUKs) there, since it's mounted and in use. But only for the time being, cause we add a hook that deletes old ISOs (and old IUKs, signatures etc etc) during boot to recover the space.

So far we've only talked about the scenario where we boot from a simple "ISO", whether it's burned to a DVD, or loop-mounted (by Grub2). There are no incremental upgrades. This is a good thing:

  • Code and design complexity would go down in several areas (in particular, no aufs/overlaysfs stacking is involved)
  • Easier and faster release process since there are no IUKs
  • It's just easier and more intuitive to reason about Tails' filesystem

So what about incremental upgrades? Well, we could get them, and also support them endlessly (#8534) if we manage to figure out how to use binary diffs. We'd probably want a format that is optimized for ISOs to make them more compact, so our experience with the IUK format will probably help. Note that if we want to keep the "always boot from ISO" concept, then this is the only way we can do incremental upgrades.

The big problem with binary diffs of large files (ISOs) are the space requirements when applying them (we have access to monster machines to generate them, of course). That may make the above approach effectively impossible unless we can use a lot of space temporarily, either in RAM, in $SYSTEM_PARTITION or on TailsData (if none is used, we could temporarily create a partition there and use the space). Making $SYSTEM_PARTITION large just for this is sad, since it eats from TailsData's space. This may be hard, and introduce a lot of "ifs", complexity and potential user confusion.

Actually, since we have #7499, the only advantage of incremental upgrades is that it requires downloading less when upgrading. Most of the time IUKs are ~250 MiB, and ISOs are ~1 GiB, so there's a 300% increase. Also note that we probably can optimize our IUKs more, especially after we have deterministic builds, so the loss of efficiency of doing this might be even larger. Still, this simplification (only support full upgrades) might be worth considering.

To end with, I know I have been digressing far and wide, but I think it'd be nice if we designed something that will solve #7496 and #8534 at the same time, or if we cannot have #8534, that it solves #7499. So there is a broader discussion for us to have, which I hope that I now have started a bit.

#17 Updated by segfault almost 2 years ago

I suggest solving #7499 together with #7878, since this would reduce the time the user has to wait for the ~1GiB download, therefore improving UX (a lot).

#18 Updated by sajolida almost 2 years ago

Thanks for brainstorming on this. I find it useful when people have "crazy" ideas that look a bit further than the next release sometimes :)

I'm not against the idea of killing IUKs if the cost/benefit is good enough. Right now the benefits you're outlining are #7496 (this ticket), a bit less RM work, a simplier file system (cf. recent issues while upgrading, etc) and I'm not sure they are worth it at the moment. But that's not a good reason to not investigate your idea a bit further!

A few more related comments:

  • Downloading through Tor is on average not that much slower than outside of Tor. Let's say less than twice slower in most cases (see #9320#change-39626).
  • I'd love to have stats about what upgrade paths our users follow. For example, we're only providing IUKs from the last version. This assumes that our users are all regular users that update every 6 weeks. Confronting this with real data might change our vision a lot. Maybe the answer would be to ship IUK for n-1 and n-2, or to go for your solution. I'll try to get such stats, see #10478.

#19 Updated by segfault almost 2 years ago

  • File tails-verifier.tar.gz added

I wrote a script to verify a Tails device using a Tails ISO.
I'm using the hashrat package from Debian testing (https://packages.debian.org/stretch/hashrat) to create hashes of all the files in the ISO and compare them to the files on the device.
I tested it with tails-i386-1.5.1.iso and tails-i386-1.7.iso. I used the Tails installer's "Upgrade from ISO" method to install the ISO. It verifies correctly after excluding "squashfs/var/lib/urandom/random-seed" and replacing "boot/isolinux" with "boot/syslinux" in the hashfile.

I also tested if it still verifies correctly after booting the device and it does.

It is currently not checked if there are any new files on the device, which are not in the ISO.

Please take a look at the code and tell me what you think about it.
Should I put this in a fork of Tails or somewhere else?

#20 Updated by segfault almost 2 years ago

  • File tails-verifier.tar.gz added

Put the files inside the tar in a directory.

#21 Updated by sajolida almost 2 years ago

ioerror: are you volunteering to test and review this script or shall we look for somebody else? If so, shall we set the target version to something sooner than "2017". See https://tails.boum.org/contribute/calendar/ for time estimates of the upcoming versions.

#22 Updated by segfault almost 2 years ago

  • File tails-verifier.tar.gz added

I fixed two minor issues (printing a warning if hashrat is not found and counting 'failures' in the hashrat output as well as 'errors'. The second could have resulted in the output "Successfully verified" while the verification actually failed).

I also took a look at the code of hashrat to inspect what would have to be done to check for additional files. I found it very bloated with 22000 lines of C code. It doesn't seem to have many users, the only issue on github was created by me. And it is not very well maintained, the very simple issue is still unfixed after almost 3 months.
So I think about writing a python script to replace hashrat. But I know you prefer including Packages from upstream rather than writing code for Tails.
What do you think about this?

#23 Updated by segfault almost 2 years ago

I added functionality to check for additional files on the device and to check the MBR.

Now the only remaining problem I see is verifying ldlinux.sys and verifying the VBR. The ldlinux.sys files on my devices differ only in a few bytes, but I was not (yet) able to find out why they differ.

#24 Updated by u almost 2 years ago

  • Assignee changed from ioerror to segfault

Assigning this ticket to segfault who has been working on it and has a working PoC.

Segfault, you can ask on tails-dev for reviewing this as is or redo it in Python as you planned before of you wish to do so.

Cheers!

#25 Updated by segfault over 1 year ago

  • % Done changed from 0 to 80
  • Type of work changed from Research to Security Audit

I put my work on the Tails Verifier on gitlab:
https://gitlab.com/segfault_/tails_verifier

I realized half way through that this project was getting far too complex and it would be better to focus on changing the Tails installation and upgrade process in a way that makes verification easier (which I intend to work on). But I finished it anyway and it's working for me.

I set the type of work to security audit, although currently I don't think we should ship this, because it will be much easier, safer and more stable if we manage to simplify the installation and upgrade process.

#26 Updated by sajolida over 1 year ago

  • Assignee deleted (segfault)
  • Target version changed from 2017 to Tails_2.2

Congrats! Since you're saying that your work is finished I'm marking this as "QA Check: Ready for QA". I'm also unassigning it from you since someone else should do the review. I'll also set it for 2.2, just to make sure we identify someone to review this during the next monthly meeting. But this doesn't mean this will be in 2.2 :)

#27 Updated by segfault over 1 year ago

  • File deleted (tails-verifier.tar.gz)

#28 Updated by segfault over 1 year ago

  • File deleted (tails-verifier.tar.gz)

#29 Updated by segfault over 1 year ago

  • File deleted (tails-verifier.tar.gz)

#30 Updated by intrigeri over 1 year ago

  • QA Check set to Ready for QA
  • Type of work changed from Security Audit to Code

sajolida wrote:

Congrats! Since you're saying that your work is finished I'm marking this as "QA Check: Ready for QA".

Now done for real.

#31 Updated by intrigeri over 1 year ago

  • Assignee set to anonym

(Ready for QA without assignee hasn't worked too well lately)

#32 Updated by anonym over 1 year ago

  • % Done changed from 80 to 50

segfault wrote:

I put my work on the Tails Verifier on gitlab:
https://gitlab.com/segfault_/tails_verifier

Impressive! This looks like the largest, by far, code contribution made to Tails! :) My initial look is very positive (code looks clean and nicely modularized). Yay! This makes me excited, but also sad since I won't have time to look at it until after Tails 2.2 is out. Don't hesitate to ping me in mid-March if you haven't seen any activity from me on this ticket.

I realized half way through that this project was getting far too complex and it would be better to focus on changing the Tails installation and upgrade process in a way that makes verification easier (which I intend to work on).

I'm very much interested in what your thoughts are on this, and if you have any opinions of my ideas in #7496#note-16 above. Perhaps you have some wild ideas of your own?

But I finished it anyway and it's working for me.

I set the type of work to security audit, although currently I don't think we should ship this, because it will be much easier, safer and more stable if we manage to simplify the installation and upgrade process.

Still, it seems quite a bit of the complexity stems from the vbr and ldlinux verification, which we might remain relevant even with a new installation paradigm, and the same applies to some other parts of the code.

[Also, I've set the progress to 50% to leave some room for back-and-forth:s later.]

#33 Updated by anonym over 1 year ago

  • Target version changed from Tails_2.2 to Tails_2.3

#34 Updated by segfault over 1 year ago

Impressive! This looks like the largest, by far, code contribution made to Tails! :) My initial look is very positive (code looks clean and nicely modularized). Yay!

Thanks :)

I'm very much interested in what your thoughts are on this, and if you have any opinions of my ideas in #7496#note-16 above. Perhaps you have some wild ideas of your own?

I have opinions on your ideas and some ideas of my own, but nothing perfect yet. I think I will create a blueprint about this, where we can balance pros and cons of the ideas.

Still, it seems quite a bit of the complexity stems from the vbr and ldlinux verification

indeed

which we might remain relevant even with a new installation paradigm

I think that these parts could actually become irrelevant if we would use only grub as the bootloader instead of syslinux (which could be done easier and faster than reworking the whole installation paradigm).

#35 Updated by intrigeri over 1 year ago

I think that these parts could actually become irrelevant if we would use only grub as the bootloader instead of syslinux (which could be done easier and faster than reworking the whole installation paradigm).

... and is something I've been forcing myself to ignore quite a few times already, but I must say I'm very much tempted! (for USB installations, I mean; on DVD I see no reason to move away from isolinux)

#36 Updated by anonym over 1 year ago

I've opened a ticket about "Endless automatic upgrades", #11131, and a blueprint (which can be edited by anyone using the web interface) that includes installation verification as part of the things to consider. I guess this will be a better place to continue the discussion.

#37 Updated by anonym over 1 year ago

  • Target version changed from Tails_2.3 to Tails_2.4

#38 Updated by anonym over 1 year ago

  • Target version changed from Tails_2.4 to Tails_2.6

Sorry, segfault, but it seems incredibly unlikely I'll have the time to look at this for a while. Really, I am sorry.

#39 Updated by Dr_Whax about 1 year ago

  • Description updated (diff)
  • Target version changed from Tails_2.6 to 2017

#40 Updated by intrigeri about 1 year ago

  • Target version changed from 2017 to Tails_2.6

(Indeed we added it to our roadmap, but now that the code is supposedly ready, let's make sure this is on the reviewers' radar.)

#41 Updated by anonym about 1 year ago

  • Target version changed from Tails_2.6 to Tails_2.7

Sorry, segfault...

#42 Updated by bertagaz 11 months ago

  • Target version changed from Tails_2.7 to Tails_2.9.1

#43 Updated by anonym 10 months ago

  • Target version changed from Tails_2.9.1 to Tails 2.10

#44 Updated by anonym 9 months ago

  • Target version changed from Tails 2.10 to Tails_2.11

#45 Updated by anonym 8 months ago

  • Target version changed from Tails_2.11 to Tails_2.12

#46 Updated by anonym 6 months ago

  • Target version changed from Tails_2.12 to Tails_3.0

#47 Updated by intrigeri 5 months ago

  • Target version changed from Tails_3.0 to Tails_3.3

#48 Updated by intrigeri 5 months ago

  • Target version changed from Tails_3.3 to Tails_3.2

#49 Updated by intrigeri about 1 month ago

  • Target version changed from Tails_3.2 to Tails_3.6

I think this should be put on hold until the #11679 sprint: it's been ready for QA since 1.5 years, so it can as well wait a bit more until we know how future-proof the current implementation is.

#50 Updated by intrigeri about 1 month ago

  • Related to Feature #11679: Rethink the installation process and upgrade process added

Also available in: Atom PDF