Project

General

Profile

Feature #6452

Factor out stuff into a Tails Python library

Added by anonym almost 5 years ago. Updated 3 months ago.

Status:
Confirmed
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
11/29/2013
Due date:
% Done:

0%

QA Check:
Feature Branch:
Type of work:
Code
Blueprint:
Starter:
No
Affected tool:

Description

There's some code duplication between at least Tails Greeter (feature/spoof-mac branch) and WhisperBack which should be eliminated by being refactored into a shared Tails Python library:

  • the cb_request_starting() callback used for the help browser
  • probably more of the help browser setup code (since it was pretty much copy-pasted from WhisperBack into TG)
  • __get_localised_doc_link() from WhisperBack (isn't in TG yet, but will at some point)

Related issues

Related to Tails - Feature #6786: Be consistent when checking if persistence is enabled in tails-additional-software In Progress 02/27/2014
Related to Tails - Feature #6145: Factor out stuff into a shell library Confirmed 11/14/2013
Related to Tails - Feature #9372: Use 'set -e' and 'set -u' in every shell script where reasonably doable In Progress 05/11/2015
Related to Tails - Feature #11198: Port complex shell scripts into Python In Progress 08/27/2016
Related to Tails - Feature #6525: Localise help links in Tails Greeter Confirmed 12/20/2013

Associated revisions

Revision 059f3ca3 (diff)
Added by anonym over 2 years ago

Install the tailslib python library.

This library contains useful stuff for our effort of converting shell
scripts into python.

Refs: #6452

Revision cfd31098 (diff)
Added by anonym over 2 years ago

Install python3-sh.

This library provides nice shell-like facilities, and an easy
interface to calling commandline tools, and so will be highly useful
in our effort to convert shell scripts into python.

Refs: #6452

Revision 77c250d8 (diff)
Added by anonym over 2 years ago

Convert 19-install-tor-browser-AppArmor-profile into python.

An example conversion of one of our shell scripts into python.

Refs: #6452

History

#1 Updated by intrigeri almost 5 years ago

  • Status changed from New to Confirmed

#2 Updated by alant over 4 years ago

  • Target version set to Sustainability_M1

#3 Updated by intrigeri over 4 years ago

  • Subject changed from Tails Python library to Factor out stuff into a Tails Python library

#4 Updated by intrigeri over 4 years ago

  • Related to Feature #6145: Factor out stuff into a shell library added

#5 Updated by intrigeri over 3 years ago

  • Blocks deleted (Feature #6525: Localise help links in Tails Greeter)

#6 Updated by intrigeri over 3 years ago

We now have a Python (3) library: https://git-tails.immerda.ch/pythonlib. But it's not installed in Tails yet, and only used by infrastructure bits and during the release process.

#7 Updated by intrigeri about 3 years ago

kurono volunteered to help. I'll let you folks discuss it.

#8 Updated by anonym about 3 years ago

I have a few additional ideas for such a Python library (plus steps towards a grander Python vision for Tails in general, see below). For starters, parts of the Shell library should migrate into the Python library, so the shell functions are thin wrappers around the Python library. The priority should be:

  • Anything that heavily uses lists! (Think: whitespace issues in the shell)
  • Anything complex should move there, e.g. hardware.sh:mod_rev_dep() (recursion in the shell? why not, but please no :)), and I think we can get common.sh:wait_until()/try_for() functions implemented in python with real timeouts, instead of the "sort of"-timeouts we have now.
  • Many things in tor.sh can be implemented reliably and trivially via stem.
  • Python probably has proper localization stuff, instead of our sed hack in localization.sh.

However, maybe we can convert everything in the Shell library, and even aim for replacing more or less all our Shell scripts in the future? There are several Python modules that could make this easier for us. We could use e.g. plumbum for this:

  • It can import any command-line tool in your $PATH and integrate it into python like a function. It's super easy to extract the exit code, stdout and stderr. There's a simple interface to use such imported commands through Popen too, when needed. There's a simple interface to use such imported commands through Popen too, when needed. All this works so well that I'm afraid we might continue to use the Shell tools instead of the more appropriate Python equivalents. :)
  • It provides simple ways to do piping, backgrounding and similar convenient shell concepts.
  • We get the equivalent of set -eu by default, and can work with it more easily since the program won't exit, but throw an exception, so exception handling will make things much more easier for us.
  • It makes command nesting with sudo, ssh etc much simpler and clearer than in the shell.

Overall, plumbum is quite brilliant. :) There are also alternatives, like sh, scriptine and so on, but I do not know what is best for our needs (I suspect plumbum, though). Also, for more complex subprocess interaction we may want to look at e.g. sarge as a replacement for the standard (but IMHO awkward) subprocess Python module.

I'm digressing, a bit. I guess we really need another ticket. :)

#9 Updated by intrigeri about 3 years ago

Overall, plumbum is quite brilliant. :) There are also alternatives, like sh,

FYI we already use sh in our Python lib.

#10 Updated by anonym about 3 years ago

intrigeri wrote:

Overall, plumbum is quite brilliant. :) There are also alternatives, like sh,

FYI we already use sh in our Python lib.

After a brief look, it looks good, too (and indeed very similar; apparently plumbum has sh as an inspiration). A major difference is how piping works:

  • Normal shell: ls -1 | wc -l
  • Python plumbum: (ls['-1'] | wc['-l'])() or chain = ls['-1'] | wc['-l']; chain()
  • Python sh: wc(ls('-1'), '-l')

So while plumbum works more similarly to the shell, the (...)() stuff is ugly. The sh way, i.e. function composition, is actually how it'd be done in most programming languages, which might be a good thing: instead of making us thing that we are "writing shell scripts in python", it may promote the feeling that we are "writing python scripts with an easy way to invoke the shell". ;)

Any way, most of the time we use pipes in the shell it's for text processing, and Python has much better facilities for that. On the top of my head, we do some piping into nc for control port stuff, but we should use stem any way. I'm sure there's more but, again, I digress.

#11 Updated by alant almost 3 years ago

intrigeri wrote:

kurono volunteered to help. I'll let you folks discuss it.

kurono, feel free to ask any question or start working on this. You can assign the ticket to you if you want. I'll be happy to give any help you wish and to review your work. I'll try not to let you wait for a whole week, but don't expect me to answer daily.

#12 Updated by kurono almost 3 years ago

  • Assignee changed from alant to kurono

#13 Updated by intrigeri over 2 years ago

  • Related to Feature #9372: Use 'set -e' and 'set -u' in every shell script where reasonably doable added

#14 Updated by intrigeri over 2 years ago

This is one of the last two remaining tickets on the obsolete "Sustainability_M1" (that doesn't mean much anymore, since we have our new roadmap). kurono, please consider moving this ticket to e.g. the 2016 or 2017 milestone, or just emptying the Target version field entirely.

#15 Updated by kurono over 2 years ago

  • Target version changed from Sustainability_M1 to 2016

#16 Updated by anonym over 2 years ago

  • Feature Branch set to feature/6452-python-scripting

Here's an example conversion. Perhaps it can act as an inspiration, kurono?

#17 Updated by intrigeri over 2 years ago

Note that another idea (for build -time stuff) is to migrate to Ansible, so perhaps converting run -time stuff to Python is more likely to be useful on the long run. But of course, at runtime we need to be more careful about memory usage.

#18 Updated by anonym over 2 years ago

  • Related to Feature #11198: Port complex shell scripts into Python added

#19 Updated by anonym over 2 years ago

  • Feature Branch deleted (feature/6452-python-scripting)

I'm not sure whether this should be a child of #11198 or just related.

#20 Updated by intrigeri about 2 years ago

  • Target version deleted (2016)

This got removed from our roadmap, but it still would be very welcome! Feel free to set whatever target version you want :)

#21 Updated by kurono 3 months ago

  • Target version set to 2019

#22 Updated by intrigeri 3 months ago

  • Target version changed from 2019 to Tails_4.0

(This is not on our roadmap, so I'm setting "Target version" to the latest 2019 release we're already tracking.)

#23 Updated by u about 2 months ago

  • Related to Feature #6525: Localise help links in Tails Greeter added

Also available in: Atom PDF