Project

General

Profile

Feature #8054

Add support for SOCKS proxy to check-mirrors

Added by sajolida about 3 years ago. Updated 18 days ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
Start date:
10/12/2014
Due date:
% Done:

75%

QA Check:
Ready for QA
Feature Branch:
Type of work:
Code
Blueprint:
Easy:
Affected tool:
check-mirrors

Description

So far check-mirror is run on lizard but if we want to be able to run it on our machines, it would be useful to allow using it through a proxy server for all network operations.

To do that we can try the ruby-em-socksify library.


Subtasks

Feature #8074: Test check-mirrors with torsocksResolvedsajolida

Feature #8437: Test whether we could use curl for all HTTP requests in check-mirrorsConfirmed

Feature #9548: Have torsocks 2.1 in jessie-backportsResolved

Feature #9549: Have torsocks 2.1 in wheezy-backports-sloppyRejected


Related issues

Related to Tails - Bug #11736: torsocks complains when used with Monkeysign, but works anyway Resolved 08/27/2016
Blocks Tails - Feature #7667: Create a public repo for check-mirrors Confirmed

History

#1 Updated by intrigeri about 3 years ago

wrote (10 Oct 2014 08:06:49 GMT) :

So far check-mirror is run on lizard but if we want to be able to run it on our
machines, it would be useful to allow using it through a proxy server for all
network operations.

IIRC, most of check-mirror's features work when run with torsocks. If some features don't, maybe fixing them to work with torsocks would be cheaper than adding SOCKS proxy support to the script?

#2 Updated by sajolida almost 3 years ago

  • Assignee set to sajolida

I found two libraries to do SOCKS with Ruby:

So for the time being, I'll give a try at socksify.

#3 Updated by sajolida almost 3 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from sajolida to intrigeri
  • QA Check set to Ready for QA

Use simple HTTP proxy usually involves caching and this is no good in our situation.

So I implemented this in commit 08d3364 using socksify optionnally.

#4 Updated by intrigeri almost 3 years ago

  • Assignee changed from intrigeri to sajolida
  • QA Check changed from Ready for QA to Info Needed

Having to rely on a 3rd-party library just to add SOCKS support for HTTP requests feels, well, wow. I see two alternatives:

  • Using Ruby binding around Curl, e.g. ruby-curb or ruby-ethon (both are in Jessie but not in Wheezy). Requiring Jessie feels better on the mid-term than requiring a 3rd-party library. I could look into a backport if needed.
  • Worst case, if Ruby support for our needs in Debian is that poor, then we could turn the http_request function into a wrapper around the curl or wget binary, whichever better supports what http_request does.

Now, it depends on who wants to run this script in which environment: do we want to support running it in Tails/Wheezy? Or is it OK to require Jessie?

#5 Updated by intrigeri almost 3 years ago

Also, this seems to be inconsistent:

  opts.on('-o', '--torify [PORT]', Integer,
          "Use HOST:PORT as HTTP proxy") do |port|

#6 Updated by sajolida almost 3 years ago

  • Assignee changed from sajolida to intrigeri

First, I feel the need to give more background to the work I did on check-mirrors the other day. As you know already, I don't code really often because I'm not good at it, so it becomes quite painful, and I prefer doing things that are more interesting and more successfully to me.

Still, the work I did on Friday fulfills two important objectives to me:

- Being able to publish that code. To hopefully stop being the only one working on it. I'd rather have this taken over by someone else than continue patching it myself wilding forever. The good thing about it is that it is really self-contained. This is now made possible by 801b6eb.

That said, replacing http_request and wget by some curl binding is a good idea and would probably simplify the code greatly. That's now #8437.

- Being able for me, because I'm at the same time managing the pool of mirrors, to check mirrors from my own system (Tails as of now). This is made possible by this crappy SOCKS support. Also note that the code as of now doesn't require this external library if you don't want to use SOCKS. For example, this code can be run on our server without that library and work as usual.

So, my main priority now is to solve #7667. If it is not acceptable for you to have my fix for #8054 in the master branch, then I can put it somewhere else and apply it only locally. I really don't mind.

#7 Updated by sajolida almost 3 years ago

Also, this seems to be inconsistent:

>   opts.on('-o', '--torify [PORT]', Integer,
>           "Use HOST:PORT as HTTP proxy") do |port|
> 

Fixed with 3232337.

#8 Updated by intrigeri almost 3 years ago

Still, the work I did on Friday fulfills two important objectives to me: [...]

Yay \o/

So, my main priority now is to solve #7667.

ACK.

If it is not acceptable for you to have my fix for #8054 in the master branch, [...]

It's OK. I just don't want to call this ticket marked as resolved in the current state of things. Going to make #8437 a subtask of it, then.

#9 Updated by intrigeri almost 3 years ago

  • Assignee deleted (intrigeri)
  • QA Check changed from Info Needed to Dev Needed

#10 Updated by BitingBird almost 3 years ago

  • Category changed from 214 to Infrastructure
  • Affected tool set to check-mirrors

#11 Updated by sajolida over 2 years ago

I tried again to run check-mirrors on 1.3 with torsocks2 and without the custom library and it seems to work. Except that I didn't manage to do the DNS request through 127.0.0.2 in order to have the full list of IPs.

I get an "Operation not permitted" from torsocks when running either:

  • torsocks ruby check-mirrors.rb --debug --fast
  • torsocks host dl.amnesia.boum.org 127.0.0.2
  • torsocks host dl.amnesia.boum.org

Is there a way to either ask torsocks to do such kind of queries itself? or let them through?

If not then, I guess I'll have to adapt check-mirror to run torsocks only when possible and that would imply getting rid of my http_request function and do all HTTP requests using curl (#8437).

#12 Updated by intrigeri over 2 years ago

Is there a way to either ask torsocks to do such kind of queries itself? or let them through?

See #8074#note-9.

#13 Updated by sajolida over 2 years ago

  • Assignee set to sajolida
  • Type of work changed from Code to Wait

:)

So I subscribed myself to Tor#8137 and this is now a Wait (unless I do #8437 first).

#14 Updated by sajolida over 2 years ago

  • Blocks Feature #7667: Create a public repo for check-mirrors added

#15 Updated by BitingBird over 2 years ago

https://trac.torproject.org/projects/tor/ticket/8137 is fixed and the other ticket was never created.

#16 Updated by intrigeri over 2 years ago

torsocks 2.1 (currently in Debian testing/sid) has the feature we need. Will track the next Debian steps in subtasks.

#17 Updated by intrigeri over 2 years ago

  • Type of work changed from Wait to Code

Sadly, even with torsocks 2.1 (built locally as part of #9549) and setting AllowOutboundLocalhost 1 in torsocks.conf, and disabling the netfilter firewall, this still doesn't work: I suspect that AllowOutboundLocalhost 1 only lets TCP connections to the loopback iface go through, while our full-featured DNS resolver is listening on 127.0.0.2:53 over UDP:

$ torsocks ruby ./check-mirrors.rb --fast --debug --dns 127.0.0.2
/usr/lib/ruby/1.9.1/resolv.rb:761:in `initialize': Operation not permitted - socket(2) - udp (Errno::EPERM)
    from /usr/lib/ruby/1.9.1/resolv.rb:761:in `new'
    from /usr/lib/ruby/1.9.1/resolv.rb:761:in `initialize'
    from /usr/lib/ruby/1.9.1/resolv.rb:537:in `new'
    from /usr/lib/ruby/1.9.1/resolv.rb:537:in `make_udp_requester'
    from /usr/lib/ruby/1.9.1/resolv.rb:495:in `each_resource'
    from /usr/lib/ruby/1.9.1/resolv.rb:391:in `each_address'
    from /usr/lib/ruby/1.9.1/resolv.rb:379:in `getaddresses'
    from ./check-mirrors.rb:290:in `<main>'

So, it seems that we're back to square one. I see a few solutions:

  1. add an option to torsocks to allow even UDP connections to localhost (perhaps the existing one can be extended to do that, its documentation doesn't really pretend it's blocking UDP so far);
  2. only use torsocks, from within check-mirrors, for connections to the Internet, and then we can use 127.0.0.2:53 over UDP for DNS resolution;
  3. add proper SOCKS support to check-mirrors, use it for connections to the Internet, and then we can use 127.0.0.2:53 over UDP for DNS resolution.

#18 Updated by intrigeri over 2 years ago

intrigeri wrote:

  1. add an option to torsocks to allow even UDP connections to localhost (perhaps the existing one can be extended to do that, its documentation doesn't really pretend it's blocking UDP so far);

I've discussed this with Yawning (who wrote the initial patch for this feature) and he's happy to extend that option for our needs => someone should file a ticket on Tor's trac so that he doesn't forget.

#19 Updated by sajolida about 2 years ago

  • QA Check deleted (Dev Needed)
  • Type of work changed from Code to Wait

#20 Updated by sajolida over 1 year ago

Tor#16765 is now closed and I guess should be released in the next version of torsocks. We're getting there!

#22 Updated by intrigeri about 1 year ago

commit:10db2ed5dca5bd91b37d6781039718f5441dd807 sets AllowOutboundLocalhost 1. Once we ship torsocks 2.2, perhaps we'll want AllowOutboundLocalhost 2 instead so that the mirror pool maintainers can use check-mirror from Tails without any special configuration tweak.

#23 Updated by intrigeri about 1 year ago

  • Related to Bug #11736: torsocks complains when used with Monkeysign, but works anyway added

#24 Updated by intrigeri 12 months ago

  • Type of work changed from Wait to Code

I've uploaded torsocks 2.2 to jessie-backports, so the nightly builds from our stable and devel branch should have it in 6-24 hours. Then you can test if AllowOutboundLocalhost 2 is sufficient.

#25 Updated by intrigeri 5 months ago

  • Subject changed from Add support for SOCKS proxy to check-mirror to Add support for SOCKS proxy to check-mirrors
  • QA Check set to Ready for QA

If I get the history of this ticket right, this should now be a solved problem, so next things to do are 1. verify that it's really solved with current torsocks; 2. perhaps some custom SOCKS code should be removed?; 3. move on to #7667 so we can eventually get some more help on check-mirrors :)

#26 Updated by sajolida 5 months ago

  • Target version set to Tails_3.1

Exactly, so I'll put this back on my radar.

It would feel good to have other people help with this script.

#27 Updated by intrigeri 2 months ago

  • Target version changed from Tails_3.1 to Tails_3.2

#28 Updated by anonym 18 days ago

  • Target version changed from Tails_3.2 to Tails_3.3

Also available in: Atom PDF