Project

General

Profile

Feature #14588

Self-host our website

Added by intrigeri 11 months ago. Updated 3 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
Start date:
09/04/2017
Due date:
% Done:

0%

QA Check:
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

New design

ikiwiki and web hosting

ikiwiki would run on www.lizard which also serves the generated content and runs ikiwiki.cgi (until #9174 is done):

  • ikiwiki on www.lizard pushes changes back to the master/central tails.git repo (PO files updates, changes done in the web interface aka. ikiwiki.cgi). That's exactly what the current production setup does.
  • Ensure logging policy is OK:
    • nginx' own logs: no IPs: OK
    • Journal, if relevant: OK, I've not found anything nginx-related in the Journal, even after hitting a 404
  • There's currently a cronjob that extracts connection stats from access.log (Tails boots, downloads of the detached ISO signature) and emails them to . We need to import that too.
  • The current setup is heavily based on Apache features, while we usually run nginx on our infra:

language negotiation

I got language negotiation to work fine with this included in the http {} block:

map $http_accept_language $lang {
    default en;
    ~*^de de;
    ~*^fa fa;
    ~*^es es;
    ~*^fr fr;        
    ~*^it it;
    ~*^pt pt;
}

… and this included in the vhost:

location / {
    try_files $uri $uri/index.$lang.html $uri/index.en.html $uri/index.html =404;
}

Our initial options were:

Master/central tails.git repo

The master/central tails.git repo would move to git.puppet.t.b.o (sic) i.e. hosted on lizard's gitolite:

  • we'll be able to use gitolite to manage ACLs which is better than the current setup at b.o, e.g. we can manage users & keys ourselves, give access to some more people to a subset of branches (e.g. we could allow some developers to push to feature/* and bugfix/* so their stuff is built and tested in Jenkins, but they would not be allowed to push to protected branches such as master, stable, testing and devel)
  • maybe we can use a better CNAME that does not say "puppet"; but anyway, unless using the Tor onion service, a SSH config is needed because that Gitolite runs behind a non-standard port, so contributors with push access can as well use the onion service and we don't bother about DNS names
  • Git validation hook (file size, obsolete rewritten history, etc.) copied from the one that's set up at b.o
  • Git hook that triggers an ikiwiki update with the pingee plugin
  • post-update hook that pushes to all mirrors (copied from b.o)

Migration plan

migrate the master/central tails.git repo

  1. copy everything over to lizard and have gitolite push updates to all mirrors (see design above); mirrors: immerda, labs.r.n (Redmine), GitLab
  2. have b.o disable their Git hook that pushes updates to mirrors: immerda, labs.r.n (Redmine), GitLab, and finally lizard when we're ready for the next steps
  3. announce downtime to Git committers
  4. forbid Git committers write access to the former master/central tails.git
  5. drop the ACL that allows the former repo to push to the new one: next step is to have the new repo push to the old one, and we don't want an infinite loop
  6. have the new master/central tails.git notify the website (pingee plugin) on updates so that our website, still running at b.o at this point, is updated
  7. have the production website (at b.o) use the new repo as its Git remote
  8. update contribute/git doc and ask Git committers to update their config

prepare the new web hosting setup

  1. set up the basic web server (including LE) and ikiwiki stuff on www.lizard under some temporary vhost name
  2. have the master/central repo trigger an ikiwiki update (on Git push) on www.lizard as well => we can test how it behaves
  3. address the remaining problems on www.lizard (logging, migrating away from Apache, etc.: see design above)
  4. ensure there's enough space in www.lizard:/var/log/nginx to host the amount of logs we need to keep around (even without hosting our website, at some point that directory took too much space: #12425)

migrate to the new web hosting setup

Once happy with the new hosting setup:

  1. rename/copy it to support tails.b.o, adjust ikiwiki.setup accordingly, rebuild
  2. copy X.509 cert+key from the old website to the new one
  3. point tails.b.o in /etc/hosts to lizard on a test machine and ensure the website hosted there works as expected
  4. point DNS to lizard

And once we're convinced the new hosting setup works well enough:

  1. drop the temporary vhost
  2. ask b.o to delete our website, Git repo and cronjob
  3. If time allows, do #9174 and/or #12408

Related issues

Related to Tails - Feature #10034: Translation web platform Confirmed 08/14/2015
Blocks Tails - Feature #12408: Ensure our website is ready for temporary surge of new users Confirmed 03/29/2017
Blocks Tails - Feature #9174: Migrate our blueprints to blueprints.tails.boum.org Confirmed 04/07/2015
Blocks Tails - Feature #13284: Core work 2017Q2→2019Q1: Sysadmin (Adapt our infrastructure) Confirmed 06/30/2017
Blocks Tails - Feature #15202: Onboard new members to the mirror team In Progress 01/19/2018

History

#1 Updated by intrigeri 10 months ago

  • Blocks Feature #12408: Ensure our website is ready for temporary surge of new users added

#2 Updated by sajolida 6 months ago

  • Blocks Feature #9174: Migrate our blueprints to blueprints.tails.boum.org added

#3 Updated by u 5 months ago

#4 Updated by intrigeri 3 months ago

  • Blocks Feature #13284: Core work 2017Q2→2019Q1: Sysadmin (Adapt our infrastructure) added

#5 Updated by intrigeri 3 months ago

  • Target version changed from 2019 to Tails_3.10

#6 Updated by intrigeri 3 months ago

  • Description updated (diff)
  • Status changed from Confirmed to In Progress

#7 Updated by intrigeri 3 months ago

  • Description updated (diff)

#8 Updated by intrigeri 3 months ago

  • Description updated (diff)

#9 Updated by intrigeri 3 months ago

  • Description updated (diff)

#10 Updated by intrigeri 3 months ago

  • Description updated (diff)

#11 Updated by intrigeri 3 months ago

  • Description updated (diff)

#12 Updated by intrigeri 3 months ago

  • Description updated (diff)

#13 Updated by intrigeri 3 months ago

  • Description updated (diff)

#14 Updated by intrigeri 3 months ago

  • Description updated (diff)

#15 Updated by intrigeri 3 months ago

  • Description updated (diff)

#16 Updated by intrigeri 3 months ago

  • Description updated (diff)

#17 Updated by intrigeri 3 months ago

  • Description updated (diff)

#18 Updated by intrigeri 3 months ago

  • Description updated (diff)

#19 Updated by intrigeri 3 months ago

  • Description updated (diff)

#20 Updated by intrigeri 3 months ago

  • Description updated (diff)

#21 Updated by intrigeri 3 months ago

  • Description updated (diff)

#22 Updated by intrigeri 3 months ago

  • Description updated (diff)

#23 Updated by intrigeri 3 months ago

  • Description updated (diff)

#24 Updated by intrigeri 7 days ago

Also available in: Atom PDF