Project

General

Profile

Feature #6303

Feature #5926: Freezable APT repository

Feature #9489: Implement packages importing and freezing

Adapt our infrastructure to be able to handle tons of packages

Added by intrigeri almost 5 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
Infrastructure
Target version:
Start date:
01/04/2016
Due date:
% Done:

100%

QA Check:
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

When we'll import all the packages we need into our own APT repository, it's unlikely that our various cronjobs etc. handle the load as is. Same for our fail-over and backup systems.


Subtasks

Feature #10851: Give lizard enough free storage to host our freezable APT repositoryResolved


Related issues

Related to Tails - Feature #9508: Evaluate freezable APT repo's storage needs Resolved 05/30/2015
Blocked by Tails - Feature #6295: Evaluate consequences of importing large amounts of packages into reprepro Resolved 09/26/2013

History

#1 Updated by intrigeri almost 5 years ago

  • Starter changed from No to Yes

#2 Updated by intrigeri over 4 years ago

  • Category set to Infrastructure

#3 Updated by intrigeri about 3 years ago

  • Blocked by deleted (Feature #6295: Evaluate consequences of importing large amounts of packages into reprepro)

#4 Updated by intrigeri about 3 years ago

  • Blocks deleted (Feature #6299: Regularly pull packages from foreign APT repositories)

#5 Updated by intrigeri about 3 years ago

  • Blocks deleted (Feature #6310: Publish source Debian packages used by Tails ISO images)

#6 Updated by intrigeri about 3 years ago

  • Assignee set to intrigeri
  • Target version changed from Sustainability_M1 to Tails_2.2
  • Parent task changed from #5926 to #9489

#8 Updated by intrigeri about 3 years ago

  • Subject changed from Adapt our APT repository infrastructure to be able to handle tons of packages to Adapt our infrastructure to be able to handle tons of packages
  • Description updated (diff)

#9 Updated by intrigeri about 3 years ago

  • Related to Feature #9508: Evaluate freezable APT repo's storage needs added

#11 Updated by intrigeri over 2 years ago

  • Blocked by Feature #6295: Evaluate consequences of importing large amounts of packages into reprepro added

#12 Updated by intrigeri over 2 years ago

  • Blocks Feature #6296: Configure reprepro to pull from foreign APT repositories added

#13 Updated by intrigeri over 2 years ago

  • Priority changed from Normal to Elevated
  • Target version changed from Tails_2.2 to Tails_2.0
  • Starter deleted (Yes)

This will be blocking the actual deployment so I'd like to get it done early.

#14 Updated by intrigeri over 2 years ago

  • Status changed from Confirmed to In Progress
  • Target version changed from Tails_2.0 to Tails_2.2

The urgent part is being handled in the #10851 subtask, and what's left (backups, and the fail-over system we don't have yet) can wait.

#15 Updated by intrigeri over 2 years ago

  • Assignee changed from intrigeri to bertagaz
  • QA Check set to Ready for QA

(The only remaining open subtask is on bertagaz' plate.)

#16 Updated by intrigeri over 2 years ago

  • Assignee changed from bertagaz to intrigeri
  • Target version changed from Tails_2.2 to Tails_2.3
  • QA Check deleted (Ready for QA)

Oops. Actually there's also the backup and fail-over bits that I need to tackle at some point.

#18 Updated by intrigeri over 2 years ago

  • Blocks deleted (Feature #6296: Configure reprepro to pull from foreign APT repositories)

#19 Updated by intrigeri over 2 years ago

  • Assignee changed from intrigeri to bertagaz
  • Target version changed from Tails_2.3 to Tails_2.4

bertagaz: I'm reassigning to you because the only small bit left to do here is on your plate. But feel free to ignore the details of my reasoning, and skip directly to your name. Now, if you want to review my reasoning, I certainly don't mind :)

Both the failover and the backups will only have to carry tagged snapshots, not time-based snapshots (that we can afford losing). One initial data point is that #6295#note-29 says it will take 90GB after a year, but this was based on assumptions ("we import them in a single, persistent reprepro instance, even though the filterlist etc. used for importing are volatile") that might not hold anymore. After running reprepro update in each of the reprepro config trees created by a run of tails-prepare-tagged-apt-snapshot-import, I see 6.4 GB used (including i386+source + most amd64 packages). So worst case, it's 6.4 GB * 15 releases each year = 96 GB / year + some room for expected growth.

So I say that:

  • short/mid-term:
    • let's get ourselves 200GB for each set of backups; the one I'm managing has this already, so I'm reassigning this ticket to bertagaz, so that he does the same (I could give a hand but I bet the logistics overhead won't be worth it)
    • I've adjusted our backup system so that it skips time-based snapshots, but backs up tagged snapshots
    • for the failover: https://tails.boum.org/blueprint/lizard_failover/#system-specs already plans enough storage so we're good
  • long-term, i.e. whenever the occupied space grows enough to become problematic: it should be fairly easy to de-duplicate the stored files, e.g. using the hardlink tool or a filesystem that can de-duplicate itself at the block or file level.

#20 Updated by intrigeri over 2 years ago

  • Target version changed from Tails_2.4 to Tails_2.3

#22 Updated by bertagaz about 2 years ago

  • Assignee changed from bertagaz to intrigeri

intrigeri wrote:

So I say that:

  • short/mid-term:
    • let's get ourselves 200GB for each set of backups; the one I'm managing has this already, so I'm reassigning this ticket to bertagaz, so that he does the same (I could give a hand but I bet the logistics overhead won't be worth it)

I have that amount of storage dedicated to this backups.

  • long-term, i.e. whenever the occupied space grows enough to become problematic: it should be fairly easy to de-duplicate the stored files, e.g. using the hardlink tool or a filesystem that can de-duplicate itself at the block or file level.

I would have said ticket++ to track this space issue, but nowadays our monitoring system should prevent us to do that :). Congrats for this plan, I'm all for it. Fell free to close unless something else is needed from me.

#23 Updated by intrigeri about 2 years ago

  • Target version changed from Tails_2.3 to Tails_2.4

#24 Updated by intrigeri about 2 years ago

  • Status changed from In Progress to Resolved
  • Target version changed from Tails_2.4 to Tails_2.3

#25 Updated by intrigeri about 2 years ago

  • Assignee deleted (intrigeri)

Also available in: Atom PDF