Project

General

Profile

Bug #11858

Monitor if isobuilders systems are running fine

Added by bertagaz about 1 year ago. Updated about 1 month ago.

Status:
Confirmed
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
Start date:
10/03/2016
Due date:
% Done:

0%

QA Check:
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Easy:
Yes
Affected tool:

Description

We experienced times where our isobuilders were slowly getting all down when a branch was triggering the OOM during its build.

We should use our monitoring system to check using systemd and/or anything else if the isobuilders systems are running fine, so that we know if we have to restart them or their jenkins-slave service.


Related issues

Related to Tails - Bug #11632: ISO builds from branch that need more RAM can break all our Jenkins isobuilders without us being notified Resolved 08/11/2016
Related to Tails - Bug #12009: Jenkins ISO builders are highly unreliable Resolved 12/01/2016
Related to Tails - Bug #13582: Monitoring bridge Duplicate 08/04/2017
Blocks Tails - Feature #13242: Core work 2017Q4: Sysadmin (Maintain our already existing services) Confirmed 06/29/2017

History

#1 Updated by intrigeri about 1 year ago

  • Assignee set to bertagaz

(Assuming that's what you meant given you've set a target version.)

#2 Updated by intrigeri about 1 year ago

  • Related to Bug #11632: ISO builds from branch that need more RAM can break all our Jenkins isobuilders without us being notified added

#3 Updated by bertagaz 11 months ago

  • Target version changed from Tails_2.7 to Tails_2.9.1

#4 Updated by intrigeri 11 months ago

  • Related to Bug #12009: Jenkins ISO builders are highly unreliable added

#5 Updated by anonym 10 months ago

  • Target version changed from Tails_2.9.1 to Tails 2.10

#6 Updated by anonym 9 months ago

  • Target version changed from Tails 2.10 to Tails_2.11

#7 Updated by bertagaz 7 months ago

  • Target version changed from Tails_2.11 to Tails_2.12

#8 Updated by bertagaz 7 months ago

  • Target version changed from Tails_2.12 to Tails_3.0

#9 Updated by bertagaz 6 months ago

  • Target version changed from Tails_3.0 to Tails_3.1

#10 Updated by bertagaz 5 months ago

  • Target version changed from Tails_3.1 to Tails_3.2

#11 Updated by intrigeri 4 months ago

  • Blocks Feature #13233: Core work 2017Q3: Sysadmin (Maintain our already existing services) added

#12 Updated by groente 2 months ago

  • Easy set to Yes

A simple check whether

systemctl --quiet is-failed \*

returns 0 (in which case something is wrong) should do the trick, both for the isobuilders and #13582

#13 Updated by groente 2 months ago

#14 Updated by intrigeri 2 months ago

systemctl is-system-running might do exactly what we want.

#15 Updated by bertagaz about 1 month ago

  • Target version changed from Tails_3.2 to Tails_3.3

#16 Updated by bertagaz about 1 month ago

  • Blocks deleted (Feature #13233: Core work 2017Q3: Sysadmin (Maintain our already existing services))

#17 Updated by bertagaz about 1 month ago

  • Blocks Feature #13242: Core work 2017Q4: Sysadmin (Maintain our already existing services) added

Also available in: Atom PDF