Project

General

Profile

Bug #10068

Upgrade to Jenkins 2.x, using upstream packages

Added by bertagaz about 3 years ago. Updated 11 days ago.

Status:
In Progress
Priority:
High
Assignee:
Category:
Continuous Integration
Target version:
Start date:
01/08/2018
Due date:
% Done:

0%

QA Check:
Dev Needed
Feature Branch:
puppet-tails:feature/10068-upstream-jenkins-deb
Type of work:
Research
Blueprint:
Starter:
No
Affected tool:

Description

The current state of the Jenkins Debian package is quite scary: it's lagging a lot behind Jenkins' LTS version, and it has quite a bunch of known security bugs (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=781223)

The Debian package state will probably not be fixed, given the Jenkins LTS release fast pace. In this situation, we're quite stuck using outdated Jenkins plugins too, given they often depends on precise Jenkins versions. This doesn't help our Jenkins plugins upgrade sysadmin task.

jenkins.debian.net is using upstream's LTS package. We probably should discuss the situation with them, as they'll probably stumble upon the same problem as they want DSA to take care of the sysadmin maintenance of their instance.

Some discussions are planned at the 2015 Debconf about jenkins.d.n. Could be good to follow what happen on this front.

We could use the upstream APT repo and their Debian package, but it would need some review from our side, to see how the packaging is done and what it really install. Sadly, the Debian package sources don't seem to be available.


Subtasks

Feature #15155: Upgrade the jenkins Puppet moduleConfirmedbertagaz


Related issues

Related to Tails - Feature #10117: Design how to run our test suite in Jenkins Resolved 08/28/2015
Related to Tails - Feature #6270: Publish our Jenkins read-only on the web Confirmed 09/10/2013
Related to Tails - Feature #11739: Upgrade our isotesters to Stretch Resolved 08/28/2016
Related to Tails - Feature #15798: Jenkins access for new FT members In Progress 08/16/2018
Blocks Tails - Feature #10328: Clean up features with Scenario Outlines Confirmed 10/03/2015
Blocks Tails - Feature #13284: Core work 2017Q2→2019Q2: Sysadmin (Adapt our infrastructure) Confirmed 06/30/2017
Blocks Tails - Feature #15502: Update Jenkins modules: 2018Q2 → 2018Q3 edition Confirmed 09/30/2018
Blocks Tails - Feature #15501: Server hardware (2017-2019 edition): evaluate some of the options Confirmed 04/08/2018

History

#1 Updated by bertagaz about 3 years ago

  • Type of work changed from Sysadmin to Research

#2 Updated by bertagaz about 3 years ago

  • Target version changed from Tails_1.6 to Tails_1.7

Delaying, that's a long research/discussion I won't have time to work on until the #5288 is deployed.

#3 Updated by intrigeri about 3 years ago

Note that recent versions of plugins (e.g. ParameterizedTrigger) require a version of Jenkins that's not in Debian.

#4 Updated by intrigeri about 3 years ago

bertagaz wrote:

jenkins.debian.net is using upstream's LTS package. We probably should discuss the situation with them, as they'll probably stumble upon the same problem as they want DSA to take care of the sysadmin maintenance of their instance.

Actually not: DSA is fine with having the upstream package used when transitioning to jenkins.debian.org.

#5 Updated by intrigeri about 3 years ago

  • Related to Feature #10117: Design how to run our test suite in Jenkins added

#6 Updated by bertagaz about 3 years ago

intrigeri wrote:

bertagaz wrote:

jenkins.debian.net is using upstream's LTS package. We probably should discuss the situation with them, as they'll probably stumble upon the same problem as they want DSA to take care of the sysadmin maintenance of their instance.

Actually not: DSA is fine with having the upstream package used when transitioning to jenkins.debian.org.

Ah, interesting.

I realized lately that https://jenkins.openstack.org was using the same old version we do btw. And they do expose their instances in the WildWildWeb, using Jenkins' security matrix setup.

Infos about their deployment can be found here

#7 Updated by intrigeri about 3 years ago

I realized lately that https://jenkins.openstack.org was using the same old version we do btw.

Wow! That's surprising, since https://git.openstack.org/cgit/openstack-infra/puppet-jenkins/tree/manifests/master.pp explicitly enables the upstream's APT repo, which currently proposes 1.609.2. It might be because the repo's key URLs (both the one in that Puppet manifest, and the one advertised on http://pkg.jenkins-ci.org/debian-stable/) give me a 403. Anyway.

#8 Updated by bertagaz about 3 years ago

Yes, I've been surprised too after reading their manifest. Maybe they use some kind of static or self-generated webpage with an outdated infos regarding the version.

#9 Updated by bertagaz almost 3 years ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Info Needed

I've discussed a bit with weasel about this. They are using the upstream LTS Debian package in the Torproject infra. He didn't audited that package much (e.g was suprised of its 50M size), but says he is happy with it and works well. He confirmed that DSA is willing to use that package too when jenkins.d.n will be taken care of by them.
So if both of this projects decided to do so, maybe it's worth considering doing so too (even if I'm a bit afraid of such an upgrade ;)). If we do, I don't think I'll do this upgrade soon anyway, let just finish the auto test deployment first.

#10 Updated by intrigeri almost 3 years ago

So if both of this projects decided to do so, maybe it's worth considering doing so too

I find it baffling that there's no Debian solution to this problem in sight, especially with all the big players involved who rely on that package for mission critical gatekeeping tasks. I am worried that we start relying on essentially non-free software ourselves, which will be problematic if/once we want to trust a Jenkins instance of ours more than we currently do. But all in all, it seems potentially worse to be using for long periods of time a version of Jenkins with known security issues, especially once we want to make it more public.

So if you think we should go ahead and do the switch, feel free to.

#11 Updated by intrigeri almost 3 years ago

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Info Needed to Dev Needed

#12 Updated by bertagaz almost 3 years ago

  • Target version changed from Tails_1.7 to Tails_1.8

Postponing, as that's clearly something that won't happen before 1.7.

#13 Updated by intrigeri almost 3 years ago

  • Related to Feature #6270: Publish our Jenkins read-only on the web added

#14 Updated by bertagaz almost 3 years ago

Postponing

#15 Updated by bertagaz almost 3 years ago

  • Target version changed from Tails_1.8 to Tails_2.0

#16 Updated by bertagaz over 2 years ago

  • Target version changed from Tails_2.0 to Tails_2.2

Postponing, won't work on that for the rest of 2.0 cycle.

#17 Updated by bertagaz over 2 years ago

  • Target version changed from Tails_2.2 to Tails_2.3

Postponing, this ticket won't be worked on during this release.

#18 Updated by intrigeri over 2 years ago

Jenkins was removed from Debian: https://bugs.debian.org/811522. So this now blocks #11113 (I can't set up new isotesters given I can't install jenkins-slave on them). I may then give it a try in the next 1-3 days. Given that this has been regularly postponed since 4+ months, I guess you won't mind if I do it.

#19 Updated by intrigeri over 2 years ago

  • Subject changed from Use a more recent Jenkins version to Upgrade to upstream packages for Jenkins

#20 Updated by intrigeri over 2 years ago

#21 Updated by intrigeri over 2 years ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10
  • Feature Branch set to puppet-tails:feature/10068-upstream-jenkins-deb
  • master: see some preliminary work (untested, probably broken in many ways) in the topic branch; I have no plans to work more on this, it's back on your plate
  • slaves: I'm going to upload the jenkins-slave package to our own APT repo, it's just some glue that gets the JAR from the master and turns it into a service; I would like to complete this part as it is what blocks me for #11113.

#22 Updated by intrigeri over 2 years ago

The jenkins-slave side of things is done, I'll let you handle the master part, or shout for help.

#23 Updated by intrigeri over 2 years ago

#24 Updated by bertagaz over 2 years ago

  • Target version changed from Tails_2.3 to Tails_2.4

#25 Updated by bertagaz over 2 years ago

  • Target version changed from Tails_2.4 to Tails_2.5

#26 Updated by bertagaz over 2 years ago

  • Target version changed from Tails_2.5 to Tails_2.6

Other things are claiming me for the next release.

#27 Updated by anonym about 2 years ago

  • Target version changed from Tails_2.6 to Tails_2.7

#28 Updated by bertagaz about 2 years ago

  • Target version changed from Tails_2.7 to Tails_2.9.1

#29 Updated by intrigeri almost 2 years ago

  • Blocks Feature #10328: Clean up features with Scenario Outlines added

#30 Updated by anonym almost 2 years ago

  • Target version changed from Tails_2.9.1 to Tails 2.10

#31 Updated by anonym over 1 year ago

  • Target version changed from Tails 2.10 to Tails_2.11

#32 Updated by bertagaz over 1 year ago

Note for myself (and the reviewer): that will be the right time to document which plugin we should not update. Most likely it will only be the priority sorter plugin.

#33 Updated by bertagaz over 1 year ago

  • Target version changed from Tails_2.11 to Tails_2.12

#34 Updated by bertagaz over 1 year ago

  • Target version changed from Tails_2.12 to Tails_3.0

#35 Updated by intrigeri over 1 year ago

#36 Updated by bertagaz over 1 year ago

  • Target version changed from Tails_3.0 to Tails_3.1

#37 Updated by bertagaz over 1 year ago

Note to myself: when upgrading to the upstream package, it's likely that we'll update the cucumber test report plugin too, and then we'll be able to remove the custom cucumber package we've installed on our isotesters. See #11739 for details.

#38 Updated by bertagaz over 1 year ago

  • Target version changed from Tails_3.1 to Tails_3.2

#39 Updated by intrigeri about 1 year ago

  • Subject changed from Upgrade to upstream packages for Jenkins to Upgrade to Jenkins 2.x, using upstream packages

#40 Updated by intrigeri about 1 year ago

  • Blocks Feature #13284: Core work 2017Q2→2019Q2: Sysadmin (Adapt our infrastructure) added

#41 Updated by bertagaz about 1 year ago

  • Target version changed from Tails_3.2 to Tails_3.3

#42 Updated by intrigeri 12 months ago

Note that my plans for #11680 might require Jenkins plugins that want a newer Jenkins (and I might not be in the mood to cope with bugs in older version of these plugins if they've been fixed in newer versions already). I want to work on this mid-December. Can you please give me an ETA for this ticket? It's been postponed to "next release" regularly since almost two years, so you'll understand I take the current Target version with a grain of salt :)

Other options (if you can't give an ETA or if it's in too long):

  • I postpone #11680 and focus on the Puppet 4 migration first, to give you some more time here.
  • groente or I takes it over.

#43 Updated by bertagaz 12 months ago

  • Target version changed from Tails_3.3 to Tails_3.5

#44 Updated by intrigeri 11 months ago

Ping wrt. the question I've asked 3 weeks ago? I want to make sure you're at least aware of the fallback options that I may have to go with if you can't give me a suitable ETA.

#45 Updated by bertagaz 11 months ago

  • Blocks Bug #14875: Build reproducibility Jenkins tests: confusing UX and implementation added

#46 Updated by bertagaz 11 months ago

intrigeri wrote:

Ping wrt. the question I've asked 3 weeks ago? I want to make sure you're at least aware of the fallback options that I may have to go with if you can't give me a suitable ETA.

I had a look at my future schedules and it should be doable if that's the next big task I'm tackling, meaning I'll have to probably delay a few others.

#47 Updated by intrigeri 11 months ago

I had a look at my future schedules and it should be doable […]

I've asked you for an ETA and you tell me "it should be doable". Does this implicitly mean it'll be done (tested, debugged, deployed, fixed) by mid-December?

#48 Updated by bertagaz 11 months ago

intrigeri wrote:

I've asked you for an ETA and you tell me "it should be doable". Does this implicitly mean it'll be done (tested, debugged, deployed, fixed) by mid-December?

Yes, that's what I meant.

#49 Updated by intrigeri 11 months ago

intrigeri wrote:

I've asked you for an ETA and you tell me "it should be doable". Does this implicitly mean it'll be done (tested, debugged, deployed, fixed) by mid-December?

Yes, that's what I meant.

Thanks for clarifying :)

#50 Updated by intrigeri 9 months ago

  • Priority changed from Normal to High

(As per sysadmin team sprint.)

#51 Updated by intrigeri 9 months ago

FWIW I've noticed today, while working on #15154, that the Puppet module we use to manage Jenkins "does not presently support Jenkins 2.x due to incompatible changes with 1.x. Support is planned for a future release" as of 1.7.0 (last upstream release, August 2016).

#52 Updated by anonym 8 months ago

  • Target version changed from Tails_3.5 to Tails_3.6

#53 Updated by intrigeri 8 months ago

#54 Updated by bertagaz 6 months ago

  • Target version changed from Tails_3.6 to Tails_3.7

#55 Updated by intrigeri 6 months ago

  • Blocks Feature #15502: Update Jenkins modules: 2018Q2 → 2018Q3 edition added

#56 Updated by intrigeri 6 months ago

  • Blocks Feature #15501: Server hardware (2017-2019 edition): evaluate some of the options added

#57 Updated by intrigeri 6 months ago

I'd like to plan my sysadmin work for this year and this ticket blocks #15501, which I'd like to tackle in 2018Q4 to the latest. So, let's do the ETA dance again. Are you in a position to:

  1. open your agenda
  2. schedule/block enough time for doing this work, with some safety margin to take into account unscheduled AFK emergencies/unavailability and unexpected technical issues
  3. tell me when you are confident this will be done

?

If not, well, let's come back to this topic in ~August or so and then we'll see how we can organize this in a way that works for everyone.

Thanks in advance.

#58 Updated by intrigeri 6 months ago

  • Blocks deleted (Bug #14875: Build reproducibility Jenkins tests: confusing UX and implementation)

#59 Updated by bertagaz 5 months ago

  • Target version changed from Tails_3.7 to Tails_3.8

#60 Updated by intrigeri 3 months ago

  • Target version changed from Tails_3.8 to Tails_3.9

#61 Updated by intrigeri about 1 month ago

#62 Updated by intrigeri 19 days ago

  • Target version changed from Tails_3.9 to Tails_3.10

#63 Updated by intrigeri 12 days ago

Hi bertagaz, welcome back! It would be very useful if you could reply to #10068#note-57 aka. "let's do the ETA dance again" one of these days.

From my side, two data points:

  • I won't work on #15501 this year but I'd like to schedule it for the first half of 2019 and that work is still blocked by the upgrade to a recent Jenkins.
  • As I've just reported on #10328 the old version of Jenkins we have forces us to write Gherkin scenarios in suboptimal ways. I'm excited at the idea of being able to clean this up :)

#64 Updated by bertagaz 11 days ago

intrigeri wrote:

Hi bertagaz, welcome back! It would be very useful if you could reply to #10068#note-57 aka. "let's do the ETA dance again" one of these days.

From my side, two data points:

  • I won't work on #15501 this year but I'd like to schedule it for the first half of 2019 and that work is still blocked by the upgrade to a recent Jenkins.
  • As I've just reported on #10328 the old version of Jenkins we have forces us to write Gherkin scenarios in suboptimal ways. I'm excited at the idea of being able to clean this up :)

Ack, I'll think about that and will come back with a plan.

Also available in: Atom PDF