Project

General

Profile

Feature #11833

Feature #11834: Migrate our infrastructure to Puppet 4

Make our Puppet code compatible with the "future" parser

Added by intrigeri almost 2 years ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
Start date:
09/24/2016
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

This will be the first necessary step towards migrating to Puppet 4.


Related issues

Blocked by Tails - Feature #11835: Upgrade Puppet master and clients to 3.8 Resolved 09/24/2016
Blocks Tails - Feature #13284: Core work 2017Q2→2019Q1: Sysadmin (Adapt our infrastructure) Confirmed 06/30/2017
Blocked by Tails - Feature #11836: Stop stringifying Puppet facts Resolved 09/24/2016

History

#1 Updated by intrigeri almost 2 years ago

#2 Updated by intrigeri almost 2 years ago

  • Blocks deleted (Feature #11834: Migrate our infrastructure to Puppet 4)

#3 Updated by intrigeri almost 2 years ago

  • Parent task set to #11834

#4 Updated by intrigeri almost 2 years ago

#5 Updated by intrigeri almost 2 years ago

  • Blocked by Feature #11835: Upgrade Puppet master and clients to 3.8 added

#6 Updated by intrigeri almost 2 years ago

  • Assignee deleted (intrigeri)

#7 Updated by intrigeri over 1 year ago

  • Assignee set to intrigeri

#8 Updated by intrigeri about 1 year ago

  • Target version set to Tails_3.5

#9 Updated by intrigeri 6 months ago

My plan (based on https://docs.puppet.com/puppet/4.5/upgrade_major_pre.html#enable-the-future-parser-and-fix-broken-code) is to set up a Puppet environment (cloned from our production one) with the future parser enabled and test each of our systems with it, one after the other, so that I have an initial list of problems I shall fix. I'll minimize how long each system is managed by this test environment. Then I'll fix these issues, push to that test environment, rince and repeat until I can enable the future parser on our current Puppet 3.x production environment, which should ensure we don't write new code that's not compatible with it.

#10 Updated by intrigeri 6 months ago

  • Blocks Feature #13284: Core work 2017Q2→2019Q1: Sysadmin (Adapt our infrastructure) added

#11 Updated by intrigeri 6 months ago

  • Target version changed from Tails_3.5 to Tails_3.6

(See the parent ticket for the big picture & planning info.)

#12 Updated by intrigeri 5 months ago

  • Target version changed from Tails_3.6 to Tails_3.7

#13 Updated by intrigeri 4 months ago

#14 Updated by intrigeri 4 months ago

intrigeri wrote:

My plan (based on https://docs.puppet.com/puppet/4.5/upgrade_major_pre.html#enable-the-future-parser-and-fix-broken-code) is to set up a Puppet environment (cloned from our production one) with the future parser enabled and test each of our systems with it, one after the other, so that I have an initial list of problems I shall fix.

After improving our puppet-sync setup to support multiple environments, I realized there's an easier way: run puppet agent --test --parser future on each system. Done everywhere and it works but the future parser is only partially used (types are not enforced) as long as we have stringify_facts = true, which is why https://docs.puppet.com/puppet/4.5/upgrade_major_pre.html recommends to set this to false (#11836) before trying the future parser.

#15 Updated by intrigeri 4 months ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 50

Deployed on all systems, seems to work.

#16 Updated by intrigeri 4 months ago

  • Assignee changed from intrigeri to groente
  • QA Check set to Ready for QA

The "future" parser is now enabled for all our systems; we also now have stringify_facts = false everywhere, which is a prerequisite for the type checks performed by the "future" parser to be fully enabled.

I had to adjust quite a few bits in our manifests and in the tails module. Other modules were already fine thanks to the collective effort the shared Puppet modules group has put into making them compatible with Puppet 4 (including those I ported myself, see #11834#note-17). I occasionally had to upgrade to the latest upstream version of a module, or to merge the "future" branch of a shared module (that's the branch we use there in case the changes required for Puppet 4 compatibility don't work without the future parser enabled), or to cherry-pick specific Puppet 4 compatibility fixes without upgrading to the latest upstream code (in the rare case when the upstream master branch only supports Puppet 4+).

All changes are tracked in our manifests repo => please review 415b6acf752831ef69e52dc42f9ea0508e8a2516..5ea8b7681089e40506d4e5e171d1ed2e2c514546 (and of course the 13 submodule updates referenced by these commits). https://docs.puppet.com/puppet/4.5/upgrade_major_pre.html#enable-the-future-parser-and-fix-broken-code is a good starting point to learn about the "future" parser; it should explain most, if not all, of the changes I had to apply.

The only remaining problem AFAICT is a dependency issue, see https://git-tails.immerda.ch/puppet-tails/commit/?id=d07bfba43f39deaffce08b49b1a68dca4c24a22a, https://git-tails.immerda.ch/puppet-tails/commit/?id=59fb58f7e1a2f613578f7e028b81e44d68d33931, and https://git-tails.immerda.ch/puppet-apt/commit/?id=edeab55358e73cb9cab42ec6cbe34b35dbbd7ef6. I've requested help about this on https://gitlab.com/shared-puppet-modules-group/apt/issues/26 and will implement whatever better solution I'm recommended there. I don't think it's a blocker nor worth a dedicated ticket: worst case we have to run Puppet one more time, during the initial setup of a system, until things converge; our stuff has never converged in one single run anyway. If you feel differently, I'm all ears :)

If anything is unclear / if you have any question, let me know.

#17 Updated by groente 3 months ago

  • Assignee changed from groente to intrigeri
  • % Done changed from 50 to 80
  • QA Check changed from Ready for QA to Info Needed

the reprepro manifests have a lot of integers that are now declared as strings, particularly at commits

  • e3a073a3fedad32c0f2a539d13a259f5f16e54f5
  • 0839691743b2a1f193ef27b0a33607accc0d7fad
  • aa4a13b5b54b0c3b3109f6b43e4304e17453eaa5
  • 7563d33379e92dce6eb255e2502903e4fda84337
  • 4d65663061f818be20fa04a6c0b781f536e98318
  • 6d36bc8549885001e9dbb0c7b49cb5b6b0e0c43e

is there a particular reason these should be strings?

#18 Updated by groente 3 months ago

#19 Updated by intrigeri 3 months ago

  • Assignee changed from intrigeri to groente
  • QA Check changed from Info Needed to Ready for QA

the reprepro manifests have a lot of integers that are now declared as strings, particularly at commits

All these were fixed later and are not strings anymore in current puppet-tails.git: they now have proper data types and often stronger validation constraints than what we used to have. For details, see git log -p --grep "Use more appropriate data types" :)

#20 Updated by groente 3 months ago

  • Status changed from In Progress to Resolved
  • Assignee changed from groente to intrigeri
  • % Done changed from 80 to 100
  • QA Check changed from Ready for QA to Pass

okay, that was a bit too narrow a focus on that set of commits, thanks for clearing it up :)

Also available in: Atom PDF