Project

General

Profile

Feature #12616

Feature #5630: Reproducible builds

Document our vagrant based build setup in Jenkins

Added by bertagaz 6 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Build system
Target version:
Start date:
05/30/2017
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
master
Type of work:
Contributors documentation
Blueprint:
Easy:
Affected tool:

Description

Once we've finalized the deployment of the vagrant based build setup in Jenkins and have polished it so that it's robust enough, we'll have to move part of what is written in the blueprint in a design documentation.

Associated revisions

Revision f2201a63 (diff)
Added by bertagaz about 1 month ago

Move part of the vagrant setup explanations from blueprint to design page.

This part is already in production.

Refs: #12616

Revision 3a1edab1 (diff)
Added by bertagaz about 1 month ago

Document specific aspects of using Vagrant in Jenkins.

Refs: #12616

Revision 4aa55151 (diff)
Added by bertagaz about 1 month ago

Reword explanation about sharing of Tails repo to the Vagrant buildbox.

Refs: #12616

Revision 6c959022 (diff)
Added by bertagaz about 1 month ago

s/clean/delete/ obsolete baseboxes.

Refs: #12616

Revision 2dc498cb (diff)
Added by bertagaz about 1 month ago

Be more precise about which baseboxes we're talking about.

Refs: #12616

Revision acd1152c (diff)
Added by bertagaz about 1 month ago

Fix end of phrase.

Refs: #12616

Revision bb8925b7 (diff)
Added by bertagaz about 1 month ago

Remove unnecessary reason to use nested VMs to build Tails in Jenkins.

Refs: #12616

Revision 1604f678 (diff)
Added by bertagaz about 1 month ago

Aerate a bit the layout and point to Jenkins specifics page for build setup.

Refs: #12616

Revision ceb58920 (diff)
Added by bertagaz about 1 month ago

Clarify syntax.

Refs: #12616

Revision b2ad965b (diff)
Added by bertagaz about 1 month ago

Clarify and add links showing how we use our build options in Jenkins.

Refs: #12616

Revision 764289ed (diff)
Added by bertagaz about 1 month ago

Clarify extproxy usage in our Jenkins infra.

Refs: #12616

Revision fc94ebba (diff)
Added by bertagaz about 1 month ago

Link back moved parts of reproducible build's blueprint.

Refs: #12616

Revision 9de3812a (diff)
Added by intrigeri about 1 month ago

Fix typo (refs: #12616).

Revision ab3df30b (diff)
Added by intrigeri about 1 month ago

Fix wrong info (refs: #12616).

We don't do that for "every Tails release".

Revision 60e2b31d (diff)
Added by intrigeri about 1 month ago

Be more accurate (refs: #12616).

There's no such thing as an apt-cacher-ng "shared in our infra":
that's only correct in the context of the lizard virtualization host.

Revision 760c09bc (diff)
Added by intrigeri about 1 month ago

Restructure to explain why we do garbage collection (refs: #12616).

Previously we described the implementation details without explaining
the purpose.

Revision 3e12fcef (diff)
Added by intrigeri about 1 month ago

Improve phrasing and Markdown formatting (refs: #12616).

Revision 8c9fb392 (diff)
Added by intrigeri about 1 month ago

Use consistent tenses (refs: #12616).

Revision aabc31e6 (diff)
Added by intrigeri about 1 month ago

Be more direct (refs: #12616).

Revision 3581453e (diff)
Added by intrigeri about 1 month ago

Add TOC (refs: #12616).

Revision bbabc91c (diff)
Added by intrigeri about 1 month ago

Fix wrong filesystem name (refs: #12616).

Revision 85e83f76 (diff)
Added by intrigeri about 1 month ago

Fix typo and grammar mistake (refs: #12616).

Revision a7b83e8b (diff)
Added by bertagaz about 1 month ago

Reimport removed necessary parts to keep in the blueprint.

Refs: #12616

History

#1 Updated by anonym about 2 months ago

  • Target version changed from Tails_3.2 to Tails_3.3

#2 Updated by bertagaz about 1 month ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from bertagaz to intrigeri
  • % Done changed from 0 to 50
  • QA Check set to Ready for QA

I've pushed two commits referencing this ticket that move some of the content from the blueprint to two different pages: contribute/build/vagrant for the generic parts, and contribute/working_together/roles/sysadmins/automated_builds_in_jenkins for what is specific to the deployment of Vagrant in Jenkins. How does it sound?

#3 Updated by intrigeri about 1 month ago

  • Feature Branch set to master

#4 Updated by intrigeri about 1 month ago

  • Assignee changed from intrigeri to bertagaz

Hi,

contribute/build/vagrant for the generic parts

I think you mean contribute/build/vagrant-setup.

There's a broken link on the first line of contribute/working_together/roles/sysadmins/automated_builds_in_jenkins. Again, please build & check the output locally before submitting for QA.

Taking a step back, it's not obvious what "clean obsolete baseboxes" means. I suggest s/clean/delete/.

s/one copy of a basebox/one copy of a given basebox/ to remove some ambiguity.

"and would result in failures of subsequent builds" does not work grammatically speaking. I suggest making the subject of the verb explicit (and thus correct).

I don't think "mountpoints" are a problem, but mounted filesystems are.

"For simplicity and security reasons, we are using nested virtualization": I don't think we have a strong case for simplicity here; I can see how one could argue this way, but it's not obvious and not spelled out in the doc, so I suggest dropping the "simplicity" part.

"In our Jenkins setup we instead use" builds an opposition between something undefined and our Jenkins setup. Either specify what is that undefined other thing, or drop the comparison. Also, please point to the place where "use an existing, external apt-cacher-ng" is implemented for Jenkins builds. Finally, I don't get how #11979 has anything to do with the Jenkins setup.

In various places you mention using this or that Rake target; good. But I think this should link to the actual pieces of https://git-tails.immerda.ch/jenkins-jobs/ (or whatever relevant repo) where things are actually set up as described.

Dropping stuff from the "How we will make it happen" section of wiki/src/blueprint/reproducible_builds.mdwn has two problems:

  1. this text was our proposal for MOSS and I think it should remain there;
  2. you dropped "First of all" and left "Secondly", which doesn't work.

I could not find any place where we link to contribute/working_together/roles/sysadmins/automated_builds_in_jenkins so right now it's not discoverable on our website (== well-hidden except for people who'll find it by change in Git). Please take the PoV of the target audience (i.e. user-centric approach) when self-evaluating your work before sending for QA.

Is "The Vagrantfile shares the local clone of the Tails repository inside the basebox" correct? Or is this sharing only happening when we actually start a VM based on a build basebox? (I'm not 100% clear with the Vagrant terminology myself, sorry.)

#5 Updated by intrigeri about 1 month ago

  • QA Check changed from Ready for QA to Dev Needed

#6 Updated by bertagaz about 1 month ago

  • Assignee changed from bertagaz to intrigeri
  • % Done changed from 50 to 60
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:

There's a broken link on the first line of contribute/working_together/roles/sysadmins/automated_builds_in_jenkins. Again, please build & check the output locally before submitting for QA.

a8bfc1ecbfdffda395e98080fa1df396f39a3fab

Taking a step back, it's not obvious what "clean obsolete baseboxes" means. I suggest s/clean/delete/.

6c959022769327d6faf8c6e35b13695031dab4a7

s/one copy of a basebox/one copy of a given basebox/ to remove some ambiguity.

2dc498cbc5ee46c719e29fc0c3e867e9f5629878

"and would result in failures of subsequent builds" does not work grammatically speaking. I suggest making the subject of the verb explicit (and thus correct).

I don't think "mountpoints" are a problem, but mounted filesystems are.

bc7bf085e85bf239363c83ad84a6974ca38552e5

"For simplicity and security reasons, we are using nested virtualization": I don't think we have a strong case for simplicity here; I can see how one could argue this way, but it's not obvious and not spelled out in the doc, so I suggest dropping the "simplicity" part.

bb8925b71076adf72a0067a5a549c1ce98c209cb

"In our Jenkins setup we instead use" builds an opposition between something undefined and our Jenkins setup. Either specify what is that undefined other thing, or drop the comparison.
Also, please point to the place where "use an existing, external apt-cacher-ng" is implemented for Jenkins builds. Finally, I don't get how #11979 has anything to do with the Jenkins setup.

764289ed305b0d7443da617f0209d892f69923be

In various places you mention using this or that Rake target; good. But I think this should link to the actual pieces of https://git-tails.immerda.ch/jenkins-jobs/ (or whatever relevant repo) where things are actually set up as described.

b2ad965b1011f0403d9be968d63021a05d530403

Dropping stuff from the "How we will make it happen" section of wiki/src/blueprint/reproducible_builds.mdwn has two problems:

  1. this text was our proposal for MOSS and I think it should remain there;
  2. you dropped "First of all" and left "Secondly", which doesn't work.

I could not find any place where we link to contribute/working_together/roles/sysadmins/automated_builds_in_jenkins so right now it's not discoverable on our website (== well-hidden except for people who'll find it by change in Git). Please take the PoV of the target audience (i.e. user-centric approach) when self-evaluating your work before sending for QA.

1604f6789288f22845b8cf14363286285c3dcaca

Is "The Vagrantfile shares the local clone of the Tails repository inside the basebox" correct? Or is this sharing only happening when we actually start a VM based on a build basebox? (I'm not 100% clear with the Vagrant terminology myself, sorry.)

4aa55151ba4bd3d26e113ab0c14a071c3c4632ce

#7 Updated by bertagaz about 1 month ago

intrigeri wrote:

  1. this text was our proposal for MOSS and I think it should remain there;
  2. you dropped "First of all" and left "Secondly", which doesn't work.

Sorry, forgot this one. What about fc94ebba87b438641fe0a2afa0cd6a575e5538bb ?

#8 Updated by intrigeri about 1 month ago

  • Assignee changed from intrigeri to bertagaz
  • % Done changed from 60 to 80
  • QA Check changed from Ready for QA to Dev Needed

Thanks for fixing all these problems! I've pushed a bunch of other fixes and improvements => please review.

bertagaz wrote:

intrigeri wrote:

  1. this text was our proposal for MOSS and I think it should remain there;
  2. you dropped "First of all" and left "Secondly", which doesn't work.

Sorry, forgot this one. What about fc94ebba87b438641fe0a2afa0cd6a575e5538bb ?

That commit goes a bit further in treating our proposal as a working document that we can update incrementally. This could have been an option, but we didn't do that so far, and doing it for one part of the implementation and not for the other bits is confusing. So I'd rather not do that at all => please revert our proposal to its original state. It's fine if some text is duplicated between our original proposal and the final design doc. For extra clarify, again this comment of mine is only about the "How we will make it happen" section (some bit further down on the blueprint were WIP design documentation and it's great that you moved it to a better place :)

#9 Updated by bertagaz about 1 month ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:

Thanks for fixing all these problems! I've pushed a bunch of other fixes and improvements => please review.

fine with me.

bertagaz wrote:

Sorry, forgot this one. What about fc94ebba87b438641fe0a2afa0cd6a575e5538bb ?

That commit goes a bit further in treating our proposal as a working document that we can update incrementally. This could have been an option, but we didn't do that so far, and doing it for one part of the implementation and not for the other bits is confusing. So I'd rather not do that at all => please revert our proposal to its original state. It's fine if some text is duplicated between our original proposal and the final design doc. For extra clarify, again this comment of mine is only about the "How we will make it happen" section (some bit further down on the blueprint were WIP design documentation and it's great that you moved it to a better place :)

a7b83e8b

#10 Updated by intrigeri about 1 month ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 80 to 100
  • QA Check changed from Ready for QA to Pass

LGTM, thanks!

Also available in: Atom PDF