JayeX - Cloud Native CI/CD Built On Kubernetes – Jenkins X Blog

Blog: JayeX Most Valuable Contributor 2026 🏆

Wed, 08 Apr 2026 00:00:00 +0000

The Continuous Delivery Foundation gives awards to recognize all the work that makes this community and the progress of Continuous Delivery possible.

Vote on someone for the award for JayeX Most Valuable Contributor 2026

This award recognizes excellence of technical contributions to JayeX. This individual commits key pieces to projects and, more importantly, contributes in a way that benefits the project neutrally as a whole.

You vote by someone by adding a thumbs on the comment with their name in this issue. Voting is now open and will close on Tuesday, May 5, 2026.

Winners

The winners are announced during cdCon: May 18–20, 2026.

More information

Blog: Jenkins X changes name to JayeX!

Thu, 19 Mar 2026 00:00:00 +0000

Background

During the early days of Jenkins X the Jenkins part of the name made sense for two reasons: Jenkins was used as the main pipeline engine and the development was done within CloudBees, the company that also is the main developer of Jenkins.

Why change?

Nowadays the Jenkins X name is more of a burden than boon marketing wise: It attracts people interested in Jenkins, who will dissapointed. It detracts people that are looking for an alternative to Jenkins, who might actually find what this project offers attractive.

What happens now?

Technically nothing happens immediately. We will by and by replace the Jenkins X name with JayeX within the project without affecting the functionality. Any remaining integrations with Jenkins should be seen as deprecated though and will eventually be removed.

We will renew our marketing efforts using the new name.

Blog: JayeX Most Valuable Contributor 2026 🏆

Thu, 05 Mar 2026 00:00:00 +0000

The Continuous Delivery Foundation gives awards to recognize all the work that makes this community and the progress of Continuous Delivery possible.

Nominate someone to the award for JayeX Most Valuable Contributor 2026

You nominate people by adding a comment with their name in this issue. Nominations are now open and will close on Thursday, April 2, 2026

Winners

The winners are announced during cdCon: May 18–20, 2026.

More information

Blog: Migrate to Google Artifact Registry

Mon, 06 May 2024 00:00:00 +0000

Google has announced that container registry will be shut down some time after March 18, 2025. For GKE clusters created with version 1.12.0 or later of terraform-google-jx it’s unlikely that anything needs to be done, but for older clusters you should upgrade your cluster while considering our advice regarding migration from container registry to artifact registry.

If you are using a Google Service Account to run terraform you need to add the role requirement roles/artifactregistry.admin. See our guide regarding Google Service Account for details.

Blog: Improve your changelogs

Wed, 24 May 2023 00:00:00 +0000

Background

A standard part of the Jenkins X pipelines since a long time is the execution of jx changelog create that takes the commit messages between the release currently being created and the previous one and creates a change log from these. The change log is then stored as a release note in GitHub or other git provider.

During the last year some improvements have landed in various Jenkins X components to improve the changelogs and their usefulness. So I’ll take this opportunity to describe these improvements and also in general give hints to how to get useful changelogs.

Overview of major improvements

Changelogs haven’t been very informative with regard to upgrades, ie those applied with jx promote or jx updatebot. One example of this is the release notes of jx after the split out of most functionality to plugins. Lately these have improved due to new functionality to propagate changelogs via pull requests.

One place where changelogs have been completely lacking is in cluster repositories. But using the functionality for propagation of changelogs and some changes in jx boot job you can now a get a changelog for every successful application of changes in a cluster.

Example

An example of what this functionality achieves can be seen in a release of jx:

https://github.com/jenkins-x/jx/releases/tag/v3.10.81

If you scroll past the boilerplate installation instructions you first see the changelog of jx itself generated from commit messages: https://github.com/jenkins-x/jx/compare/v3.10.80...v3.10.81

Then there is a separator followed by the changelog of jx-gitops. If you look in the commit messages referred to above you will see a link to the pull request on jx for this upgrade: https://github.com/jenkins-x/jx/pull/8564. As you can see this pull request includes the changelog for version 0.14.8 of jx-gitops as generated by jx changelog create.

Essentially what happens is that jx changelog create apart from storing the release information of jx-gitops also stores it in a file. This file is then put in the pull request on jx by jx updatebot. Finally jx changelog create looks for changelogs in the pull requests referred to in the merge commit.

How to write commit messages

To get a good changelog the first requisite is to write good commit messages. jx changelog create expects commit messages to adhere to the Conventional commits standard. It is the first line of each commit message that is used, with a couple of exceptions I’ll describe below.

When following this standard the first line should adhere to this format:

<type>[optional scope]: <description>

While jx changelog create can handle any type some will be expanded for a better looking changelog:

feat: New Features
fix: Bug Fixes
perf: Performance Improvements
refactor: Code Refactoring
docs: Documentation
test: Tests
revert: Reverts
style: Styles

Since the description is used verbatim in the changelog it should be meaningful to the audience of the changelog. Is for example fixed typo meaningful? Maybe it is better to write what you try to achieve. While doing this it is good to know that duplicate descriptions are ignored by changelog. So if you add the same description again when doing a fix doesn’t mean that you get duplicate lines in the changelog.

An optional scope is prefixed to the description in the changelog.

The other common part of the commit message that is reflected in the changelog are references to issues. These can be put anywhere in the commit message. These issues are then linked to both for the specific commit and in a separate list of affected issues.

An example:

feat: support dependency updates

fixed typo

relates to #1234

A less common part of a commit message that affects the changelog regards breaking changes. A footer to a commit message of the format BREAKING CHANGE: <description> can be added and this description will be put at the top of the changelog under the heading BREAKING CHANGES. If you add an exclamation mark before the colon in the first line without the breaking change footer the ordinary description will be put under this heading.

A couple of examples regarding braking changes:

fix: upgraded library foo

BREAKING CHANGE: don't support http anymore

#98765

fix!: removed insecure method fubar

resolves #56789

Good to know is that jx release version also conforms to Conventional Commits so the commit messages also affects which part of the semantic version is increased for a release.

There are of course other uses of good commit messages than to generate changelogs. While code itself together with comments should be self-explanatory with regard to what is done and how I often resort to commit messages to understand why a change was made. I typically use the git blame functionality for this. This is another reason to not use messages only saying something like fixed typo, since that typically isn’t a useful answer to the question of why for a future developer.

For more tips regarding writing good commit messages see for example How to Write Better Git Commit Messages.

Manually edit changelog

Note that since changelogs for dependencies are propagated through pull requests you can edit the changelog for an application manually in the pull request before it is merged.

Configuration

Some of the features described above work out-of-the-box while others require some tweaking.

Changelog for cluster repository

Apart from the other advantages of switching from kubectl apply to kpt live apply for reconciliation of cluster repository with the cluster you also get a changelog generated for each successful reconciliation.

This changelog also includes a list of upgraded / added / deleted releases and the associated versions.

See a previous blogpost for more information.

Reuse pull requests

With the default settings one pitfall with the propagating the changelog via pull requests is if not all pull requests are merged. Let’s say version 1.0.0 of an application is released and pull request with a big changelog is created for the production namespace. But before it is merged an error is detected and version 1.0.1 is released and a PR for the version is created which only includes the changelog for the bug fix. When that PR is merged only the changelog for the bug fix ends up in the changelog for the cluster.

To mitigate this you can enable reuse of pull requests. This means that if a PR for upgrading a dependency already exists when executing jx promote or jx updatebot the commits for that PR are replaced with one for the new upgrade. The existing changelog in the PR is kept though and the changelog for the new version is appended. This way no changelog is missed.

Enabling reuse of pull requests for jx promote is done in jx-requirements.yaml by setting reusePullRequest to true for an environment. It can also be done in the same way when configuring environments others ways. See https://jayex.io/v3/develop/environments/config/ for more details about configuring environments.

More information about reuse of pull requests in a future blog post.

Custom pipelines

In the pipelines defined in https://github.com/jenkins-x/jx3-pipeline-catalog/ the necessary incantations of jx changelog, jx promote and jx updatebot are made for promoting changelogs. But if you have a custom release pipeline you might need to do some modifications. In particular jx changelog create need to save the changelog to a file. This is done by adding an argument like --output-markdown=../changelog.md. To have this changelog added to the PR the option --add-changelog=../changelog.md needs to be added to jx promote (or jx updatebot if applicable).

Jira as issue tracker

In the examples of commit messages above the references to issues are to the git provider. This is the default, so if you for example are using GitHub as your git provider and also use the issue tracker in GitHub no extra configuration is needed.

But there also is support for Jira as issue tracker. If you enable this support you can refer to issues both at the git provider and in Jira.

You enable Jira support and configure it in the jx-requirements.yaml of the cluster repository. The issueProvider field is added under spec.cluster:

@@ -13,5 +13,9 @@ spec:
     gitKind: github
     gitName: github
     gitServer: https://github.com
+    issueProvider:
+      jira:
+        serverUrl: https://myjira.atlassian.net
+        userName: jirauser@example.com
     project: "123456789"
     provider: eks

For security reasons the credentials for the user that will fetch data from Jira is not put there. Instead jx changelog expects it in the environment variable JIRA_API_TOKEN. The easiest way to supply this environment variable is by adding it to the secret jx-boot-job-env-vars in namespace jx since the content av this secret are exposed as environment variables to all standard build pipelines. If the secret doesn’t exist you can create it yourself like this:

# lets make sure we are in the jx-git-operator namespace
jx ns jx-git-operator

kubectl create secret generic jx-boot-job-env-vars --from-literal=JIRA_API_TOKEN='S!B*d$zDsb'

The Terraform module for Jenkins X in EKS also have support for adding variables to jx-boot-job-env-vars.

More customizations

In the documentations for jx changelog create information about more customisations can be found.

References

The enhancement proposal for propagation of changelogs

Blog: Reconcile with kpt live apply

Thu, 09 Mar 2023 00:00:00 +0000

Since the dawn of Jenkins X 3 the default last step of reconciling the state of the files in your cluster repository to your cluster has been to execute kubectl apply. You can find more details about this here.

There are some drawbacks with kubectl apply though. The one that made me start looking for alternatives was that if you remove a resource from your cluster repository it may not be removed from your cluster. The way deletion works with kubectl apply is that it is handed the option --prune which will remove resources that are not in the manifests. Except that it doesn’t always work as expected. It will only remove certain kinds of resources defined in kubectl. In my case I removed an HorizontalPodAutoscaler from my cluster repository, but it wasn’t removed from my cluster.

When trying to find a solution to this I first tried to override this default list in kubectl of things to prune, but this turned out to be difficult in the general case. I also tried the already existing alternative of using kapp to apply the manifests, but I couldn’t get that to work. Looking for other options I settled for kpt live apply.

Configuration

For all the functionality described here to work you need to have a cluster that is upgraded later than January 24th 2023.

You enable the use of kpt live apply by adding

KUBEAPPLY = kpt-apply

to the Makefile of your cluster repository anywhere before include versionStream/src/Makefile.mk. This works both in a dev cluster and in a remote cluster. (With the caveat that all bets are off if you have done changes yourself to versionStream/src/Makefile.mk.)

After you have pushed this you can watch the log of the boot job using jx admin log as usual. When kpt live apply has been executed the first time you can optionally add let kpt live apply --dry-run be executed for pull requests, giving a better assurance that the content of a pull request will apply nicely. But this only work properly for pull requests to the dev cluster, since it’s in the dev cluster the pull request pipeline will execute. So in a remote clusters this should not be enabled. That said this functionality can be enabled by adding

PR_LINT = kpt-apply-dry-run

to the Makefile of your cluster repository.

Waiting for resources to be reconciled

While the fact that I can be sure that removed resource actually are gone is the killer feature of kpt live apply for me the opposite is true as well. With this I mean that kpt live apply(and as a consequence) the boot job won’t succeed unless all resources are verified to have been applied. This includes for example that the rollout of new applications versions with deployments have succeeded.

This can be seen as a problem though: the default is that kpt live apply tries for 15 minutes before timing out with an error. But since the boot job is normal k8s job the job controller will create new pods on failure up to the backoff limit, that is set to 4. This can block the application of subsequent changes to the cluster repo.

Tagging and release notes

When activating kpt live apply you will also see that tags are added to the cluster repository upon successful application of changes. These tags can have 3 prefixes:

crd for custom resource definitions
cluster for cluster wide resources
ns for namespaced resources (the most common)

For ns tags release notes will also be generated and added to the git provider (for example GitHub). This means that you can see what has changed in your cluster by looking at the releases of the cluster repository.

One handy use of this is that you can get a notification from GitHub when changes has been successfully deployed by going to the Watch menu for the repo, select Custom and check Releases. If you get the notification by email or on the GitHub site depends on the notification settings for your GitHub account.

This functionality can be further customized by adding scripts to the extensions directory of your cluster repository.

crd-reconciled will be executed after a crd tag has been added
cluster-reconciled will be executed after a cluster tag has been added
ns-reconciled will be executed after a ns tag has been added

The default behaviour is equivalent to adding the following script as ns-reconciled:

#!/bin/sh

jx changelog create --tag-prefix ns-

Here is a somewhat hacky variety that sends out a mail with the release notes similarly to what you can subscribe to for yourself:

#!/bin/sh
set -xe

jx changelog create --tag-prefix ns-

token=$(kubectl get secret jx-boot -o jsonpath="{.data['password']}" | base64 -d)
repourl=$(kubectl get secret jx-boot -o jsonpath="{.data['url']}"    | base64 -d)
owner="$(echo $repourl | cut -d/ -f4)"
repo="$(echo $repourl | cut -d/ -f5 | cut -d. -f1)"
(
cat <<EOF
to: changelog@example.com
From: dev@example.com
MIME-Version: 1.0
Return-Path: <>
Precedence: Bulk
Content-Type: text/html; charset=utf-8
Subject: Deploy to kubernetes cluster

<html>
<body>
<h1>Deploy to kubernetes cluster</h1>
EOF
curl -s  -H "Authorization: Bearer $token" https://api.github.com/graphql -X POST -d '{"query":"query { repository(owner:\"'$owner'\", name:\"'$repo'\") { release(tagName: \"ns-'${TS}'\") { descriptionHTML }}}"}' | yq .data.repository.release.descriptionHTML
echo "</body></html>"
) | sendmail -t -S smtp.example.com

To adapt for your own use, the minimal changes would be to find and replace the smtp server and mail addresses now added for the domain example.com. Don’t forget to make the script executable with chmod +x ns-reconciled.

Blog: Foreign aliases

Thu, 09 Feb 2023 00:00:00 +0000

Background

In an organisation with many repositories and developers that are frequently shifting the maintenance of OWNERS and OWNERS_ALIASES files can be tedious. In the passing year a couple of functionalities has been added to help with this.

Foreign aliases

To avoid maintaining the OWNERS_ALIASES file in many repositories you can now refer to the OWNERS_ALIASES file in another repository. In the Jenkins X project we have the main OWNERS_ALIASES file in the jx-community repository. So in the jx repository the OWNERS_ALIASES file only looks like this:

foreignAliases:
- name: jx-community

The organisation defaults to be the same as for the repository, but can specify as well. So in the jx-project repository the OWNERS_ALIASES file looks like this:

foreignAliases:
- name: jx-community
  org: jenkins-x

Using the filed ref you can also specify a branch or tag to use instead of the default one of the repository.

OWNERS and OWNERS_ALIASES in new repositories

When creating or importing a repository using jx project the default content of OWNERS and OWNERS_ALIASES isn’t that useful since only the current user are put in the files.

If you create your own quickstarts you place the OWNERS and / or OWNERS_ALIASES files with the content of your liking in those.

A recent new functionality is that you can put OWNERS and / or OWNERS_ALIASES files in the extensions directory of your cluster repository. These files will then be used as the default content of the files in new repositories.

Blog: Project ideas for Google Summer of Code 2023 ☀️

Mon, 06 Feb 2023 00:00:00 +0000

We have put together some project ideas as part of our application to participate in the Google Summer of Code 2023 program.

1. CD events integration with Jenkins X

Description

The cdEvents project standardises the way systems talk to each other, which enables Interoperability between systems so they speak a common language through the cdEvents spec in the event. Creating a capability in Jenkins X that can receive and sent a cdEvent would benefit the project and the DevOps ecosystem in general, by stopping glue code used to integrate systems and power innovation by letting end users swap out tools with no effort.

Expected Outcomes

Ability to receive cdEvents
Ability to parse cdEvents so Jenkins X understands them
Ability to send cdEvents

Recommended skills

Jenkins X, Kubernetes, golang, cdEvents

Mentors

Brad McCoy

Resources

Expected Size of project

350 hours

Difficulty rating

Hard

2. Implement drift detection (gitops)

Description

Jenkins X only applies changes to cluster when contents of the gitops repository changes as part of a git pull request or git commit. This does not satisfy one of the requirements of the gitops model where we need continuous reconciliation. It would be nice to detect drift between the current state (kubernetes) and the desired state (git) and apply only those changes. This has the side effect of making the boot job faster.

Also, our bootjob is made up of a makefile which calls cli commands written in golang, instead it would be desirable to move to a controller based approach similar to other tools in the kubernetes ecosystem.

Expected Outcomes

Drift detection in Jenkins X
Configurable interval when Jenkins X will do a drift detection and apply the changes
Updated documentation

Recommended skills

Kubernetes, golang, Jenkins X, kubebuiler, basic understanding of gitops.

Mentors

Resources

Expected Size of project

350 hours

Difficulty rating

Hard

3. RBAC (Role Based Access Control) in Jenkins X UI

Description

In the last GSoC project, we created a modern UI which has more functionalities than the older viewer UI. However, as we added the ability to start/stop pipelines from the new UI, this opened up an issue around access. Users of Jenkins X would like to restrict who can start/stop pipelines from the UI.

As we add more features to the UI involving repositories, this would also require ability to restrict access using RBAC (Role Based Access Control). In addition, an audit trail would also help the users keep track of activities in the UI.

Expected Outcomes

Fully functional Jenkins X Dashboard running in the kubernetes cluster
Drop in replacement for existing read only UI
Ability to restrict access in the UI
Audit trail of who took what action in the UI
Updated documentation

Recommended skills

golang, basic understanding of sveltejs and sveltekit (or any modern js framework like react, vue. angular or solid) and kubernetes

Mentors

Resources

Expected Size of project

350 hours

Difficulty rating

Medium

4. Simplifying Jenkins X pipeline syntax

Description

Jenkins X uses the raw tekton pipeline syntax for pipelines. The issue with this syntax is that it’s too verbose. The verbosity makes sense for low level pipeline building libraries like tektoncd, but for an user facing tool like Jenkins X, it makes the pipeline files too big for no apparent reason. We want to investigate a new syntax which will set the boilerplate so that the syntax is recognized by tekon, but is still user friendly like gitlab/github CI.

Expected outcomes

Simpler and easier to understand pipeline syntax (for developers)
More verbose pipeline syntax for power users (if there is a need)
Updated documentation

Recommended skills

Golang, kubernetes, tekton, cue

Resources

https://github.com/cue-lang/cue

Mentors

Expected Size of project

350 hours

Difficulty rating

Medium

5. Multi-tenancy in Jenkins X

Description

Installing multiple versions of Jenkins X in a single kubernetes cluster is not supported. There are both architectural and scalability issues around it. This would be beneficial for small teams within organizations who want to own their entire CI/CD platform instead of relying on a central release management team which can lead to more productive teams.

Expected outcomes

A multi-tenant enabled Jenkins X install (jenkins X in different namespaces or a central jx install controlling pipelines running for different tenants)
Better scaling and security model for Jenkins X
Updated documentation

Recommended skills

Golang, kubernetes, event driven architecture

Resources

https://github.com/jenkins-x/enhancements/pull/46

Mentors

Expected Size of project

350 hours

Difficulty rating

Hard

Next Steps

If Jenkins X gets selected, we will create a followup blog with additional details.

Blog: GSoC 2022 Final Report: Building Jenkins X UI

Sun, 13 Nov 2022 00:00:00 +0000

Jenkins X New UI

It is a web application built with Golang for the backend and Sveltekit for the frontend, both of which are built together and used in the same container. To function properly, it must be installed as a helm chart with Jenkins X CRDs.

🌟 It has light and dark themes.

Why need a new UI?

A good UI is essential for a CI/CD tool, as not everyone is familiar with the CLI. The current UI (jx-pipeline-visualizer) is a read-only UI, the user can view the logs of PipelineActivity but neither can start nor stop the pipeline.

Features that the UI will provide:

Start and Stop a PipelineActivity.
Have an audit trail.
A graphical representation of PipelineActivity.
RBAC to limit access to certain functionalities.

New Jenkins X UI focus on Simplicity, Security and a Superb User Experience.

This is NOT GA (General Availability) yet. Visit the project repo here to try it.

How to use it?

Go to the helmfiles/jx/helmfile.yaml in your cluster repo.

Replace jx-pipelines-visualizer chart with the jx-ui chart.

- chart: jxgh/jx-ui
  version: <latest-version>
  name: jx-ui
  values:
  - ../../versionStream/charts/jxgh/jx-ui/values.yaml.gotmpl
  - jx-values.yaml

Do git push.
Add the code snippet to ~/.config/ngrok/ngrok.yml under tunnels:.
```
ui:
proto: http
addr: 9200
schemes:
  - http
```
Add that domain to the ingress file, Run the following command and replace dashboard.jx.change.me with ngrok link.
```
kubectl edit ing jx-ui
```
And boom, visit localhost:9200 and it works, you should see the homepage screen.

Work Done

Stop a running or pending PipelineActivity from UI

We have added a button in pipelines page and pipelineDetails page, it asks for confirmation and on selecting Yes it will stop the PipelineActivity.

We can stop the PipelineActivity from the pipelines tables.

We can also stop the PipelineActivity from the pipelines details page.

Issue: https://github.com/jenkins-x/jx-ui/issues/28

PRs:

Show message with status of the PipelineActivity

When a pipeline is cancelled or it has timedout, we update the status of the pipeline to cancelled or timed out. However, we do not show the reason why a job was timed out. That information should be displayed on the pipeline details page (along with the step which has timed out). Issue: https://github.com/jenkins-x/jx-ui/issues/23

PRs:

Implement a DAG for PipelineActivity

To make the UI more user friendly and to explain the sequence of the steps in a PipelineActivity, we have added a DAG (Directed Acyclic Graph) chart to graphically visualize the PipelineActivity.

We used D3.js library to create a custom component which create a DAG chart of PipelineActivity for us. D3-hierarchy is an important part of our component. We’re unit testing the component using vitest.

Issue: https://github.com/jenkins-x/jx-ui/issues/48

PRs:

What’s next?

The UI project is just started, we have also start a seperate UI-sig which you can join from calendar.
There are some amazing features that we are planning to add to the new UI:
- Dynamically update the list of pipelines
- Upgrade version stream from the UI
- Import repositories using the UI
- Break down pipeline logs into individual steps
- Mask secrets in the logs
- Display logs for cancelled and timed out pipelines

To read more read the issues created here

Acknowledgements

I never imagined I would be this sad after finishing an internship. Just as Osama noted in his work report, this was the most difficult experience we have had thus far, but the encouragement of the jx community and the direction of our mentor made it the best educational experience I have ever had.

I learned a lot here, but there are still many things I need to learn (I haven’t finished my 30-day plan yet). I knew I could learn a lot from open source, but I didn’t expect this much. In addition, I think the CD Foundation is very special, and the community and the ways in which we can collaborate with other projects are also very valuable.

I started my intern by attending conferences like kubecon and CD Summit, we started UI-SIG, and in the last 2 months we’ve given 3 talks about Jenkins X. As I mentioned above there are still thing to do in the UI project and since we all loved the GSoC program, I’d love to became a part of it in future with Jenkins X.

Again Big Thanks to all my mentor Ankit, Christoffer, Marten, Tom and gsoc fellow Osama and JX community for such a great opportunity.

Blog: GSoC 2022 Final Report: Improving Supply Chain Security

Tue, 08 Nov 2022 00:00:00 +0000

Project Description

Supply chain security is a rising concern in the current software era. Securing the software supply chain encompasses vulnerability remediation and the implementation of controls throughout the software development process. Due to massive increase in attacks on software supply chain and the diversity of its types, Jenkins X has to make efforts to ensure that the build process is secure. As part of securing Jenkins X installation by default I worked on both securing our own components and enabling our users to use these features in their build and release steps..

Work Done

The work done so far covers these four sections.

Enhancing the jx version output
Integrating with Tekton Chains to sign TaskRuns and PipelineRuns
Software Bill of Materials (SBOM)
Signing Jenkins X artifacts

Enhancing the `jx version` output

Description:

A first step towards securing Jenkins X supply chain is to increase the amount of information gained from running jx version command.

Implementation

The issue created for this task is here. The PR to fix it is here.

Integrating with Tekton Chains to sign TaskRuns and PipelineRuns

Description:

As Jenkins X uses tekton as its pipeline execution engine, TaskRun and PipelineRun are considered the key components of Jenkins X pipeline activities and steps Tekton Chains monitors the execution of all TaskRun and PipelineRun inside the cluster and takes a snapshot upon completion of each of them to sign with user-provided cryptographic keys and store them on the backend storage. The payload and signature cn be verified later using cosign verify-blob.

Implementation

I used the helm chart developed by Chainguard for integrating Chains with Jenkins X. To integrate the chart and added support for it on jx3-versions to make installation of helm chart easy for our users. The list of PRs for this:

PR	Short Description
https://github.com/jenkins-x/jx3-versions/pull/3359	Supporting Tekton Chains helm chart with jx3-versions
https://github.com/jenkins-x/jx-docs/pull/3660	Documentation on how to install and integrate Tekton Chains with Jenknins X

NOTE: The integration is tested only on k3s cluster and work is in progress to test it on GKE and EKS.

Software Bill of Materials (SBOM)

Description:

Software Bill Of Materials (SBOM) is a complete formally structured list of the materials (components, packages, libraries, SDK) used to build (i.e. compile, link) a given piece of software and the supply chain relationships between all these materials. It is an inventory of all the components developers used to make this software. It has many formats and many generating tools but all have the same purpose in the end.

Implementation:

I first began with investigating available standards and formats for SBOMs and tools for generating them. The results of my investigation were written to a blog post here.

I’ve settled for using syft for SBOM generation and spdx as the standard format for SBOMs. Also, I’ve added those installation as pre-defined steps in the Jenkins X pipeline catalog so it will also be available for our users to use in their pipelines and applied those steps to Jenkins X own pipelines. The list of PRs for this work are:

PR	Short Description
https://github.com/jenkins-x/jx/pull/8312	Install syft using github action and generate SBOM with goreleaser
https://github.com/jenkins-x/jx3-pipeline-catalog/pull/1166	Add Syft Installation to JX3 pipeline Catalog so it can be used to generate SBOMs
https://github.com/jenkins-x/jx3-pipeline-catalog/pull/1173	Add Oras Steps to support pushing SBOMs as OCI artifacts
https://github.com/jenkins-x-plugins/jx-pipeline/pull/490	Use Syft installation step in jx-pipeline from jx3-pipelines-catalog
https://github.com/jenkins-x-plugins/jx-pipeline/pull/491	Use Oras bushing SBOMs step from jx3-pipeline-catalog
https://github.com/jenkins-x/jx-docs/pull/3643	Add documentation for supporting sbom generation and storing for other jx components and our users
https://github.com/jenkins-x/jx/pull/8351	Applying SBOM management to JX
https://github.com/jenkins-x/jx3-pipeline-catalog/pull/1190	Add grype tasks for SBOM vulnerability scanning to jx3-pipeline-catalog
https://github.com/cdfoundation/presentations/pull/44	Add materials for my talk with the CDF Supply Chain Securtiy SIG about SBOMs with Jenkins X

There is also this issue to apply all steps on our active JX repositories here (WIP as there are many repositories to be updated) and the list of PRs are referenced in the issue (except those of JX because they included testing and fixing PRs at the beginning).

Signing Jenkins X artifacts

Description:

Jenkins X is a collection of many components and tools that are used to provide the end-to-end CI/CD solution. Those components are used by our users and are also used by Jenkins X itself. It is important to sign those components to ensure the integrity of the components and the supply chain security of Jenkins X.

Implementation:

There are a set of tools developed by sigstore to sign and verify artifacts. The most used tool in my implementation is cosign. I first started with signing the Jenkins X CLI with keys stored in github secrets and then moved to signing the Jenkins X docker images. After that, I switched to use cosign keyless signing (an experimental feature using other Sigstore components (rekor and fulcio) to save end users from the hassles of key (mis)management ) The list of PRs for this work are:

PR	Short Description
https://github.com/jenkins-x/jx/pull/8432	Sign goreleaser artifacts with cosign
https://github.com/jenkins-x/jx/pull/8441	Sign published images with cosign
https://github.com/jenkins-x/jx/pull/8461	Keyless signing for goreleaser artifacts

What’s next?

The work for supply chain security is still in progress and there are many things to be done. At Jenkins X, we are planning to develop new features and integrate more solutions to provide a better supply chain security for our users. The list of things to be done are:

Add a cli subcommand in JX to verify sbom/signed binaries against the public keys.
Add openSSF security scorecard to Jenkins X repositories.
Integrate FRSCA work with Jenkins X.

Acknowledgements

Out of all internships and trainings I’ve done, this one was the most challenging, rewarding and educational experience. I would like to specially thank my mentor Ankit Mohapatra for his constant support and his patience to teach me a lot of his rich knowledge he gained by experience and thank the whole team of Jenkins X for their support and guidance and for welcoming me to the community. This internship was a great experience. I had a great time working on this project and I learned a lot from it. I hope to continue contributing to Jenkins X and the open-source community in the future. Thank you.

Blog: Hacktoberfest 2022

Mon, 03 Oct 2022 00:00:00 +0000

We are excited to announce that Jenkins X will be participating in Hacktoberfest again this year! Hacktoberfest is a month-long global celebration of open source software.

All backgrounds and skill levels are encouraged to participate in Hacktoberfest and join a global community of open source contributors.

Learn more about Hacktoberfest and sign up here.

Contribute to Jenkins X

We welcome your contributions to the Jenkins X project!

Issues labelled “hacktoberfest” generally indicate good first issues. However, all pull requests will count towards your Hacktoberfest challenge.

Refer to the contribution guides for making code and documentation changes.

Refer to this document to get an idea about the location of the different Jenkins X source code repositories. Normally the labels assigned to the ticket will help you in deciding which part of the Jenkins X codebase to look at. The maintainers will also try to link the relevant repository for hacktoberfest issues in the issue comment.

Once you are done with the contribution, request a review from the maintainers by adding the comment in your pull request.

/cc @ankitm123 @babadofar @msvticket @osamamagdy @rajatgupta24 @tomhobson

Ask us questions

We’re happy to help if you have any questions. Talk to us on our slack channels, which are part of the Kubernetes slack. Join Kubernetes slack here and find us on our channels:

#jenkins-x-dev for developers of Jenkins X
#jenkins-x-user for users of Jenkins X

Find out more about becoming involved in the Jenkins X community here.

We look forward to seeing you in open source, fixing all the things!

Blog: Introduction to Software Bill Of Materials

Sun, 24 Jul 2022 00:00:00 +0000

Introduction

Before going through Software Bill Of Materials (SBOMs), we need to set the ground for a rising concern in the software industry which is Software Supply Chain Security. Like traditional industries, deploying a piece of a software artifact goes through multiple stages composed of collecting source code components, libraries, tools, and processes used in those stages.

Fig. 1 https://blog.convisoappsec.com/en/is-your-software-supply-chain-secure/

A supply chain attack can occur along the chain from submitting unauthorized malicious code in your source, unauthorized injection of harmful dependencies, and even replacing packages after being built with other compromised artifacts. A more detailed explanation about those types of attacks is here

Due to its importance and being a critical issue, generating SBOM for your software adds another layer of protection to this threat.

Definition: What is SBOM?

As far as we know, developers around the world are building web applications using hundreds of third-party open-source libraries and packages. You can confidently tell that 90% of the software products around the world are built over open-source components. With that in mind, we need to keep track of using these dependencies while building our applications. What if there are vulnerabilities in the libraries we use? How to efficiently protect ourselves against it?.

Software Bill Of Materials (SBOM) is a complete formally structured list of the materials (components, packages, libraries, SDK) used to build (i.e. compile, link) a given piece of software and the supply chain relationships between all these materials.

It is an inventory of all the components developers used to make this software. It has many formats and many generating tools but all have the same purpose in the end. Example: a simple formatted SBOM of Ubuntu alpine docker image using syft

✔ Loaded image  
 ✔ Parsed image  
 ✔ Cataloged packages      [14 packages]
NAME                    VERSION      TYPE 
alpine-baselayout       3.2.0-r18    apk   
alpine-keys             2.4-r1       apk   
apk-tools               2.12.7-r3    apk   
busybox                 1.34.1-r3    apk   
ca-certificates-bundle  20191127-r7  apk   
libc-utils              0.7.2-r3     apk   
libcrypto1.1            1.1.1l-r7    apk   
libretls                3.3.4-r2     apk   
libssl1.1               1.1.1l-r7    apk   
musl                    1.2.2-r7     apk   
musl-utils              1.2.2-r7     apk   
scanelf                 1.3.3-r0     apk   
ssl_client              1.34.1-r3    apk   
zlib                    1.2.11-r3    apk

Here it shows only softwares included in the final layer of the container (default choice by syft). If we want to view a detailed SBOM with one detailed format, we can run syft alpine -o spdx-json. This will view the output as .json file following the spdx format (will discuss that later)

Use cases in Supply chain security

What makes a supply chain attack susceptible is the lack of transparency and visibility about whether the software gets affected by a recent exploit or not. This greatly affects both producers and customers of the product.

On the user’s side if they know the components of the software and that there is one component affected by certain vulnerabilities, they are better aware and ready to protect against potential attacks. This is crucial in many cases, especially with open-source tools.

On the software producer’s side, it happens a lot that they are not fully aware of all the third parties used inside the project, and in turn, they can not track vulnerabilities in the system that could pose a threat. Cases like the Log4Shell vulnerability are an example of a component (in this case, a logging library) that many producers never bothered to check because it isn’t a direct software dependency, but rather a transitive one that is depended upon by other components.

SBOM can also be useful in licensing and legal issues with some formats. SPDX (Software Package Data Exchange) standard for SBOM identifies the licenses of the used components and can be checked for compliance later.

Next, we view the different standards and formats for SBOMs and the specifications of each one. See them here

Blog: Software Bill Of Materials Formats

Sun, 24 Jul 2022 00:00:00 +0000

Prerequisite

If you don’t understand what is Software Bill of Materials (SBOM), please read this blog post first.

Different SBOM formats comparison

The National Telecommunications and Information Administration (NTIA) in the U.S. defined minimum requirements for SBOM formats:

Identifying the supplier of the software component.
Identifying the details about the version of the component.
Including unique identifiers for the component like cryptographic hash functions.
Including the relationships between all dependencies inside the component.
Including a timestamp of when and by whom the SBOM report was created or last modified

In this section, we discuss different kinds and formats for SBOM standards and make a brief comparison between them. Three commonly used standards achieved the NTIA minimum requirements for SBOM generation and each one results in a different final SBOM document.

1 - The Software Package Data Exchange (SPDX)

History:

SPDX is an open-source machine-readable format adopted by the Linux Foundation as an industry standard. The specifications are implemented as a file format that identifies the software components within a larger piece of computer software and fulfilling the requirements of NTIA. The SPDX project started in 2010 and was initially dedicated to solving the issues around open source licensing compliance. It evolved over the years to adhere supply chain security challenges and has seen extensive uptake by companies and projects in the software industry. Companies like Hitachi, Fujitsu, and Toshiba contributed to furthering the standard in the SPDX v2.2.2 specification release.

Specs:

The SPDX specification describes the necessary sections and fields to produce a valid SPDX document. Note that the only mandatory field in all spdx documents is the “Document Creation Information” section. The presence of other sections (and subset fields of each section) is dependent on your use case and the information you want to provide.

Fig 1: https://spdx.dev/wp-content/uploads/sites/41/2020/05/spdx-2.2-document.png

Document Creation Information – Denotes who created the document, how it was created, and other useful information related to its creation. It provides the necessary information for forward and backward compatibility for processing tools (version numbers, license for data, authors, etc. )
Package Information – This section provides information about the “package”. A package can be one or more files. These files could be one or more files of any type including but not limited to source, documents, binaries, containers, and so forth. The package information contains the originator, where it was sourced from, a download URL, a checksum, and so forth. It also contains summary licensing for the package.
File Information – This is information about a specific file. It can contain the file copyrights found in the file (if any), the license of the file, a checksum for the file, file contributors, and so forth.
Snippet Information – Snippets can optionally be used when a file is known to have some content that has been included from another source. They are useful for denoting when part of a file may have been created under another licenseSnippet information can be used to define licensing for ranges within files.
Other Licensing Information – Other licensing information provides a way to describe licenses that are not on the SPDX License List. You can create a local (to the SPDX document) identifier for the license and place the license text itself in the document as a well and then reference it for files just like you would a license from the license list.
Relationships – Relationships were introduced in the 2.0 specification and are a very powerful way of expressing how SPDX documents relate to one another. See an example of how the SPDX represents those here.
Annotations – Annotations are comments made by people on various entities and elements within the document. For example, someone reviewing the document may make an annotation about a file and its license. Annotations are useful for reviews of SPDX documents and for conveying specific information about the package, file, creation, license, file(s), etc.

In the SPDX specification release 2.2.2, additional output formats of JSON, YAML, and XML are supported. A diverse set of examples for SPDX are available on this github repo

Further information on the data model and SPDX guide can be found on the SPDX website.

Use Cases:

SBOM for software components
Tracking of intellectual property (licensing, copyright) of software components
Listing contents of a software distribution
Container contents inventory
Associating CPEs with specific packages
Identifying provenance of lines of code embedded in files

Key Features:

Documented artifacts can be checked using the provided hash values
Rich facilities for intellectual property and licensing information
Flexible model able to scale from snippets and files up to packages, containers, and even operating system distributions
Ability to add mappings to other package reference systems

2 - Software Identification (SWID) Tags

History:

It is a standard implemented by the National Institute of Standards and Technology (NIST) in the U.S. that was published in 2009, then revised in 2015. They were designed to provide a transparent way for organizations to track the software installed on their managed devices. Standard SWID tags are not generated at the end of certain software creation, instead, they define a lifecycle where a new SWID tag is added to an endpoint with the software installation process and is deleted with the uninstall process. When this lifecycle is followed, the presence of a given SWID Tag corresponds directly to the presence of the software product that the Tag describes.

Fig 2: https://d3i71xaburhd42.cloudfront.net/496312d64dc77b223803a4ee1b717be8e528e86f/16-Figure1-1.png

Note that the present SWID tags change depending on the current state of the software.

Specs:

The NISTIR 8060 Guideline identifies the standards of SWID tags and the components of each tag. We here go over the necessary types mentioned in the figure above. To capture the lifecycle of a software component, the SWID specification defines four types of SWID tags: primary, patch, corpus, and supplemental

Corpus Tag – A SWID Tag that identifies and describes an installable software product in its pre-installation state. A corpus tag can be used to represent metadata about an installation package or installer for a software product, a software update, or a patch.
Primary Tag – A SWID Tag that identifies and describes a software product installed on a computing device.
Supplemental Tag – A SWID Tag that allows additional information to be associated with any referenced SWID tag. This helps to ensure that SWID Primary and Patch Tags provided by a software provider are not modified by software management tools while allowing these tools to provide their software metadata.
Patch Information – A SWID Tag that identifies and describes an installed patch that has made incremental changes to a software product installed on a computing device.

Note that Corpus, primary, and patch tags have similar functions in that they describe the existence and/or presence of different types of software (e.g., software installers, software installations, software patches), and, potentially, different states of software products. In contrast, supplemental tags furnish additional information not contained in the corpus, primary, or patch tags.

SWID tags are mainly implemented in XML format while JSON format is under development. Some tag examples can be found here

Use Cases:

SBOM for software components
Continuous monitoring of installed software inventory
Identifying vulnerable software on endpoints
Ensuring that installed software is properly patched
Preventing installation of unauthorized or corrupted software
Preventing the execution of corrupted software
Managing software entitlements

Key Features:

Provides stable software identifiers created at build time
Standardizes software information that can be exchanged between software providers and consumers as part of the software installation process
Enables the correlation of information related to software including related patches or updates, configuration settings, security policies, and vulnerability and threat advisories.

3 - CycloneDX

History:

CycloneDX is a lightweight SBOM standard designed for use in application security context and supply chain component analysis as it was originally intended to identify vulnerabilities and supply chain component analysis. It also supports checking for licensing compliance. The CycloneDX project was initiated in 2017 in the OWASPcommunity, then it became a dedicated open source project and included other working groups from Sonatypeand ServiceNow. Supported file formats for CycloneDX are (XML, JSON, and protocol buffers)

Specs:

CycloneDX provides schemas for both XML and JSON, defining a format for describing simple and complex compositions of software components. It’s designed to be flexible, and easily adaptable, with implementations for popular build systems. The specification encourages the use of ecosystem-native naming conventions and supports SPDX license IDs and expressions, pedigree, and external references. It also natively supports the Package URL specification and correlating components to CPEs. The CycloneDX object model is defined in the figure.

Fig 3: https://cyclonedx.org/theme/assets/images/high-level-object-model-small.svg

BOM Metadata Information – BOM metadata includes the supplier, manufacturer, and the target component for which the BOM describes. It also includes the tools used to create the BOM, and license information for the BOM document itself.
Components Information – Components describe the complete inventory of first-party and third-party components. Component identity can be represented as: – Coordinates (group, name, version) – Package URL – Common Platform Enumerations (CPE) – SWID – Cryptographic hash functions (SHA-1, SHA-2, SHA-3, BLAKE2b, BLAKE3)
Services Information – Services describe external APIs that the software may call. Services describe endpoint URIs, authentication requirements, and trust boundary traversals. The flow of data between software and services can also be described including the data classifications and the flow direction of each type.
Dependency Relationships – CycloneDX provides the ability to describe components and their dependency on other components. The dependency graph is capable of representing both direct and transitive relationships. Components that depend on services can be represented in the dependency graph and services that depend on other services can be represented as well.
Compositions – Compositions describe constituent parts (including components, services, and dependency relationships) and their completeness. The aggregate of each composition can be described as complete, incomplete, incomplete first-party only, incomplete third-party only, or unknown.
Vulnerabilities – Known vulnerabilities inherited from the use of third-party and open source software and the exploitability of the vulnerabilities can be communicated with CycloneDX. Previously unknown vulnerabilities affecting both components and services may also be disclosed using CycloneDX, making it ideal for both VEX and security advisory use cases.
Extensions – Multiple extension points exist throughout the CycloneDX object model allowing fast prototyping of new capabilities and support for specialized and future use cases. The CycloneDX project maintains extensions that are beneficial to the larger community. The project encourages community participation and the development of extensions that target specialized or industry-specific use cases.

Use Cases:

Inventory of all software components.
Identifying known vulnerabilities.
Integrity verification using the hash functions.
Authenticity of the software components using a digital signature.
License compliance
Provenance

Generate SBOMs manually? definitely not

SBOMs are frequently updated with each release of the software, so we need tools and packages to be integrated with our ci/cd pipeline. We talk about this here

Blog: Software Bill Of Materials generation tools

Sun, 24 Jul 2022 00:00:00 +0000

Prerequisite

Before you read this, you have to understand what are SBOMs and what are different formats of SBOMs

Different SBOM generation tools comparison

If you got this far, you already realize the importance of SBOM generation, and also it should meet certain requirements to achieve its purpose. Due to various requirements depending on what standard you’re following, there has to be a way to automatically generate different output formats for different standards. Also, it has to be suited for ci/cd solutions to keep up with the increasing number of releases for each organization.

Note: Here we’re only considering open source tools

1 - Anchore Syft

Introduction:

Anchoreis a platform that implements sbom-powered supply chain security solutions for developers and enterprises. For generating SBOMs, a CLI tool and library named Syft was developed by Anchore that could be injected into your ci/cd pipeline to generate SBOMs from container images and filesystems at each step.

Integration and Support:

Syft is supported on Linux, Mac, and Windows and it can run as a docker container which makes it a great suit for CI systems. Other than the 3 SBOM standards, Syft can generate its JSON standard format to be input for other Anchore tools like Grype which is a vulnerability scanner for container images and filesystems. It supports projects based on the following package managers:

Alpine (apk)
C (conan)
C++ (conan)
Dart (pubs)
Debian (dpkg)
Dotnet (deps.json)
Objective-C (cocoapods)
Go (go.mod, Go binaries)
Haskell (cabal, stack)
Java (jar, ear, war, par, sar)
JavaScript (npm, yarn)
Jenkins Plugins (jpi, hpi)
PHP (composer)
Python (wheel, egg, poetry, requirements.txt)
Red Hat (rpm)
Ruby (gem)
Rust (cargo.lock)
Swift (cocoapods)

Features and Specs:

Easy to use
- Syft can generate a simple basic sbom by just running syft <image> this will only include the softwares included in the image’s final layer. Or syft <image> --scope all-layers for more verbose sbom to include all image layers
Different formats support the ability to convert between them.
- Syft JSON
- SPDX 2.2 JSON
- SPDX 2.2 tag-value
- CycloneDX 1.4 JSON
- CycloneDX 1.4 XML
Cryptographically sign and attest SBOMs
- Syft uses in-toto attestations with syft attest command and the digital signature management is integrated with sigstore cosign. You can view more here.
Support a variety of sources to generate SBOMs from
- OCI and docker image formats syft <image>
- Container images archives syft path/to/image.tar
- Filesystems and local directories syft path/to/dir

For more resources about Syft capabilities refer to the source repo and official documentation

2- Opensbom’s Spdx-Sbom-Generator

Introduction:

Opensbom-Generator is an open source project initiated by the Linux Foundation SPDX workgroup to generate SBOMs using CLI tools. Currently, they support the standard spdx 2.2 formats and JSON with their spdx-sbom-generator tool based on golang. It can only be used to generate SBOMs from a repository containing package files (no container images or archives support yet). They aim to provide SBOM generation support in ci/cd solutions.

Integration and Support:

You can download the binaries and install the tool on your system. The available binaries to install are for Linux, Windows, and macOS and it can also be used as a docker container from this spdx repo. It can detect which package managers or build systems are being used by the software. It is supporting the following package managers:

GoMod (go), Cargo (Rust), Composer (PHP), DotNet (.NET), Maven (Java), NPM (Node.js), Yarn (Node.js), PIP (Python), Pipenv (Python), Gems (Ruby), Swift Package Manager (Swift)

Features and Specs:

CLI easy to use and simple interface
Automatic detection of the package manager

3- Kubernetes BOM

Introduction:

BOM is a general-purpose CLI tool developed by kubernetes-sigs (Special Interest Groups) that can generate SBOMs from directories, container images, single files, and other sources. The utility has a built-in license classifier that can check for license compliance of your packages with around 400+ licenses in the SPDX catalog.

Integration and Support:

BOM is supported as a Golang package that can be installed on any system having to go with go install sigs.k8s.io/bom/cmd/bom this adds the support for Linux, Mac, and Windows. It is compatible with creating SBOMs from files, images, and docker archives (images in tarballs). It also supports pulling images from remote container registries for analysis. BOM is mainly generating SBOMs in SPDX formats.

Features and Specs:

CLI usage to support CI/CD solutions
Golang dependency analysis
Full .gitignore support when scanning git repositories
Ability to check for license compliance with SPDX catalog
Support for different sources to generate sboms
Other than the command bom generate, it uses bom document to work with already present SPDX documents to outline and draw a structure for them
It doesn’t necessarily require a whole project directory but it can specify a single file to analyze with the -f path/to/file flag and it can have a collection of those files to be analyzed together.
It also supports the namespacing separation between SBOM documents using the -n <URI> flag to isolate each document from the other

4- Microsoft SBOM tool

Introduction:

Recently, Microsoft open-sourced their SBOM generation tool which is described as a general purpose, enterprise-proven, build-time SBOM generator. They have been developing the tool internally since 2019 and tuning its feature according to their needs and providing other companies with the solution. What is new is that Microsoft has chosen to merge efforts with the Linux Foundation’s work and use Software Package Data Exchange (SPDX) for all SBOMs generated, and to do this for all software produced. It has a promising number of features like including other SBOM documents recursively to provide its users with the ability to have a full dependency tree that goes to the origin of every package.

Integration and Support:

Microsoft SBOM is supported on Linux, macOS, and Windows. It can be easily integrated into and auto-detects the following package managers

NPM, NuGet, PyPI, CocoaPods, Maven, Golang, Rust Crates, RubyGems, Linux packages within containers, Gradle, Ivy, GitHub public repositories

and Microsoft is adding more detectors to improve deeper integration with the community. The tool is currently committed to generating the SPDX 2.2.1 format for its users and is still in development to include all optional fields before integrating with other formats.

Features and Specs:

Enterprise ready and highly scalable as already used at scale by Microsoft
Adding build provenance information to the SBOM
Auto-detection of the underlying package manager
Supports namespacing separation between SBOM documents with the -nsb flag
Validating SBOMs at release using hashes and digital signatures

Fig 1: https://devblogs.microsoft.com/engineering-at-microsoft/wp-content/uploads/sites/72/2021/10/adiglio-figure-1.png

For more information about the tool visit the GitHub repoand refer to the documentation here

5- Tern

Introduction:

Tern is a VMware-originated open source inspection tool used to generate SBOMs following standard formats. It gathers metadata for the packages installed in container images and Dockerfiles. Tern starts to analyze the contents of a container (through the image itself or the Dockerfile), layer by layer, without requiring the user to have in-depth technical knowledge about how the container was built.

Integration and Support:

Tern itself is available as a Github Action but it is mainly supported to be installed as a CLI tool on Linux. With Linux installation, Tern is built with python so it requires python, pip, and jq installed mainly (more about installation here). Some features like analyzing Dockerfiles and the lock function require Docker installation in your Linux. Moreover, Tern can run as a docker container which makes it easy to use in ci systems and can be used as a workaround to run on other operating systems like Windows, and macOS. Also, this helps to deploy Tern as a Kubernetes job with a host mount to retrieve generated SBOMs.

For now, Tern only supports container images built using Docker using image manifest version 2, schema 2 and it will support Docker images and it is aimed to support other images that follow the OCI standards in the future.

For license compliance, Term doesn’t have its file-level license analyzer. So, it allows you to extend its analysis using an external CLI tool or Python packages as extensions. An example is Scancode which is a CLI tool used for license compliance along with other supported integrations like cve-bin-tool for vulnerability scanning.

Tern supports generating reports with multiple formats:

Human Readable Simple format
JSON format
HTML format
YAML format
SPDX tag-value Format
SPDX JSON Format
CycloneDX JSON Format

Features and Specs:

CLI is easy to use and can be installed as a docker container which makes it suitable for CI/CD solutions
Multiple supported output formats which are readable for both humans and machines
Ability to generate SBOMs following both SPDX and CycloneDX formats
Support for Dockerfile locking to create more reproducible Docker images which is a unique feature supported by Tern only, view more about it here.
The concept of extensions/plugins is only supported by Tern between all the previously mentioned tools. This is a great feature that is proven to extend its capabilities in the coming future and open for creativity from the open source community (this was proven before to increase the functionality and popularity of other tools like Jenkins).

For more information about the tool visit the GitHub repo and refer to the documentation here.

Blog: GSoC 2022 Community Bonding Period with Jenkins X

Tue, 12 Jul 2022 00:00:00 +0000

Introduction

Hello everyone, I am Rajat Gupta, pursuing my bachelor’s in Information Technology. In 2022, I have been selected as a student developer in Google Summer of Code under Jenkins X. We will be building a new UI for Jenkins X. I got this news on May 20th, as I received an email from google.

How I started

The technologies needed were Golang, Kubernetes, and GitOps. I used golang only once before, while linting Jenkins X codebase, I only used Kubernetes once before while setting up a k3s cluster to run Jenkins X pipelines. These tasks were necessary to do for all GSoC participants. Apart from that, I was a total beginner.

What I learned during Community Bonding Period

So, when I got selected, I had a lot to learn, my mentors gave me a 30 Day plan. They also suggested some resources and conference talks which made it simple for me to start.

30 Day plan was:

Learn golang
Getting started with Kubernetes
Learn what is a CRD and how they are used?
Complete a tutorial on Kubebuilder
Getting started with Tekton pipelines
Learn Jenkins X

We also had some amazing pair programming sessions, where I used to share my screen and my mentor guided me through, which is not common because mentors have a very busy schedule. But my mentors helped me a lot. These sessions helped me in working with the massive Jenkins X codebase and learning its complex workflows.

I also learned that testing and documentation are very important as they make it easier for developers to understand these workflows. Hence testing also became an important part of my learning period.

We also started the UI-sig a week or two after my selection where we work on Jenkins X UI. The learning I did in this community bonding period was just amazing, I learned more in 3 weeks than that what I learned in the past 3 months.

Highlights

I also attended the kubeCon, cdCon, and Jenkins X Contributor Summit 2022 during this period. When I meet other contributors of Jenkins X, people from CD Foundation were very excited about our plans for the future. I also gave a small introduction to my project.

If you are looking for a project where you can learn about CI/CD, DevOps, Cloud, Golang, Jenkins X is the best place for you.

Thanks to all my mentors @ankitm123, @babadofar, @msvticket, and @tomhobson

Blog: Kubernetes 1.22 - Breaking change!

Fri, 22 Apr 2022 00:00:00 +0000

To allow Jenkins X to support Kubernetes 1.22, we had to update our version of Tekton. This updated version of Tekton contains breaking changes that has consequences if you made your own custom Jenkins X pipelines.

To make sure that your custom pipelines continue to work after this upgrade, you must edit the resource settings in your pipelines. Otherwise your pipelines will most likely not be able to start at all, or if they do, consume a lot of resources.

Changes in Tekton version 28

Tekton made changes in how to calculate the resources needed to run a pipeline, in order to support the concept of LimitRange in Kubernetes (introduced in Kubernetes version 1.10). Previously, Tekton simply used the maximum requested cpu and memory of any single step, and set that as limits for the all steps in the pipeline.

For more details, please read the Tekton documentation on LimitRange.

How to prepare for upgrade

In the Tekton pipeline files, the StepTemplate needs to be changed to not specify resource requests, but only setting an empty resource limits:

stepTemplate:
  image: uses:jenkins-x/jx3-pipeline-catalog/tasks/go/release.yaml@a5ab19ebc5a074e0402c5016b11bc11b32cc5c83
  name: ""
  resources:
    # override limits for all containers here
    limits: {}

The resource requests should be set on only one individual step:

steps:
  - image: uses:Mentor-Medier/jx3-pipeline-catalog/tasks/git-clone/git-clone-pr.yaml@versionStream
    name: ""
    resources: {}
  - name: jx-variables
    resources:
      requests:
        cpu: 400m
        memory: 512Mi

To see examples of what changes you need to apply to your custom pipelines you may investigate this PR on The Jenkins X pipeline catalog. The PR will be merged and released simultaneous with the upgrade of Tekton.

In the current version of Tekton used by Jenkins X, the resource requests are set on the stepTemplate. Running the old pipelines in the new version will lead to resource requests being multiplied by the number of steps.

To prepare for upgrade, you should create a PR applying the changes described above. We done some tests on applying the changes to the current version of Jenkins X/Tekton, and it seemed to work. But - that does not mean it will work on your machines or even pipelines!

So, please prepare for upgrade now! We want people to have a safe upgrade experience. If you have questions, you can find us on Slack.

Blog: Google Summer of Code 2022 project proposal template ☀️

Tue, 05 Apr 2022 00:00:00 +0000

Proposal template

1. Contact details

Full name:
Country:
Time zone:
Email:
Github ID:
Personal blog (optional):
Twitter/LinkedIn/others:

2. GSOC information

Have you participated in the Google Summer of Code previously? Please describe your experience.
Are you applying to any other organizations this year? If so, please list them.
How many hours will you devote to your GSoC project each week? Do you have any other commitments during the summer?
Have you ever contributed code to Jenkins X? If you have, post the Pull Request (PR) links (It’s ok if they have not been merged/approved yet)
Do you plan on contributing to the Jenkins X project after GSoC is finished?
Were you able to install Jenkins X locally on your laptop?
- If no, please explain why (Unsupported architecture, missing documentation, time constraints etc)?
- If yes, how was the installation experience (missing documentation, complexity etc)?
Have you used kubernetes in the past? Please provide a brief description and if possible links to the work.
Have you used golang in the past? Please provide a brief description and if possible some code samples.
(If you are interested in the UI project) Have you worked with a frontend framework/library in the past? Please provide a brief description and if possible some code samples.

3. Project idea

Repeat this section for as many project ideas as you want (Go in the order most likely to least likely).

Title of the idea that you are interested in
Brief description of the idea.
Have you worked on a similar project in the past? If yes, please provide a brief description (with links if possible)
How will the project benefit Jenkins X and the community
- Will it help in driving adoption of Jenkins X? If yes, then how/why?
Deliverables
- Include any milestones and deadlines
- Include time for investigation, coding and documentation

Blog: Project proposal for Google Season of Docs 2022 📄

Wed, 23 Mar 2022 00:00:00 +0000

We have put together a project proposal as part of our application to participate in the Google Season of Docs 2022 program.

Project proposal

Better documentation for real world use cases - Jenkins X

About us

Jenkins X provides automated CI/CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton.

Problem

End users of Jenkins X are unable to find information on how to use Jenkins X for real world use cases which includes:
- Production best practices
- Scanning images in Jenkins X pipelines
- How to make your app use different configuration/secrets for each environment
- Observability for your apps, logging, metrics, tracing
- Tests (Integration, e2e)
- Artifact management
- Multi cluster
- Integration with other tools in the cloud native sector

Project scope

Audit the current Jenkins X documentation and create a friction log
Use the friction log as a guide for understanding the gaps in the documentation
Interact with the Jenkins X community to create a list of top use cases
- Create a survey
- Look at the github issues and slack messages
Work with the volunteers to incorporate those changes into the documentation
Establish a feedback loop (the target customers are the end users of Jenkins X) to improve the documentation

Measuring project success

Decrease in the number of github issues raised by end users on documentation related to common real world use cases
Short 1-2 minute videos on common tasks that end users of Jenkins X would like to perform.

Recommended skills

Must have:
- Experience with github
- Communication skills to work with the Jenkins X community
- Creating online surveys
Nice to have:
- Basic experience with docker compose (local development scripts use compose)
- Good video editing skills.
- Some knowledge working with the documentation of other tools in the kubernetes ecosystem

Mentors/Volunteers

Timeline

The project will take roughly 6 months to complete. Once the technical writer is selected, we will do an orientation to bring him up to speed. This is the timeline we have in mind, but we are flexible.

Dates	Action Item
April 14 - May 16, 2022	Hire technical writer
April 14 - May 31, 2022	Orientation after hiring technical writer
June 1 - July 31	Review documentation, create actions items, get familiar with Jenkins X
August 1 - October 31	Work on the project
November	Project completion

Budget

Budget item	Amount	Running Total	Notes
Technical writer stipend	6000	6000
Volunteer stipend	2000	8000	4 volunteer stipend (500 each) X 4
Jenkins X t-shirts	300	8300	20 t-shirts X 15
Shipping for t-shirts	300	8600	15 dollars to ship, shipping rates can vary

Notes about budget

The cost of the t-shirt was taken from here.
The t-shirts are for the technical writer, volunteers and any member of the jenkins X community (not officially listed as a volunteer) who helps out by answering the technical writer’s questions.

Additional information

Previous participation in Season of Docs, Google Summer of Code or others
- Jenkins X participated in the 2020 Season of docs. The maturity level matrix created by the technical writer is still used by the community to evaluate if Jenkins X will work for their use case, as well as used by the Jenkins X maintainers to plan the roadmap for improving Jenkins X.
- Jenkins X is also participating in the google summer of code 2022. We are seeing a lot of interest from new contributors. We are confident it will lead to some amazing improvements in the Jenkins X ecosystem and make it a sustainable open source project. We expect the technical writer will help the contributors create better documentation for their work.

Other ideas

2. Reorganize documentation to make it more user friendly and easy to search

Problem

The documentation does not feel coherent, it does not read like a book from start to finish
- There’s no clear flow about how people should read documentation and no “next step” once they’ve finished reading a page.
- There’s also a lot of missing assumed knowledge in the documentation which makes it hard for new users to form a mental picture of Jenkins X
Results from older documentation sometimes shows up higher than newer results creating confusion for users

How would we measure success?

The number of repeated searches decrease (adding / removing terms)
Decrease in the number of github issues raised by users on documentation related to missing information
Decrease in slack messages from users not finding relevant information in the documentation.

3. Improve contributor/developer documentation on how to extend Jenkins X

Problem

It is not clear from our documentation how to
- add support for a new cloud provider
- add support for a new git/scm provider
- add a new quick start
- add new language packs

How would we measure success?

Increase in number of contributions to add support for more cloud providers
Increase in number of contributions to add add or extend quickstart guides
Decrease in slack messages from users not finding relevant information in the documentation.

Blog: Google Summer of Code 2022 ☀️

Sat, 12 Mar 2022 00:00:00 +0000

Project proposal template can be found here.

We are very happy to announce that Jenkins X has been selected to participate in the Google Summer of Code (GSoC) 2022!

If you are new to GSoC and want to learn more about it, check out their new site. You can find the list of project ideas from Jenkins X here. If you are interested in applying to Jenkins X, please check the timeline. Apply as early as possible.

This post aims to give GSoC contributors insight into the selection process and how to get started.

How many contributors are we looking for?

We have 3 mentors signed up at the moment and are looking to select a maximum of 2 applicants.

We want you to have a great experience contributing to Jenkins X and learn lots of new things!

What are we looking for in an ideal Jenkins X contributor?

Willingness to learn Jenkins X and engage with the community
Independence and a drive to investigate and propose solutions to complex issues that might occur during implementation
(Optional) Eventually become a Jenkins X maintainer (and some day mentor new GSoC contributors)!

What can prospective contributors do? (March 7 - April 19)

You do not have to complete all of these tasks. This is presented here so that you have some clarity around where to start and which resources to follow.

Recommended: Join the jenkins X community slack channels and say hello :)
- We prefer you post messages in the jenkins-x-dev channel
- Attend the office hours
Recommended: Interact with potential mentors to understand/review the requirements of the projects.
Recommended: Start working on your proposal early on.
Optional: Learn about docker
Optional: Learn about kubernetes
Optional: Learn the basics of golang
Optional: Understand Jenkins X concepts and fundamentals by going over the documentation
Optional: Set up Jenkins X locally and give it a go (Please don’t spend money running Jenkins X in the cloud).
- We prefer you use k3s for this, as it’s less resource intensive and can be installed locally on your workstation (linux). Mac users need to install something like multipass to first install k3s.

Interview process and selection criteria (April 19 and May 12)

We will start interviewing candidates between April 19 and May 12. We want to talk to you individually, and get to know you better.

Contributors will be scored based on the following criteria:

Important:

Independent thinker and willingness to learn
Project proposal demonstrating good understanding of the project they are interested to work on

Nice to have:

Technical expertise (basic knowledge goes a long way):
- Past experience with container technology
- Past experience with kubernetes
- Past experience with golang
- Past experience with a Frontend UI library (if interested to contribute to the UI project)
Past open source contribution experience
Any level of familiarity with Jenkins X (or other CI/CD tools)
Complete one of the three tasks listed below (If you have any questions on these tasks, please ask in our slack channel and we’ll be happy to answer your questions):
- If you have already contributed to Jenkins X, then share the GitHub links to your pull requests. Your mentor will be happy to discuss them with you and will want to know more about what you did and why you did it.
- Walk us through a local Jenkins X set up (Please don’t spend money running Jenkins X in the cloud)
- Refactor a single function within the Jenkins X organization to improve the code quality. Clone the library, make refactoring changes, describe your changes and zip up and submit to a mentor via slack. Explain your refactoring changes and why you did them.
Previous GSoC experience (first time contributors are more than welcome)

We will make an effort to talk individually to all the Jenkins X GSoC contributor applicants.

We look forward to reviewing your application and working with you.

Blog: Project ideas for Google Summer of Code 2022 ☀️

Sun, 20 Feb 2022 00:00:00 +0000

Jenkins X has been accepted into the Google Summer of Code 2022 program, take a look at the follow up blog.

Project proposal template can be found here.

We have put together some project ideas as part of our application to participate in the Google Summer of Code 2022 program.

1. Cloud events integration with Jenkins X

Description

The only way to trigger jobs/workflows in Jenkins X at the moment is by listening to events from Source Control Management (SCM) providers like github, gitlab, bitbucket, however it would be nice to listen to other event sources and trigger jobs/pipelines in Jenkins X. One interesting application would be to trigger some Jenkins X job in response to some alerting event (pagerduty, opsgenie). As a start we should focus on (emitting and listening to) cloudevents which define a common format for events produced from different sources. This will also help make Jenkins X compatible with other platforms.

Expected Outcomes

Jenkins X should be able to emit cloud events
Jenkins X should be able to listen to cloud events, and run pipelines
Updated documentation

Recommended skills

Golang, kubernetes, cloudevents, familiarity with lighthouse would be great, but not required

Mentors

Resources

Expected Size of project

350 hours

Difficulty rating

Hard

2. Supply chain security: Improve integration with sigstore and look at tekton chains

Description

With all the software breach that has happened recently, it has become necessary to add tooling to solve the issue around supply chain security. There are some good open source tools which can help with that (sigstore tools). As a CI/CD platform, Jenkins X needs to be integrated with them so that the end users can get this feature out of the box. Jenkins X leverages tekton as it’s pipeline execution engine. However, we dont integrate with tekton chain yet. Also similar to tekton chains, we should have a jenkins X operator that can take a snapshot of the jx pipeline activities or lighthouse jobs, sign them and store them in a cloud store eventually. Catalog should be updated to include trivy tasks, so that users can start using them with no to minimal effort. As a start, we should start signing the various artifacts produced by Jenkins X (binaries, docker images, helm charts).

Expected Outcomes

Jenkins X artifacts (helm charts, docker images, binaries) are all cryptographically signed.
Integration with tekton chains
An operator which can take snapshot of the jx pipeline activities, sign them and store them in the cloud storage.

Recommended skills

golang, kubernetes, (basic) understanding of security

Mentors

Resources

Expected Size of project

350 hours

Difficulty rating

Medium

3. New Jenkins X UI

Description

Currently, the way to manage (CRUD) Jenkins X resources(pipelines) is by using the cli. While the CLI is very powerful and user friendly, it should be used by power users (release team who installs and manages jenkins x clusters). We should not expect developers who are concerned with only the pipelines to install the cli (This does not scale when you have 100+ developers in the company) We do have a UI/dashboard, but it is read only, so users cannot use it to trigger release pipelines or stop running pipelines. In addition, we do not have an audit trail of who did what. So, we propose a new UI/dashboard which has feature parity with the CLI (including SSO and RBAC) The UI should make the JX cli redundant, and add more value to the user with easily available status and logs of the entire cluster

Expected Outcomes

Fully functional Jenkins X Dashboard running in the kubernetes cluster
Drop in replacement for existing read only UI
Audit trail of who took what action in the UI
Updated documentation

Recommended skills

golang, basic understanding of sveltejs (or any js framework), css, kubernetes

Mentors

Resources

Expected Size of project

350 hours

Difficulty rating

Hard

4. Quickstart Improvements

Description

Create new quickstarts that showcase interesting features and kubernetes deployment best practices. This includes prometheus integrations, database integrations for preview demos, documentation and a better interface for creating quickstarts, rootless containers. The current quickstarts were created three years ago and are not up to date. It would be good to have them updated, and also add quick starts for newer languages and frameworks.

Expected Outcomes

Updated quickstarts
Quickstarts for new languages and frameworks
Updated documentation

Recommended skills

Some knowledge on various programming languages and some deeper knowledge of Kubernetes deployment and operations.

Mentors

Resources

Expected Size of project

175 hours

Difficulty rating

Easy

5. Implement drift detection (gitops)

Description

Jenkins X only applies changes to cluster when contents of the gitops repository changes. This does not satisfy one of the requirements of the gitops model. It would be nice to detect drift between the current state (kubernetes) and the desired state (git) and apply only those changes. This has the side effect of making the boot job faster.

Expected Outcomes

Drift detection in Jenkins X
Configurable interval when Jenkins X will do a drift detection and apply the changes
Updated documentation

Recommended skills

Kubernetes, golang, Jenkins X, basic understanding of gitops.

Mentors

Resources

https://opengitops.dev/

Expected Size of project

175 hours

Difficulty rating

Medium

6. Multi-tenancy in Jenkins X

Description

Expected outcomes

A multi-tenant enabled Jenkins X install (jenkins X in different namespaces or a central jx install controlling pipelines running for different tenants)
Better scaling and security model for Jenkins X
Updated documentation

Recommended skills

Golang, kubernetes, event driven architecture

Mentors

Expected Size of project

350 hours

Difficulty rating

Hard

Next Steps

If Jenkins X gets selected, we will create a followup blog with additional details.

Blog: Jenkins X Survey Result Details

Tue, 15 Feb 2022 00:00:00 +0000

For more information on the Jenkins X survey see Survey results Some highlights from the free text answers.

What do you enjoy the most with Jenkins X?

KUBERNETES

Cloud-native feel, “you know Kubernetes, you know how to use JX”, plugins-based architecture
Complete CI/CD platform in a k8s cluster

GIT/CHAT/OPS/CONFIGURABILITY

Configuration as code, auto generation of k8s files, flexibility
easy to create preview/dev/staging/prod environments.
modularity and ease to extend/change

EASY

Everything works out of the box.
Very intuitive
Lightweight and easy to manage.

COMMUNITY

The community is incredible. I really like how everything works together.
love the community.

VARIOUS

scalability and how fast it is!
Integration with different secret backends is easy. Also love the community.
It’s opinionated
No vendor lock in

What do you believe should be improved with Jenkins X?

ON PREMISE

run it against a local running cluster to test changes to Jenkins-X before updated in GitHub.
Integration with on-premise (Gitlab) as a lot of organizations are not using public cloud due to security policies,
Proper guide on installing into existing cluster without using terraform. Give us back something like jx compliance, jx boot.
Would be nice to not be tied to terraform to boot jenkins-x. Just like kubernetes, it would be great to have a jenkins-x the hard way where everything needs to be installed manually.

DOCS

it’s very difficult to just dive in without previous knowledge of the system
extreme lack of quality documentation.
Many conversations take place in slack and users cannot find information that somebody had the same problem beforehand and how it was solved
More documentation and guides, making sure quickstart guides work without hassle
Making it easier to get started
Very hard to find any information on how people solve a similar problem before
I very much enjoy your documentation, but it is very hard to find anything on google that is a huge disadvantage.
Documents need to be clear about what works, and what does not work (kubernetes versions for example, bitbucket etc ..)
Could also be nice with an arcitecture illustration or video that could compare jenkins with jenkins X
I find that when I’m looking for information I get a lot of mixed results (v3 vs v2).

DEBUGGING

Debugging the integration
Error handing and reporting

MULTI

HA, multi-tenant
multi cluster setup
Remote clusters

GOVERNANCE

Needs focus on following up on K8s versions.
Needs a larger body of people governing it
PRs are being ignored for months at a time. The first PRs I’ve opened have been ignored to this day. I started getting some response after connecting with some devs in Slack.
A public Roadmap and prioritizing tasks by community demand would be the way to go.
who will be providing development leadership and direction.

SECURITY

permission management within Terraform repos (default settings are too wide for orgs)
separate JX permissions from per-project permissions - current pipelines gives a cluster admin scope to everyone as development teams can override steps and execute with any service account

UI

pipelines dashboard needs Re-Run button.
A fully functional UI would be nice.

VARIOUS

The install process
deploy speed
The parallel PR building causes issues.
Support for Bitbucket Server
providing the capabilities to match CircleCI. lots of missing features or very difficult to set up
Performance

VARIOUS FEEDBACK

Integration with vault is confused. Why install vault in docker in k3s environment?
I strongly believe in this project!
I use Jenkins X as a learning tool. It gives me the ability to build K8S clusters using different cloud providers and helps me to understand the mechanics of GitOps, helm charts, etc.
Better error handling when pipelines fail would be good (tekton pipeline fail, but jx pipelines dont update their status).
I’m excited to see how the product has grown in the short time I’ve used it.

Blog: Jenkins X Survey Results

Tue, 15 Feb 2022 00:00:00 +0000

The Jenkins X survey was active for four weeks and closed on February 11 2022. We received lots of valuable insights into how people are using Jenkins X. We need more contributors in the Jenkins X community, so if you feel strongly about how Jenkins X should evolve, your best bet is to dive in and get your hands dirty:) Some highlights of the free text responses we got are collected here

According to the survey, a typical Jenkinx X user works with Devops and Software Engineering

This person is using Jenkins X version 3

on Amazon

working in a company of 1-50 people.

She finds it somewhat difficult to find documentation, average 2.9 out of 5

The Jenkins X user tries to find information mainly on the main web site, sometimes on slack and less often on github. What he enjoys most about Jenkins X is that it is an easy way to learn, play around and work with Kubernetes. The git(ops(ish)) style of configuration, the preview environments, and the staging /production environments.

But one thing is sure, documentation is confusing and shold be improved.

The typical Jenkins X user would like to run Jenkins X offline, either to run on a laptop, or behind a (corporate) firewall. Proper support for multi-tenancy would be nice, and who can ignore security in these times. A clear governance of Jenkins X needs to be established,

The typical person who answers surveys on Jenkins X is planning to attend Jenkins X Office hours, which is great!

See you there next week then!

Blog: January 2022 updates from the JX community

Wed, 02 Feb 2022 00:00:00 +0000

Happy new year 2022!

This monthly blog post series is an attempt to showcase all the incredible work being done by the Jenkins X community to the wider audience.

Lot of exciting features, bug fixes and documentation improvements were made.

Community effort

We restarted the office hours this month (https://jayex.io/community/#office-hours). Drop by to say hello, we are a friendly group!
First Jenkins X survey was also created this month (https://jayex.io/blog/2022/01/21/survey-1-2022/). We have extended the deadline by 2 weeks (Feb 11, 2022 midnight UTC), so fill it out if you have not yet. We will use this for the roadmap moving forward.
Monthly blog post update series to keep up with all the amazing progress.

Features

jx-plugins/jx-gitops:
- Cron job to delete old boot jobs
- Option to Keep n boot jobs older than default age
jenkins-x/lighthouse:
- Support for using gitlab nested repositories
- Allow running lighthouse with cluster scoped permissions
- Add a flag to keep polling releases until commit status is successful
- Do not include tekton roles when tekton engine is disabled
jx-plugins/jx-pipeline:
- Use pager to aid in visualizing long pipeline logs
jenkins-x-terraform/terraform-jx-azure
- Support provisioning spot instances in the azure terraform jx module
jx-plugins/jx-verify:
- Support specifying label for jx verify install
jenkins-x-plugins/jx-registry:
- Support adding ECR registry policy using jx-registry
jenkins-x-charts/jxboot-helmfile-resources:
- Make all storage locations available as envrironment variables

Bug fixes

GoogleContainerTools/kaniko (upstream fix - outside jx codebase):
- Kaniko in jx pipelines can now push to ACR (Azure Container Registry)
jenkins-x/terraform-aws-eks-jx:
- Remove deprecated jx v2 keys from requirements configmap
- Fix issue with on-demand billing mode of dynamodb
jenkins-x-plugins/jx-promote:
- Fix local chart check when using cloud buckets to store helm charts
jenkins-x-plugins/jx-project:
- Fix jx project rendering invalid chart.yaml files on import for custom packs in catalog

Documentation improvement

Guides on configuring Azure service principle and GCP service account for terraform users (https://jayex.io/v3/admin/platforms/azure/svc_principal/ and https://jayex.io/v3/admin/platforms/google/svc_acct/)
New section showing current Jenkins X users (https://jayex.io/#users)
New section on learning resources for Jenkins X v3 (https://jayex.io/v3/learning-resources/)
Improved documentation on setting up Jenkins X v3 on AWS EKS (https://jayex.io/v3/admin/platforms/eks/)
Improved guide on installing and configuring Jenkins X on k3s (https://jayex.io/v3/admin/platforms/k3s/)
Fixed content getting clipped in mobile view

Plumbing/Quality improvements

jenkins-x/jx:
- Disable running github actions on forks
jenkins-x/jx-docs:
- Upgrade docsy submodules
- Upgrade hugo docker image
- Migrate jx-docs to new pipeline format
jenkins-x/jx-gitops:
- upgrade kubectl version for kpt
- upgrade helm and helmfile version

Huge thanks to all the contributors for their hardwork!

ankitm123
babadofar
dippynark
ia-mfriegang
jalonsoa
msvticket
rajatgupta24
rawlingsj
sergiogiuffrida
slimm609
TedGelpi
tomhobson
yelhouti

Blog: Jenkins X Survey

Fri, 21 Jan 2022 00:00:00 +0000

We have made a short survey where we try to gain insight into how people experience Jenkins X. This is meant to be used as guidelines going forward so we can be more focused on what areas to improve.

All contributions are welcome, if you are just browsing or have used it for years, we want to know you all!

We have extended the survey period and it will now be open untill Tuesday February 11 2022 midnight UTC. All submissions are anonymous and the results will be published. So don’t hesitate, fill in the Jenkins X survey today

Blog: Incident: Kaniko and ACR

Tue, 28 Dec 2021 00:00:00 +0000

We’ve recently had an issue with one of our packages come to light. We wanted to talk through the resolution steps we’re going to put into place.

So what happened?

Azure users started reporting seeing the following error within the build step:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "xyz.azurecr.io/myorg/myrepo:0.0.1": resolving authorization for xyz.azurecr.io failed: error getting credentials - err: exec: "docker-credential-acr-env": executable file not found in $PATH

This seemed to be indicating to an authorization issues with the terraform module. Other users were seemingly unaffected by this issue.

Upon further analysis, kaniko seems to have issues with the latest version and grabbing credentials for acr. The latest working version that we are aware of is 1.3. There wasn’t a massive influx of people seeing this issue due to it only occuring once versionstream had been updated with jx gitops upgrade.

So what are you going to do about it?

For starters, most users are probably using the latest kaniko features, so we’re unable to just roll this back for everyone.

We’re getting started by creating a PR to kaniko to resolve the issue. However, due to release schedules etc, this will mean that getting started with azure will be broken for quite a while.

How can I fix it in the meantime?

If you’re on azure, the resolution is quite simple, here’s a step by step guide:

1. Find which buildpack you are using and navigate to the build-container-build step:

directory: csharp/ date: 12/28/21  git: main 
› cat .lighthouse/jenkins-x/release.yaml | grep "  image: "
          image: uses:jenkins-x/jx3-pipeline-catalog/tasks/csharp/release.yaml@versionStream

2. Replace the build-container-build step to use the suggested kaniko image: gcr.io/kaniko-project/executor:v1.3.0-debug

Before

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  creationTimestamp: null
  name: release
spec:
  pipelineSpec:
    tasks:
    - name: from-build-pack
      resources: {}
      taskSpec:
        metadata: {}
        stepTemplate:
          image: uses:jenkins-x/jx3-pipeline-catalog/tasks/csharp/release.yaml@versionStream
          name: ""
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
          workingDir: /workspace/source
        steps:
        - image: uses:jenkins-x/jx3-pipeline-catalog/tasks/git-clone/git-clone.yaml@versionStream
          name: ""
          resources: {}
        - name: next-version
          resources: {}
        - name: jx-variables
          resources: {}
        - name: check-registry
          resources: {}
        - name: build-container-build
          resources: {}
        - name: promote-changelog
          resources: {}
        - name: promote-helm-release
          resources: {}
        - name: promote-jx-promote
          resources: {}
  podTemplate: {}
  serviceAccountName: tekton-bot
  timeout: 12h0m0s
status: {}

After

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  creationTimestamp: null
  name: release
spec:
  pipelineSpec:
    tasks:
    - name: from-build-pack
      resources: {}
      taskSpec:
        metadata: {}
        stepTemplate:
          image: uses:jenkins-x/jx3-pipeline-catalog/tasks/csharp/release.yaml@versionStream
          name: ""
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
          workingDir: /workspace/source
        steps:
        - image: uses:jenkins-x/jx3-pipeline-catalog/tasks/git-clone/git-clone.yaml@versionStream
          name: ""
          resources: {}
        - name: next-version
          resources: {}
        - name: jx-variables
          resources: {}
        - name: check-registry
          resources: {}
        - image: gcr.io/kaniko-project/executor:v1.3.0-debug
          name: build-container-build
          resources: {}
          script: |
            #!/busybox/sh
            source .jx/variables.sh
            cp /tekton/creds-secrets/tekton-container-registry-auth/.dockerconfigjson /kaniko/.docker/config.json
            /kaniko/executor $KANIKO_FLAGS --context=/workspace/source --dockerfile=${DOCKERFILE_PATH:-Dockerfile} --destination=$PUSH_CONTAINER_REGISTRY/$DOCKER_REGISTRY_ORG/$APP_NAME:$VERSION
        - name: promote-changelog
          resources: {}
        - name: promote-helm-release
          resources: {}
        - name: promote-jx-promote
          resources: {}
  podTemplate: {}
  serviceAccountName: tekton-bot
  timeout: 12h0m0s
status: {}

3. Do this again for your .lighthouse/jenkins-x/pullrequest.yaml file

4. Commit and push it to your repository.

We’re hoping to make this simpler

We’re working on:

Fixing it upstream (the proper fix, but could take a while to be merged)
Making a temporary azure pipeline step for azure users
Making a script so that azure users don’t have to go through this manual process each time
Allowing overriding images without having to specify a script within lighthouse

Help us find and fix things like this in future

Talk to us on our slack channels, which are part of the Kubernetes slack. Join Kubernetes slack here and find us on our channels:

#jenkins-x-dev for developers of Jenkins X
#jenkins-x-user for users of Jenkins X

Find out more about becoming involved in the Jenkins X community here.

Blog: Hacktoberfest conclusion 2021

Mon, 22 Nov 2021 00:00:00 +0000

Hacktoberfest 2021 is over, and we got quite a few contributions from the open source community.

Contributions included various document improvements, adding jira as an issue tracker for generating changelogs and adding initial support for external vault!

The top contributors to Jenkins X in hacktoberfest 2021 were:

Marten Svantesson
Christopher vig
James Rawlings
Hays Clark
Anatoli Babenia

We would also like to thank all the contributors who participated and made it a success. The strength of Jenkins X lies in it’s vast community, and we hope to see many more major contributions from them in the near and far future.

Ask us questions

Missed this year’s hacktoberfest? No worries, you can always contribute to Jenkins X. We’re happy to help if you have any questions. Talk to us on our slack channels, which are part of the Kubernetes slack. Join Kubernetes slack here and find us on our channels:

#jenkins-x-dev for developers of Jenkins X
#jenkins-x-user for users of Jenkins X

We look forward to participating in the next hacktoberfest!

Blog: Hacktoberfest 2021

Wed, 06 Oct 2021 00:00:00 +0000

We are excited to announce that Jenkins X will be participating in Hacktoberfest again this year! Hacktoberfest is a month-long global celebration of open source software.

All backgrounds and skill levels are encouraged to participate in Hacktoberfest and join a global community of open source contributors.

Learn more about Hacktoberfest and sign up here.

Contribute to Jenkins X docs

We welcome your contributions to the Jenkins X documentation project!

Issues labelled “hacktoberfest” generally indicate good first issues. However, all pull requests will count towards your Hacktoberfest challenge.

Ask us questions

We’re happy to help if you have any questions. Talk to us on our slack channels, which are part of the Kubernetes slack. Join Kubernetes slack here and find us on our channels:

#jenkins-x-dev for developers of Jenkins X
#jenkins-x-user for users of Jenkins X

Find out more about becoming involved in the Jenkins X community here.

We look forward to seeing you in open source, fixing all the things!

Blog: Moving Jenkins X v2 artifacts

Thu, 26 Aug 2021 00:00:00 +0000

ACTION REQUIRED

TL;DR - Jenkins X specific helm repositories and container registries hosted on GCP have been moved to GitHub. This will mainly affect jx v2 users but there is expected to be a small impact on v3 users too. Below describes the steps we believe are needed to keep Jenkins X installations working as normal but there will be some action needed.

Why the disruption?

When Jenkins X first started we made heavy use of GCP’s services for hosting the cloud infrastructure needed by users to install and run Jenkins X. This was great as we could use the same IAM to push and maintain content from our own hosted build infrastructure and ensure we were validating the same experience of using cloud provider hosted services wherever possible. As Jenkins X grew in popularity the cloud costs began to increase with the pricing model from GCP, specifically the networking costs of cross continent egress.

Given this, for jx3 we decided to see if switching to GitHub packages for container images and GitHub pages for helm repositories would be better, the result was it is better. In fact we have made it super easy for users to switch to using GitHub pages for releasing helm charts and using GitHub packages.

Now that we have validated GitHub is more cost effective for hosting public images and helm charts for the Jenkins X project, we want to switch to using GitHub for all v2 plus v3 users, then shutdown the GCP services which are causing unnecessary cost.

It is expected that v3 users will need a small change and v2 slightly more. Details for both will be described below but it is worth noting that there hasn’t been a v2 release in 9 months and v3 was GA in April earlier this year, so we aren’t expecting too many folks on v2. We are aiming to limit any disruption and help provide easy steps to handle the move.

We apologise for any extra work caused however, this is required to preserve the long running hosting of Jenkins X artifacts both past and present. If you experience issues that are not covered by the steps below please reach out to the community slack channel and we can help address and update this blog with details.

What is changing?

We will be shutting down a number of GCP projects that contain old helm charts plus container images and moving them to be hosted by GitHub packages and pages.

Helm

https://chartmuseum.build.cd.jenkins-x.io
http://chartmuseum.jenkins-x.io
https://storage.googleapis.com/chartmuseum.jenkins-x.io

have been moved to

https://jenkins-x-charts.github.io/v2

AND

https://storage.googleapis.com/jenkinsxio/charts

has been moved to

https://jenkins-x-charts.github.io/repo

Images

The most recently versioned images from gcr.io/jenkinsxio have been moved to ghcr.io/jenkins-x

Labs

There are some old labs images and helm charts which should not be in use as they are either deprecated or replaced with GA versions in the v3 helm repo or container registry.

v2 users

The Jenkins X own v2 build infrastructure was retired at the start of the year as no more releases were planned and to reduce costs. With that we are unable to perform a new release that automatically switches references to images from gcr.io/jenkinsxio to ghcr.io/jenkins-x. If you are still using v2 then please update your references to this container registry. An alternative which has not yet been verified is to use a image swap Kubernetes mutating admission controller which takes configuration to switch the registry on the fly. We have asked on slack for help validating the approach so if you do try it please share feedback and config used to help others in the channel, we can then update docs.
In your boot git repository, run a search for references of http://chartmuseum.jenkins-x.io and https://storage.googleapis.com/chartmuseum.jenkins-x.io replace with https://jenkins-x-charts.github.io/v2
Change the lighthouse-jx-controller deployment to use an environment variable

JX_DEFAULT_IMAGE=ghcr.io/jenkins-x/builder-maven:2.1.155-778-patch3

Environment controller (can be skipped if not using)

i) change the environment controller image to be ghcr.io/jenkins-x/builder-maven:2.1.155-778-patch3

ii) change the image used in the pipeline, needs to be changed in the jenkins-x.yaml of the enviromnet repo:
```
agent:
 container: ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
```
iii) add this environment variable in the deployment of the environment-controller
```
 - name: BUILDER_JX_IMAGE
   value: ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
```

In your boot git repository, set the following values for env/jenkins-x-platform deployment:

Value
expose.Image	ghcr.io/jenkins-x/exposecontroller
expose.ImageTag	2.3.118
cleanup.Image	ghcr.io/jenkins-x/exposecontroller
cleanup.ImageTag	2.3.118
controllerbuild.image.repository	ghcr.io/jenkins-x/builder-jx
controllerbuild.image.tag	2.1.155-778-patch3
controllerbuild.env.BUILDER_JX_IMAGE	ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
postinstalljob.image.repository	ghcr.io/jenkins-x/builder-jx
postinstalljob.image.tag	2.1.155-778-patch3
postinstalljob.env.BUILDER_JX_IMAGE	ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
controllerrole.image.repository	ghcr.io/jenkins-x/builder-jx
controllerrole.image.tag	2.1.155-778-patch3
controllerrole.env.BUILDER_JX_IMAGE	ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
gcpreviews.image.repository	ghcr.io/jenkins-x/builder-jx
gcpreviews.image.tag	2.1.155-778-patch3
gcpreviews.env.BUILDER_JX_IMAGE	ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
gcactivities.image.repository	ghcr.io/jenkins-x/builder-jx
gcactivities.image.tag	2.1.155-778-patch3
gcactivities.env.BUILDER_JX_IMAGE	ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3
gcpods.image.repository	ghcr.io/jenkins-x/builder-jx
gcpods.image.tag	2.1.155-778-patch3
gcpods.env.BUILDER_JX_IMAGE	ghcr.io/jenkins-x/builder-jx:2.1.155-778-patch3

In your boot git repository, set the following values for env/lighthouse-jx deployment:

Value
image.parentRepository	ghcr.io/jenkins-x
image.tag	0.0.164
jxcontroller.image.repository	ghcr.io/jenkins-x/lighthouse-jx-controller
jxcontroller.image.tag	0.0.164
env.JX_DEFAULT_IMAGE	ghcr.io/jenkins-x/builder-maven:2.1.155-778-patch3

In your boot git repository, set the following values for env/lighthouse deployment:

Value

image.parentRepository ghcr.io/jenkins-x

image.repository ghcr.io/jenkins-x/lighthouse

env.JX_DEFAULT_IMAGE ghcr.io/jenkins-x/builder-maven:2.1.155-778-patch3
In your boot git repository, update the jenkins-x.yml replacing any reference to image: gcr.io/jenkinsxio/builder-go:2.1.155-778 with image: ghcr.io/jenkins-x/builder-go:2.1.155-778-patch3
In your boot git repository, update the jenkins-x-release.yml replacing any reference to image: gcr.io/jenkinsxio/builder-go:2.1.155-778 with image: ghcr.io/jenkins-x/builder-go:2.1.155-778-patch3
In your boot git repository, update the systems/jxing/values.tmpl.yaml setting

Value
nginx-ingress.controller.image.repository	ghcr.io/jenkins-x/nginx-ingress-controller

In any environment managed repository (e.g. environment-*-staging | production) update env/values.yaml:

Value
expose.Image	ghcr.io/jenkins-x/exposecontroller
expose.ImageTag	2.3.118
cleanup.Image	ghcr.io/jenkins-x/exposecontroller
cleanup.ImageTag	2.3.118

In any environment managed repository (e.g. environment-*-staging | production) update env/requirements.yaml replacing any exposecontroller repository url with https://jenkins-x-charts.github.io/v2

- alias: expose
  name: exposecontroller
  repository: https://jenkins-x-charts.github.io/v2
  version: 2.3.118
- alias: cleanup
  name: exposecontroller
  repository: https://jenkins-x-charts.github.io/v2
  version: 2.3.118

In any applications repositories who has already import into jx, update the ./jenkins-x.yaml add or replace :
```
pipelineConfig:
  agent:
    image: `ghcr.io/jenkins-x/BUILDER_YOU_NEED:2.1.155-778-patch3`
```
Here is the list of the principal images who has migrated to new ghcr:

Value	Version
builder-jx	2.1.155-778-patch3
builder-nodejs	2.1.155-778-patch3
builder-nodejs12	2.1.155-778-patch3
builder-nodejs14	2.1.155-778-patch3
builder-maven-java11	2.1.155-778-patch3
builder-php5x	2.1.155-778-patch3
builder-php7x	2.1.155-778-patch3
builder-python37	2.1.155-778-patch3
builder-dotnet	2.1.155-778-patch3

Have we missed anything? Please contribute to this blog or feedback on the slack channel.

v3 users

There is not expected to be significant disruption to v3 users but if there is anything needed beyond the steps below then we are asking users to reach out asap and we can update this blog.

Run jx gitops upgrade to ensure you upgrade to the latest version stream with the old helm repository removed. If you are tracking the LTS version stream please delay until Wednesday 1st September to run this.
In your cluster git repository, run a search for references of https://storage.googleapis.com/jenkinsxio/charts replace with https://jenkins-x-charts.github.io/repo
In your cluster git repository, run a search for references of http://chartmuseum.jenkins-x.io and https://storage.googleapis.com/chartmuseum.jenkins-x.io replace with https://jenkins-x-charts.github.io/v2
Switch jx-verify helm chart repository for any application you have which is built by Jenkins X 3. This is under your applications git repository ./charts/preview/helmfile.yaml change https://storage.googleapis.com/jenkinsxio/charts to https://jenkins-x-charts.github.io/repo . Here is an example that changes the main pipeline catalog packs which are used when first creating or importing applications.
Switch any gcr.io/jenkinsxio images to ghcr.io/jenkins-x in your application git repo .lighthouse/*.yaml files if you have references there

Have we missed anything? Please contribute to this blog or feedback on the slack channel.

When will all this take place?

This blog is the initial notice which we will socialise, please help to raise awareness.
Friday 27th August - the labs project will be scheduled to shutdown, short notice because we believe no services should be used, if they are it is an easy switch to upgrade to GA versions. Labs efforts are never intended to be production grade and are used at risk.
Monday 6th September - make the GCP container registry and helm repository bucket private, during which time any image versions that have not been transferred to GitHub can be requested via the community slack channel. All helm versions have been moved to https://jenkins-x-charts.github.io/v2 as described above.
Monday 13th September - schedule for shutdown the two GCP projects hosting the container registry and helm repository.

Why are only the most recent versions v2 images copied to GitHub packages and not all versions?

There are 14 Terabytes of data that make up the jenkinsxio container registry on GCP, it would be costly and wasteful to transfer all this to GitHub so we picked the last known version of each image that was released last year. If there are specific images that you wish to use either pull / push them yourself to a container registry of your own or reach out and on a case by case effort, we can look to move them to GitHub while the read permissions are made private and before the project is shut down.

I’m on v2 and use a builder image which is not available on GitHub container registry, how do I build my own version to work with the new helm and image repositories?

The old v2 jx code lives on a branch you will need to:

fork this branch
apply search and replace changes for i) gcr.io/jenkinsxio to ghcr.io/jenkins-x example ii) https://storage.googleapis.com/chartmuseum.jenkins-x.io to https://jenkins-x-charts.github.io/v2 example
run make linux to build the updated jx binary
build and push the images you require, for example

docker build -f Dockerfile.builder-maven -t ghcr.io/jenkins-x/builder-maven:2.1.149-768-patch3 .
docker push ghcr.io/jenkins-x/builder-maven:2.1.149-768-patch3

once the builder-jx/go/maven/etc.. are pushed, you will need to configure the charts:

if a chart allows the image to be overridden using a value you can add that to your env/values.tmpl.yaml, replacing current gcr.io/jenkinsxio images with ghcr.io/jenkins-x e.g.

jenkins-x-platform:
  gcpreviews:
    image:
      repository: ghcr.io/jenkins-x/builder-jx
      tag: [your custom tag]

if you cannot override the image using a helm value you may need to:

download each needed chart/subchart needed to make jx work, exemple for jenkins-x-platform/, jxboot-resources/releases
for each charts/subcharts replace current gcr.io/jenkinsxio images with ghcr.io/jenkins-x if it exists or rebuild and host them on a private registry if not
host updated charts on a private chartmuseum
switch boot git repo to use custom charts

I’m getting a missing arg `--provider-values-dir` and helm repository https://jenkins-x-charts.github.io/v2 does not have an associated prefix in in the ‘charts/repositories.yml’ error

A few users have been hitting this error, it is related to the jx version used. This thread with the help of Francesco Capozzo contains both image tags and versions to use instead:

https://kubernetes.slack.com/archives/C9MBGQJRH/p1631112970450800

Images:

ghcr.io/jenkins-x/builder-jx:2.1.142-761-patch3
ghcr.io/jenkins-x/builder-maven:2.1.142-761-patch3
ghcr.io/jenkins-x/builder-go:2.1.142-761-patch3

Chart versions to use rather than relying on the version stream:

dependencies:
* name: jxboot-resources
  repository: https://jenkins-x-charts.github.io/v2
  version: 0.0.43
* alias: tekton
  name: tekton
  repository: https://jenkins-x-charts.github.io/v2
  version: 0.0.63
* alias: prow
  condition: prow.enabled
  name: prow
  repository: https://jenkins-x-charts.github.io/v2
  version: 0.0.1773
* alias: lighthouse
  condition: lighthouse.enabled
  name: lighthouse
  repository: https://jenkins-x-charts.github.io/v2
  version: 0.0.843
* alias: lighthouse-jx
  condition: lighthouse.enabled
  name: lighthouse-jx
  repository: https://jenkins-x-charts.github.io/v2
  version: 0.0.121
* alias: bucketrepo
  condition: bucketrepo.enabled
  name: bucketrepo
  repository: https://jenkins-x-charts.github.io/v2
  version: 0.1.42
* name: jenkins-x-platform
  repository: https://jenkins-x-charts.github.io/v2
  version: 2.0.2411
* name: jx-pipelines-visualizer
  repository: https://jenkins-x-charts.github.io/repo
  version: 1.7.3

Blog: How to debug your Tekton pipelines

Wed, 18 Aug 2021 00:00:00 +0000

TaskRun breakpoint functionality is no longer supported since Tekton 0.29.0 upgrades in 3.2.298. For more info see Kubernetes 1.22 - Breaking change!.

Tekton recently introduced a debug feature when you create TaskRun resources so that steps can be paused at a breakpoint until told to move forwards so that you can diagnose why pipeline steps fail.

The latest Tekton release only supports breakpoints on TaskRun resources but there is a Pull Request #4145 to add support also to debugging PipelineRun resources as well. If you are reading this please add your thumbs up emoji feedback to the PR #4145

We’ve switched Jenkins X to use a preview image of Tekton with PR #4145 included so that Jenkins X developers can easily debug their pipelines (which typically are PipelineRun resources).

How to debug Tekton Pipelines

Here is a demo which shows how to debug pipelines:

Prerequisites

Make sure your cluster is upgraded to the latest version stream.

If you intend to use the jx in the below examples make sure you upgrade the CLI too

Enable a breakpoint

To enable a breakpoint you can:

use the Lens UI as shown in the above video by:
- right click on a Pipeline action menu
- select Breakpoint -> Add
you can use the jx pipeline debug command then select the pipeline to add/remove a breakpoint.

Viewing breakpoints

You can view breakpoints in the Lens UI in the Breakpoints tab or via:

kubectl get lighthousebreakpoints

# you can use the short name:
kubectl get lhbp

Using a breakpoint

Once you have set a breakpoint defined for a particular Pipeline you need to trigger the pipeline. e.g. perform a git commit on the git branch to trigger a new pipeline to execute.

The pipeline will execute as normal; you’ll be able to view it execute via:

Lens UI
run jx pipeline grid to watch pipelines run and select the one you wish to view the log
run jx pipeline log to watch the log of a specific pipeline

Opening a shell

Once your breakpoint is reached the pipeline pod will pause, waiting to continue.

At this point you can then open a shell inside the container.

The easiest way to do this is via the Lens UI, click on the Pipeline action menu then Shell -> latest step and a shell will open.

Otherwise you can use:

kubectl exec -it -c $name-of-container $name-of-pod (sh | bash | ash)

Continuing after the breakpoint

If you wish to continue the execution of a pipeline there are multiple scripts you can run inside the shell you can run inside the shell in the pipeline to tell the pipeline to continue:

Script	Description
`/tekton/debug/scripts/debug-continue`	Mark the step as completed with success by writing to `/tekton/tools` so that the pipeline continues executing
`/tekton/debug/scripts/debug-fail-continue`	Mark the step as completed with failure by writing to `/tekton/tools` which can lead to the pipeline terminating

Removing breakpoints

There are a few ways to delete breakpoints.

You can run jx pipeline debug and toggle off any existing breakpoints.

You can use the Breakpoints tab in Lens UI then click the breakpoints action menu then Remove

Or find the one you want via:

kubectl get lhbp
kubectl delete lhbp whatever-the-name-is

Conclusion

So there you have it; nice and easy debugging of pipelines so you can diagnose why pipelines fail and try incrementally fix things up from inside the pipeline pods! Pretty cool eh!

Let us know via slack or the issue tracker if you can think of any ways we can make this even easier to use! Also check out the Tekton enhancement proposal 42 that covers this capability in the underlying tekton controller and pods.

Finally please add your thumbs up emoji to the tekton PR #4145 :)

Blog: How to use GitOps and Kubernetes External Secrets for better audit and security

Tue, 17 Aug 2021 00:00:00 +0000

So GitOps is a cool approach to managing kubernetes resources in a cluster, by checking in the source code for:

the kubernetes YAMLs
details of the helm charts you want to install along with any configuration
kustomize scripts.

Then everything is versioned and audited; you know who changed what, when and why. If a change breaks things, just revert via git like any other source code change.

You can then add pipelines to verify changes in the Pull Requests result in valid kubernetes YAML etc.

Then if you merge changes to git then an operator detect the change and do the kubectl apply (or helm install or whatever).

There are a number of tools out there for doing this. e.g. Anthos Config Management, argo cd, fleet, flux cd and kapp controller

So why did Jenkins X not use these tools and instead created its own git operator?

Standardising GitOps layouts

Over time it would be great to have more standardisation of the Git layout given the different tool.

Our current recommended layout that works with many GitOps tools is described here.

Why Jenkins X uses helmfile template

A number of solutions in the GitOps space define which helm charts to install in git with configuration files; or specify which kustomize templates to apply etc.

However Jenkins X defaults to using helmfile to manage installing, upgrading and configuring multiple helm charts. Then we use helmfile template to render the helm charts as kubernetes resources.

We do this for a very good reason; so that we can easily version all kubernetes resources in git including those that come from a helm chart - which means you can easily view the entire history of changes of any kubernetes resources - whether they come from inside a helm chart, some configuration values of a chart or kustomize scripts etc.

This avoids you having to mentally understand how helm charts will change over time with the helm chart version and/or helm configuration; or the effect of kustommize scripts. You can just view the history of any kubernetes resource.

We use conventions to ensure that each kubernetes resource has a canonical file name in git to make this whole process much simpler.

e.g. this is the git history of the cert manager deployment resource in our production cluster so you can see what changed when over time.

The exception to this rule is kubernetes Secrets which are stored instead as ExternalSecrets but which have their own history (e.g. in case you change the location of where the secrets are stored or modify the metadata, annotations or labels etc).

Why we use Kubernetes External Secrets

Jenkins X 3.x uses Kubernetes External Secrets to manage populating secrets from your underlying secret store such as:

Alibaba Cloud KMS Secret Manager
Amazon Secret Manager
Azure Key Vault
Hashicorp Vault
GCP Secret Manager

graph TB subgraph A[Kubernetes Cluster] sqB[External Secrets Controller] subgraph C[secrets-infra ns] sqCV[Cloud Secret Manager] end subgraph D[Kube api server] end D -- Get ExternalSecrets --> sqB sqB --> D sqB -- Fetch secrets properties --> sqCV sqCV --> sqB subgraph E[app ns] sqEP[pods] sqES[secrets] end sqB -- Upsert Secrets --> sqES end

You can then keep all your secrets inside your cloud native secret store which also allows:

easy to automatically rotate any secret at any time independently of git
use fine grained RBAC on each secret

How to use this approach to GitOps and Secrets if not using Jenkins X

If you use Jenkins X then you get all of the above benefits. But what if you want to use some other kind of GitOps operator toolchain?

One option is to use the jx-secret-postrenderer yourself if you use helm or helmfile to then render the helm charts as raw YAML you can check into your git repository and implementing the conversion from Secret to ExternalSecret.

Another option is to reuse the Jenkins X Makefile and pipeline to setup the config-root git layout after converting Secrets to ExternalSecretes and pre-populating any missing secret store values.

Summary

If you are looking at adopting GitOps then we highly recommend you check into git all of your kubernetes resources including those that come from a helm charts or kustomize scripts (apart from Secrets!) as it massively simplifies understanding how kubernetes resources change over time using just pure git.

If you are using GitOps you may want to look into using Kubernetes External Secrets to simplify integrating secrets for cloud native secret stores into your kubernetes cluster to provide finer grained RBAC and easier secret rotation.

Blog: Jenkins X 3 and Argo CD

Mon, 28 Jun 2021 00:00:00 +0000

There have been a number of requests from the Jenkins X community to use Argo CD for the last mile deployment phase of their continuous delivery pipelines. This blog explains some of the advantages of using Jenkins X and Argo CD all together.

What’s included?

Jenkins X for Cloud Infrastructure using Terraform, core installation management with GitOps, external secret management, ingress controller, quickstarts, automated CI + CD pipelines, ChatOps
Tekton for pipeline orchestration
Argo CD for end users application deployments (not the main installation) Argo CD provides a declarative GitOps approach to deploying Kubernetes based applications. There is a GUI which helps construct the deployment definition (in the form of an Application custom resource) which offers a number of configuration options, as well and providing insight into your clusters applications.

You might be wondering why you would want to use BOTH Jenkins X and Argo CD together. Jenkins X aims to embrace OSS, where possible providing a nice UX to integrate with other projects and help provide better solutions for building, developing and running software on Kubernetes. Jenkins X indeed does have a git operator that applies Kubernetes YAML from a Git repository but there are some differences:

Jenkins X uses GitOps principles for the entire installation, i.e. the starting point is a Git repository which is used to provision a cluster and manage (automatic if users wish) upgrades whereas today Argo CD uses a manual kubectl apply to manage the Argo installation itself.
External Secrets integration for Vault, Google Secrets manager etc is something that Jenkins X provides out of the box. When Kubernetes based applications require a secret we prefer the source is stored in a secrets manager and the value synchronised automatically into the cluster enabling easier secret rotation, avoiding local secrets on machines and an easier UX for working with secrets. When adding an Application via Argo CD users can leverage the Jenkins X approach to working with secrets and rely on Argo to manage the deployments. If users prefer to work with SOPS that is totally fine too and will work in the same way as they are used to.
If you do not wish to expose direct access for the Kubernetes cluster to developers then using a combination of Jenkins X UI for accessing logs and Argo CD for managing application deployments may be enough.
Using the Jenkins X approach to promoting via automated pull requests via pipelines to add new helm charts and update chart release version numbers.
In a Jenkins X multi cluster setup you can choose to use jx-git-operator or Argo CD in the remote cluster to syncronise promoted applications into the staging or production clusters. This way you get all the benefits of the integrated Jenkins X experience to manage the build infrastructure and provide the developer experience like preview environments and chatops etc.

This is done by creating a new Git repository that will contain a number of Argo CD Application Kubernetes resources. This also means we handle disaster recovery scenarios and are able to recover the Jenkins X installation which includes Argo CD plus any applications managed by Argo CD itself.

There may be a couple of models to try but here we are going to describe an approach we have validated when using Jenkins X to manage the core Kubernetes installation, using Tekton for Continuous Integration, Jenkins X for application promotion and Argo CD for deployments.

Install Argo CD with Jenkins X

Prerequisite is a working Jenkins X 3 installation

Add a Argo CD helmefile to your Jenkins X cluster git repository:

mkdir -p helmfiles/argocd

Create a new file

./helmfiles/argocd/helmfile.yaml

namespace: argocd
releases:
- chart: argocd/argo-cd
  name: argocd
- chart: ../../charts/argo-applications
  name: argo-applications

Edit the root ./helmfile.yaml and add a new path reference to the helmfile above

- path: helmfiles/argocd/helmfile.yaml

Jenkins X will resolve default helm values from the Jenkins X version stream.

Create a new git repository to contain the Argo CD Application Kubernetes resources, this means we can use Jenkins X pipelines to promote users applications via GitOps.
Back in your clusters Git repository create a new folder:

mkdir -p charts/argo-applications

create a file in the new folder and replace the repoURL: with the Git URL from step 2 above.

charts/argo-applications/applications.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: staging
  namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
  destination:
    name: ''
    namespace: argocd
    server: 'https://kubernetes.default.svc'
  source:
    path: .
    repoURL: 'https://github.com/$REPLACE_WITH_GIT_OWNER/  $REPLACE_WITH_GIT_REPO'
    targetRevision: HEAD
    directory:
      recurse: true
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - Replace=true
      - CreateNamespace=true

commit your changes and push

git add helmfiles/argocd/helmfile.yaml
git add charts/argo/applications.yaml
git commit -m 'feat: enable Argo CD'
git push

If you are on a fork then PR your changes

Follow Jenkins X admin logs to ensure Argo CD is installed

jx admin logs

Get the Argo CD UI password using something like ksd or base64 -D to decode the value:

kubectl get secret argocd-initial-admin-secret -n argocd -oyaml | ksd

Get the Argo CD UI hostname and login using the admin username + passowrd from step 6 above (you may need to accept insecure TLS browser prompt if not using production TLS)

kubectl get ing argocd-server -n argocd

Create a new quickstart or update an existing Jenkins X applications ./lighthouse/jenkins-x/release.yaml pipeline, replaceing the Tekton promote-jx-promote step to use argo instead:

- image: uses:jenkins-x/jx3-pipeline-catalog/tasks/updatebot/release.yaml@versionStream
  name: promote-release
  script: |
    #!/usr/bin/env sh
    source .jx/variables.sh
    jx updatebot argo promote --target-git-url https://github.com/$REPLACE_WITH_GIT_OWNER/$REPLACE_WITH_GIT_REPO

Now merge a change to your application or manually trigger a release pipeline jx pipeline start to get a new release and promote using Argo CD. Once the release pipeline finished for your application you should see a new application.yaml resource in your uber application Git repository we created in step 2 above. Once this change is merged to the main branch Argo detects the new chart / version and applies it to the cluster. You will be able to track the status and health of the deployment via the GUI that you logged in with step 7 above.

Conclusion

I really did like the experience of Argo CD, there’s some overlaps with Jenkins X across the wider projects however I think there’s enough of a differentiation for both to complement each other. What I loved was that both projects as well as others like flux promote the use of GitOps. Anyways, congratulations to the Argo project and I hope users from both communities find this blog useful and / or interesting.

Blog: Continuous microservices with databases in Jenkins X

Fri, 25 Jun 2021 00:00:00 +0000

A common question we get asked on the Jenkins X project is how to get started creating microservices that use databases with automated CI/CD with GitOps Promotion and Preview Environments.

To make things a little easier to get started we’ve created a new node-postgresql quickstart.

Before you start

If you are using the cloud then we prefer cloud over kubernetes for things like databases, storage, ingress and secret managers so please try use your clouds managed databases if you can.

So ideally you’d set up your database via your infrastructure as code solution, such as terraform, and then associate your kubernetes Service Account to a cloud IAM role to access the database.

However to provide something simple that just works in any kubernetes cluster this quickstart uses the postgres-operator to manage setting up each database cluster in each environment. So to be able to use this quickstart you will need to install this operator into your cluster.

You can add charts to your cluster via the CLI. From inside a git clone of your cluster git repository run the following command:

jx gitops helmfile add --chart commonground/postgres-operator --repository https://charts.commonground.nl/ --namespace postgres --version 1.6.2

This will modify the helmfile.yaml to point at a new helmfiles/postgres/helmfile.yaml file to deploy the postgres-operator chart.

Then git commit and push that change to your cluster. You can watch it run via jx admin log -w.

Create the quickstart

Make sure you have an up to date cluster as this particular quickstart is new and only shows up in recent clusters.

Now create the quickstart in the usual way…

jx project quickstart

If you know you want to create the node-postgresql quickstart you can do this to pre-filter the list for you:

jx project quickstart -f postgres

Once you have finished the import process will set up the webhooks and enable CI/CD and the application will be released and promoted to the staging environment.

If you want to know more about what happens as you create quickstarts or import projects see how it works.

You can watch via the various pipelines run in the various UI options or via the CLI use:

jx pipeline grid

When the release is done and the promotion has completed you should be able to try it out via:

jx application get

You should be able to click on the URL for the new quickstart and try it out once the database is initialised and the pods start up.

It can take a few minutes first time you deploy the quickstart for the database cluster to be setup and initialised; so you can watch the pods run via

kubectl get pod -n jx-staging -w

How does it work

In many ways this chart is fairly similar to other quickstarts in that it uses the Jenkins X pipeline catalog with tekton to add automated CI/CD pipelines and so forth.

However to support the database there is a custom chart included inside this quickstart that does a few different things…

it creates a Postgresql custom resource for the postgres-operator which will instruct the postgres-operator to spin up a database cluster and generate a Secret to access the database. You can view this in your file at charts/$myAppName/templates/ or this file in the quickstart
there is a charts/$myAppName/init.sql file or this file in the quickstart which is used to setup the database tables and populate any initial data. You can use this file to perform any startup schema migration or data population. For more realistic applications you could use a custom tool and image to implement schema migration in a more sophisticated way.
the init.sql script is then installed as a ConfigMap via the charts/$myAppName/templates/initdb-cm.yaml file or this file in the quickstart
the charts/$myAppName/templates/deployment.yaml file or this file in the quickstart defines:
- an in init container which sets up the database before the application starts to run. The nice thing about using an init container for schema migration is that it runs before any new version of your application gets any network traffic so that you can keep iterating on your microservice and keep changing your database schema across all of your environments and things work well.
  - Though make sure your init container performs database locking correctly so that multiple init containers running concurrently don’t block each other unnecessarily. If this becomes an issue you could introduce something a little more complex. e.g. include a Job with a unique name for each release in your chart to perform the migration so that only one migration Job runs at once and have your init container wait for your job to complete.
- the Deployment also uses a secret created by the postgresql operator to be able to connect to the database

Previews

Databases often need a fair amount of maintenance, backup, upgrading and clean up over time. e.g. you may periodically update your Staging database with data from Production (maybe anonymised in some way?).

So creating a whole new database from scratch for every Preview Environment to test every code change in your microservice is maybe overkill.

By default the preview environment of this quickstart is configured to reuse the Staging environments database. This speeds up the preview startup time and reduces your cloud footprint and infrastructure cost.

This is done via:

configuring the preview environment to point at the database in the staging namespace and disabling the creation of the Posgresql custom resource to create a new database cluster in the preview environment
using the helmfile hooks mechanism in previews to copy the required database secrets from the staging namespace so that the preview can connect to the staging database.

How we can improve

This quickstart is just a start but we can improve in a number of directions - fancy helping out?

More languages and frameworks

It should be easy to replicate the same kind of quickstart mechanism for other languages and frameworks if anyone fancies trying it out? :) We love contributions! Pop by and chat with us on slack if you want to discuss it further.

Cloud databases

Longer term it would be nice to support other kinds of database operators too.

We prefer cloud over kubernetes so if you are using a cloud it would be better to default to a cloud database instead of a kubernetes hosted one.

There are a number of other ways to define cloud infrastructure via Custom Resources such as:

So it’d be interesting to see if we can replicate this model for other kinds of cloud database on different cloud providers. Mostly it’d be a Custom Resources to define the database instance and a way to inject the host and secret. Some database providers require an additional sidecar proxy too.

It would be easy to add optional configuration in the quickstart to support either the postgres-operator or equivalents in AWS Controllers for Kubernetes, Azure Service Operator, Crossplane or Google Config Connector via a simple flag in the chart/$name/values.yaml file

More modularity options

In a pure microservice kind of world, each database would be owned by a single microservice; so embedding the database definition and schema migration into a single helm chart is the simplest thing that can work across multiple environments and with progressive delivery etc. It makes it easier to tie changes to the microservice and schema together into a single chart.

However sometimes you want to have multiple services sharing a database. For that you could have 1 microservice be the owner of the database and other services reuse it. Or you could separate out the database definition + migration to a separate helm chart which is released independently.

So it might make sense to make separate quickstart just to define the database definition and schema migration for these use cases: maybe via a Job rather than an init container).

Conclusion

So there’s a really quick way to spin up a node based microservice using a database with an operator handling the setup of the database cluster which works well with multiple environments, progressive delivery and Preview Environments.

Give it a try and let us know how you get on or if you can think of any more ways we can improve

Blog: Jenkins X at cdCon

Tue, 22 Jun 2021 00:00:00 +0000

cdCon 2021 is about to start with lots of great sessions.

Here’ a list of the Jenkins X related sessions:

Tuesday, June 22 GitOps Summit

Best Practices for Secret Management with GitOps - Kara de la Marck, CloudBees
- GitOps uses Git as the “single source of truth” for declarative infrastructure and enables developers to manage infrastructure with the same Git-based workflows they use to manage a codebase. Having all configuration files version-controlled by Git has many advantages, but best practices for securely managing secrets with GitOps remain contested. Join us in this presentation about GitOps and Secret Management. Attendees will learn about different approaches to secret management with GitOps, the issues involved, and the secret management solutions offered by various tools and platforms. We will discuss the pros and cons of Vault, SOPS, offerings by public cloud providers, and more.

Wednesday, June 23

MLOps with Jenkins-X: Production-ready Machine Learning by Terry Cox
- Explore ways to treat Machine Learning assets as first class citizens within a DevOps process as Jenkins-X MLOps Lead, Terry Cox demonstrates how to automate your training and release pipeline in Cloud environments, using the library of ML template projects provided with Jenkins-X.
Enabling a DevOps Mindset at Scale in an Enterprise by Jimmy McNamara & Nick Penston, Fidelity Investments
- Talk through the cultural enablers to create a strong DevOps culture within large organisations. Nick and Jimmy will talk through the cultural enablers to support large numbers of very talented and ambitious cloud engineers. Touching on strategies supporting strong communication and talent development for cloud engineers. Key themes: Talent Development Enabling DevOps culture Harnessing the voice of the developer/cloud engineer
BoF Session: Jenkins X - James Strachan & James Rawlings, Cloudbees by James Strachan & James Rawlings
- This BoF will be an open discussion on anything related to Jenkins X, automating Continuous Delivery, Kubernetes, cloud and how to Accelerate delivering value to your customers

Thursday, June 24

How Jenkins X is Integrating Observability from the Inside, and the Benefits for its Users by Vincent Behar, Dailymotion
- In this session, we’ll focus on observability for Jenkins X: what observability means for a Continuous Delivery platform such as Jenkins X, and why it’s important.
- We’ll see how Jenkins X v3 is integrating observability from the inside, using standards such as OpenTelemetry, and packaging a full observability stack using Grafana - with Loki, Promtail, Tempo, and Prometheus.
- And we’ll highlight the benefits for the users:
  - platform observability with alerting for all the Kubernetes components (Lighthouse, Tekton, Cert-Manager, …)
  - out-of-the-box dashboards for Continuous Delivery Indicators (Mean Lead Time, Time To Review, Release, and Deployment Frequency, …)
  - distributed traces for your pipelines
What’s new with Jenkins X 3 by James Rawlings & James Strachan, CloudBees
- This session will cover the architectural improvements and new features of Jenkins X v3.
- We’ll dive into the key areas that have undergone major development.
- GitOps: a better approach to installation and upgrades leveraging Helmfile, using an in-cluster operator to apply desired state stored in Git.
- Secrets: out of the box integration with external secret management solutions such as Google + Amazon cloud services Infrastructure: decoupled infrastructure management using Terraform UX: hosted dashboard to link your pull request logs along with a local Octant UI for deeper cluster visualisations
- Cloud Native Pipelines: now using vanilla Tekton pipelines with no abstraction layer.
- Along with Lighthouse to handle both in repo and shared pipeline definitions we now have a clean, extendable way to describe pipelines. Health: health checks and insight via Grafana
Embrace ChatOps by Following the git-flow as Usual by Rick Zhao, Qingcloud
- Simplicity matters. It’s not desirable for us to invest in more systems or platforms. The best scenario is to keep our daily work unchanged. For example, software engineers usually spend a lot of time writing codes or documents and also have interactions with various pull requests. Lighthouse is similar to Prow, but it supports multiple git providers. We can speed up delivery no matter if you’re using GitHub, Gitlab, or Gitea.

Blog: Don't use docker, use kubernetes

Mon, 17 May 2021 00:00:00 +0000

Are you developing software that’s intended to run on kubernetes? If so we recommend not to use docker on your laptop.

Docker on Windows/MacOS helps you run a VM that can then run linux containers easily. But why bother?

We highly recommend just use a development kubernetes cluster - build and run your containers there instead then you’re closer to a production like environment.

Why use kubernetes instead of docker?

why test on a completely different VM and container orchestrator than production? It’s better to test on a similar environment to where you are really going to deploy your code
test your kubernetes yaml / helm chart and associated configuration at the same time as you run your containers helps you catch mistakes earlier:
- it’s not just about running the container image; it’s about lots of other things too like networking, configuration, secrets, storage/volumes, cloud infrastructure, service mesh, liveness/readiness/startup probes - so why not test all of those things rather than just the image?
some corporate environments don’t let you run VMs on your laptop anyway so running docker locally isn’t an option

How to get kubernetes?

First you’ll need a kubernetes cluster.

I fully agree with James Ward that developers should not need to run kubernetes. Friends don’t let friends setup and manage kubernetes clusters by hand :).

So try ask your infrastructure team for a development cluster or, if you can, use the cloud to set-up a managed kubernetes cluster. All the public clouds have a relatively straightforward way to spin up a fully managed kubernetes cluster for you that will be relatively inexpensive & they are easy to scale down when you don’t need them.

e.g. on Google Cloud it’s a couple of clicks and about 5 minutes later you’ll have a fully managed kubernetes cluster ready to use. Its easy to enable auto-scaling too. Plus there’s a free tier.

If that’s not easy for you to achieve you could try reuse a namespace in your staging cluster? Though we don’t recommend developing on a production cluster; its too easy to accidentally mess up production; e.g. by using up too many resources or overwriting a cluster scoped resource like a CustomResourceDefinition or ClusterRole etc.

If you have zero budget you could try minikube or kind on top of docker; though its much better to reuse as close to the production setup of kubernetes as you can - there can be large variations in platform, version, setup, network, machine size and so forth.

If you are deploying software on kubernetes then I’d hope you’ve some managed kubernetes solution; so why not use that and spin up another cluster for your team for development?

If your budget is so stretched that you can’t afford 2 kubernetes clusters; one for production and one for development + staging; maybe it’s time to look at using just pure serverless / FaaS instead of kubernetes anyway? Even in that case it’s better to use your serverless / FaaS infrastructure than docker locally for similar reasons.

How do I connect to kubernetes?

So first you’ll need to install kubectl and connect to your kubernetes cluster. This step is provider specific - so refer to your kubernetes provider / cloud vendor for that bit.

You can then verify you are connected by running some kubectl get commands such as

kubectl get ns
kubectl get node
kubectl get pod

How do I replace docker with kubernetes?

docker run => kubectl run

Instead of docker run to run container images you can use kubectl run to run a container image

If you wish to expose a DNS name for a pod you can use kubectl expose

docker build => kubectl build

A drop in replacement for docker build is this kubectl plugin

compose => helm

Some folks use docker compose files to define all of their various microservices; front end, back end, database etc.

If you have deployed your applications to staging/production then you are probably already using either helm charts or kubernetes yaml to define those deployments and services already.

So just reuse all of them when running things locally in your own cluster/namespace.

Then you don’t have to keep 2 completely different configuration files in sync; you can usually just reuse the same helm charts in all environments and clusters.

If you have some compose files you could try out kompose to help use them more effectively on kubernetes

testcontainers => sidecars / kubedock

Some folks use testcontainers for running extra containers in docker to make it easier to do testing. e.g. to run a database service to run tests using the database.

You could try kubedock with testcontainers to see if that solves your problem without requiring a local docker installation.

It does depend a little on what your solution is for CI.

If you are using Jenkins X or tekton pipelines then you can define sidecars in your pipeline to make sure you have whatever additional services you need when running your tests.

If you are using Jenkins then you can add the side cars to the pod.yaml you use with the kubernetes plugin or you could reuse the tekton client plugin and use tekton pipelines and sidecars

If you are GitHub Actions then you can spin up a kubernetes cluster using kind via this kind github action - you can then spin up whatever services you need for your tests via kubectl apply or helm install

help! we are not even using kubernetes yet

If you have not even started on your journey to kubernetes and have no idea what a helm chart is, you could consider setting up Jenkins X in your cluster which will then:

automate setting up the CI / CD for your projects including automatically creating versioned container images and helm charts whenever you merge changes to the main branch
automatic promotion through environments via GitOps so that new versions of your services are automatically promoted to your Staging environment and, by default, when approved are promoted to Production
Preview Environments automatically spin up Preview Environments for your Pull Requests so you can get fast feedback before changes are merged to the main branch

Once someone on your team has setup up Jenkins X then please follow the development guide

other handy kubectl commands

You may find these handy:

kubectl logs to view the logs of any running pod (which is kubernetes terminology for 1 or more containers deployed together as a single unit on the same node)
kubectl port-forward lets you easily port forward from a pod to a local port so you can easily test things out without having to expose things via ingress

inner loop

If you want to optimise your inner loop so you can edit source code and see the changes running quickly on kubernetes then please check out the options for inner development loop

Conclusion

Lots of developers have grown fond of their docker installation over the years. Docker was a total game changer in its day!

However if you are building/testing/debugging software to deploy on kubernetes we highly recommend you consider reclaiming the memory, CPU & disk from your laptop and stop running docker locally and just use more kubernetes.

It will help you go faster, find those kubernetes related issues sooner and help you learn more about kubernetes which will be useful for figuring out production issues whenever they happen.

Blog: Jenkins X 3 - May 2021 LTS

Wed, 12 May 2021 00:00:00 +0000

May 2001 LTS release is now available!

LTS is a slower cadence version stream which contains a verified set of releases and configurations that have been used by teams tracking the bleeding edge Jenkins X.

Included in this release:

Protect Pipeline Visualiser with OAuth2 how to docs
Terraform repositories are now protected by the Jenkins X version stream
external-dns helm chart upgrade to v5.0.0
Reduce external secrets polling of cloud services to reduce cloud costs
[Jenkins] for users using Jenkins the Tekton Client plugin is now installed by default
Stackdriver format logging enabled when using GKE and services that use jx-logging library. If you enable the Stackdriver API in GCP you will get well formatted logs and alerts via Stackdriver.
Jenkins X Grafana dashboards updates with Lighthouse telemetry

Blog: Jenkins X 3.x GA is here!

Thu, 15 Apr 2021 00:00:00 +0000

I’m super excited to announce the 3.0 GA (General Availability) release of Jenkins X!

Jenkins X automates your CI/CD on kubernetes to help you accelerate:

Automated CI/CD pipelines lets you focus on your actually application code while Jenkins X automatically creates battle tested Tekton CI/CD pipelines for your project which are managed via GitOps so that its super easy to keep your pipelines up to date across your repositories or to upgrade or override pipelines or steps for specific repositories.
Automatic promotion of versioned artifacts via GitOps through your Environments such as Staging, Pre-production and Production whether they are running in the same kubernetes cluster or you are using multiple clusters for your environments
Preview Environments lets you propose code changes via Pull Requests and have a Preview Environment automatically created, running your code in kubernetes to get fast feedback from your team before agreeing to merge changes to the main branch
ChatOps comment on Pull Requests to give feedback, approve/hold changes, trigger optional pipelines for additional testing and other ChatOps commands

Demo

Here’s a demo of how to develop code with Jenkins X

Documentation

the main documentation of the changes are:

the new architecture with modular plugins and improved extension points
what has changed since 3.x started
how 3.x compares to 2.x
DevOps Guides and DevOps Patterns provides an overview of our learnings in the DevOps space

here’s a brief summary of the differences:

Changes since the 3.x beta

The following improvements have been made since the first beta:

Integrated observability and monitoring with Pipeline Tracing
Auto scale preview environments with Osiris
Enable auto upgrade to keep your cluster up to date

User Changes since 2.x

As a user the high level UX of Jenkins X is similar:

automated Continuous Delivery pipelines for using tekton for your repositories with automatic promotion between your environments
pull requests on your repositories create separate Preview Environments where your team can review your changes and give fast feedback before your changes are approved and merged into the main trunk.

New features

we now default to vanilla tekton YAML for defining pipelines while accelerating your tekton with tekton catalog
we include an open source dashboard for visualising pipelines and logs which you can invoke via:

jx dash

Platform Changes

we now use helm (3.x) and helmfile along with optionally kustomize in a GitOps style to define and configure both Jenkins X itself, your tools and applications in any namespace
support multi cluster out of the box so you can keep Staging and Production in separate clusters to your development cluster where your pipelines run, you create and release immutable container images and other artifacts.
to setup or upgrade Jenkins X we use terraform to setup your cloud resources on Azure, Amazon or Google while also supporting on-premises, minkube and OpenShift - see the Admin Guides for more detail
- the actual installation of kubernetes resources takes place using the git operator so it runs reliably inside the cluster itself
we default to using Kubernetes External Secrets to manage all secrets for Jenkins X itself, development tools and your applications too.
- This means we can support various secret backends such as Alibaba Cloud KMS Secret Manager, Amazon Secret Manager, Azure Key Vault, Hashicorp Vault or GCP Secret Manager
- It also means we can then check in all kubernetes resources and custom resources directly into git (apart from Kubernetes Secrets) so that it super easy to version, review and reason about your kubernetes resources in a GitOps way.
built in TLS and DNS support along with Heath reporting and visualising via kuberhealthy
we now have an LTS distribution which lets you switch to a much more slower cadence of releases of Jenkins X

We have been using Jenkins X 3.x in production now for many months (for CI/CD of all of the 3.x codebase and continuously upgrading our cluster in the standard way and it’s been much simpler and easier to use, operate and configure.

We have also been continuously delivering changes from Jenkins X into our production cluster for many months now and it’s been working great - GitOps FTW!

In general Jenkins X 3.x is now much simpler and more flexible. It supports lots more platforms than before and should be easy to extend and configure for other platforms too.

Getting started

If you have never tried 3.x before then please follow the Admin Guide to get Jenkins X installed on your cloud provider, on-premises kubernetes cluster or minikube.

If you previously tried the 3.x alpha then the migration instructions are here.

For folks on older 2.x versions of Jenkins X please see the 2.x migration instructions

Once your cluster has been installed or migrated then check out the User Guide on how to develop software continuously with Jenkins X.

Final thoughts

A huge thanks goes out to all the contributors, folks in the Jenkins X community and the community around all the open source projects we reuse who’ve helped get this beta together. The improvements in Jenkins X 3.x since 2.x are totally amazing, well done everyone!

Please give Jenkins X a try and let us know what you think or raise an issue. All feedback highly appreciated - particularly how we can keep improving to help you develop faster.

If you are at all interested in Continuous Delivery with kubernetes using tools like helm, helmfile, knative, lighthouse and last but definitely not least, tekton then please join the community - its great fun!

For any questions and feedback please reach out on slack https://jayex.io/community/#slack

Blog: Jenkins X 3 - April 2021 LTS

Mon, 12 Apr 2021 00:00:00 +0000

This is the second LTS release for Jenkins X 3.x.

LTS is a slower cadence version stream which contains a verified set of releases and configurations that have been used by teams tracking the bleeding edge Jenkins X.

Initially when we decided to maintain an LTS version stream we thought we’d aim for monthly releases however this second release comes two months after the first. This has given us more chances to run fixes and chart upgrades on Jenkins X own infrastructure to verify stability.

Note This LTS release is intended to be the final one before Jenkins X 3 is made Generally Available so stay tuned for the exciting news coming very soon! We will of course continue to develop and release LTS post GA.

Included in this release:

General beta bug fixes and helm chart upgrades
Enable Observability
Enable Slack notifications
Support for GitLab, Gitea, BitBucket Server and GitHub Enterprise

Please be aware of these changes

Breaking changes
If using Vault move it outside of being managed by Jenkins X GitOps important notes

Blog: Traces for your pipelines

Thu, 08 Apr 2021 00:00:00 +0000

Now that Jenkins X has solid integration with Grafana for its observability, it’s time to start building fun things!

And the first one is tracing for all your pipelines:

With it, you can easily see the timings of all your pipelines, stages, and steps. This is great to inspect a “slow” pipeline and quickly see the slower steps.

We are using OpenTelemetry to generate a “logical” view of the pipeline, with 1 trace per pipeline and 1 span for each stage and step.

By default, these traces are ingested by Grafana Tempo. But if you prefer to export them to a different destination, it’s very easy, and thanks to the OpenTelemetry Collector you can export to a lot of different services. You can see the full list here and here.

The trace identifier is also stored in the pipeline itself so that the Jenkins X Pipelines Visualizer UI can link directly to the trace.

How can you benefit from it in your own Jenkins X cluster?

You just need to enable the observability stack, as explained in the observability admin guide.

Then, trigger a pipeline, and once it’s finished, go to the web UI, and click on the “Trace” button on the top-right. That’s it!

What’s next?

This is only the first step of native tracing support in Jenkins X. Stay tuned for more!

Blog: Jenkins X v3: now with built-in observability

Thu, 01 Apr 2021 00:00:00 +0000

As a Continuous Delivery platform, Jenkins X has a central part in your infrastructure. If it becomes unstable or unusable, it will impact the whole software delivery of your organization.

This is why observability is a critical topic for Jenkins X, and work has started to get observability built-in for Jenkins X v3:

Platform Observability: visualize logs and metrics for everything running in the Kubernetes cluster: Jenkins X’s own components - Tekton, Lighthouse, cert-manager, … - but also your own applications, that will be deployed either in preview environments or in the staging/prod environments.
Continuous Delivery Indicators: visualize pull requests, pipelines, releases, and deployments metrics, collected from cluster events and git events.

We’re using Grafana as the central visualization component: the main entry point from which you can get a complete overview of both your application’s lifecycle - development, build, tests, releases, deployments, runtime - and your Continuous Delivery platform.

Platform Observability

Platform observability is not enabled by default for the moment, so the first step is to enable it, as explained in the platform observability admin guide.

Once it’s done, you’ll get a running Grafana instance, pre-configured with data sources for applications logs - using Loki - and applications metrics - using Prometheus. But most important, it comes with a set of pre-defined Grafana dashboards for the main platform components: Tekton, Lighthouse, cert-manager, …

Here is an example of such a dashboard, using a mix of data sources to display cert-manager metrics collected by Prometheus - including the certificates expiration dates - and logs collected by Loki/Promtail:

Continuous Delivery Indicators

Continuous Delivery Indicators’ main goal is to give people insights into their workflows/processes so that they can continuously improve them. This is based on the DORA devops metrics and the SPACE framework.

The CD Indicators addon is not enabled by default for the moment, so the first step is to enable it, as explained in the continuous delivery indicators admin guide.

Once it’s done, you’ll get a running collector, along with a PostgreSQL database. The collector will listen for various events, both from the cluster and the git repositories, and store pull requests, pipelines, releases, and deployments data in the PostgreSQL database. The addon will also expose a new Grafana data source along with pre-configured Grafana dashboards, which will be picked up by your running Grafana instance.

Here is an example of such a dashboard, displaying various indicators for a single repository/application: contributors, reviews, pull requests, releases, deployments, …

Roadmap

This is only the beginning! The next steps - in no particular order:

configure alerting - using Prometheus alertmanager and Grafana alerting features - with a set of pre-defined alerts
improve the dashboards
enable it by default, so that users can benefit from it out of the box

Contributions are welcomed:

Blog: Scaling Preview Environments with Osiris

Thu, 01 Apr 2021 00:00:00 +0000

One of Jenkins X’s core features is the preview environments: temporary environments created automatically for each Pull Requests, to deploy your application and its dependencies. You can then use this preview environment to run integration tests, or manually use/test your application.

This is all great until you have more and more applications, each with a few dependencies (postgresql, mongodb, …) and a few opened pull requests at any time. This means that you’ll get more and more pods running in your Kubernetes cluster, in addition to Jenkins X’s own components, your build pipelines, and of course your staging and production applications - unless you are using multi-cluster. The result is that you’ll need more nodes or bigger nodes. Which means more money.

But, these preview environments are in fact idle most of the time: they are only used for the integration tests, and sometimes when someone manually uses them. The rest of the time - including all night for example - they are just staying there, idle, and consuming resources. What if we could easily scale them down when they are idle, and automatically bring them up when we need them? So that a Pull Request staying opened for 2 weeks because someone went on vacation won’t consume resources in your cluster.

Osiris

Enter Osiris! Initially created by the Deislabs team, Osiris is a Kubernetes component that will automatically scale down your “idle” pods, and scale them up when a request comes in. Although the original project has been archived, the Dailymotion team has taken over the maintenance of a fork. And they have been using it with success in their Jenkins X dev cluster for more than 2 years: they regularly have around 50 preview environments active at any time, and… 0 pods from these environments running at night - or on weekends. Coupled with a cluster autoscaler, it means that their Kubernetes cluster use between 3 and more than 20 nodes depending on the workload. Being able to scale down to a minimum number of nodes is a great benefit when using cloud resources.

How can you benefit from it in your own Jenkins X cluster?

first, you’ll need to follow the admin guide to enable Osiris in your dev cluster
and then, you’ll need to add annotations to your Deployment/Statefulset and Service manifests - as explained in the Osiris documentation - in your application’s Helm chart

Note that if you store data, you’ll need to use persistent volumes (for postgresql, mongodb, …) so that you won’t lose your data after a scale down/up of your pods.

How does it work?

Osiris will automatically inject itself as a proxy in front of all the pods with the right annotation, so that it can see all requests for your pods. If it doesn’t see any request for a configurable amount of time - 10 minutes by default - it will scale down the deployment (or statefulset), and place itself behind the associated service. So that if a new request comes in, Osiris will be able to scale up the deployment (or statefulset), and then forward the request to the new pod.

Blog: cdCon 2021 - Call for Jenkins X Proposals

Thu, 25 Feb 2021 00:00:00 +0000

Hear ye! Hear ye! Jenkins X Community,

cdCon 2021 (the Continuous Delivery Foundation’s annual flagship event) is happening June 23-24 and its call for papers is open!

This is your chance to share what you’ve been doing with Jenkins X. Are you building something cool? Using it to solve real-world problems? Are you making things fast? Secure? Or maybe you’re a contributor and want to share what’s new. In all cases, we want to hear from you!

Submit your talk for cdCon 2021 to be part of the conversation driving the future of software delivery for technology teams, enterprise leadership, and open-source communities.

Submission Deadline: Friday, March 5 at 11:59 PM PST

Topics

Here are the suggested tracks:

Continuous Delivery Ecosystem – This track spans the entire Continuous Delivery ecosystem, from workflow orchestration, configuration management, testing, security, release automation, deployment strategies, developer experience, and more.
Advanced Delivery Techniques – For talks on the very cutting edge of continuous delivery and emerging technology, for example, progressive delivery, observability, and MLOps.
GitOps & Cloud-Native CD – Submit to this track for talks related to continuous delivery involving containers, Kubernetes, and cloud*native technologies. This includes GitOps, cloud-native CD pipelines, chatops, best practices, etc.
Continuous Delivery in Action – This track is for showcasing real-world continuous delivery addressing challenges in specific domains e.g. fintech, embedded, healthcare, retail, etc. Talks may cover topics such as governance, compliance, security, etc.
Leadership Track – Talks for leaders and decision-makers on topics such as measuring DevOps, build vs buy, scaling, culture, security, FinOps, and developer productivity.
Community Track – There is more to open source than code contributions. This track covers topics such as growing open source project communities, diversity & inclusion, measuring community health, project roadmaps, and any other topic around sustaining open source and open source communities.

Singular project focus and/or interoperability between:

Jenkins
Jenkins X
Ortelius
Spinnaker
Screwdriver
Tekton
Other – e.g. Keptn, Flagger, Argo, Flux

View all tracks and read CFP details here.

We look forward to reading your proposal!

Submit your talk

Blog: GitOps your cloud native pipelines

Thu, 25 Feb 2021 00:00:00 +0000

Tekton pipelines are cloud native and are designed from the ground up for kubernetes and the cloud:

there’s no single point of failure and the pipelines are elastically scalable
each pipeline is completely declarative and self defined
each pipeline executes independently of any others
pipelines are orchestrated via the sophisticated kubernetes scheduler:
- can use pipeline specific metadata for resource limits and node selectors: memory, CPU, machine type (GPU, windows/macOS/linux etc)
its easy to associate pipelines with Cloud IAM roles to avoid you having to upload cluster admin secrets to your public CI service which really helps security and helps reduce accidental bitcoin mining on your cloud account

In a previous blog we talked about how you can accelerate your use of tekton with Jenkins X.

The problem

We are moving towards a microservice kind of world with many teams writing many bits of software in many repositories. So there are lots and lots of pipelines. These pipelines keep getting more sophisticated over time; doing much more (all kinds of building, analysis, reporting, testing, ChatOps etc) and the software/images/approaches they use change.

So how can we manage, configure and maintain them all so that there are many pipelines for many repositories; where each repository can customise anything it needs but we can easily maintain everything continuously and its easy to understand and tool around?

Previous solutions

We’ve tried to tackle this problem in a number of ways over the years; each has pros and cons.

One option is to put all your pipelines in a shared library. You can then reference the pipelines by name in each of your repositories.

But what if you want to change a bit of a pipeline for a specific repository? If you change it globally for everyone you can break things. You may just want local customisation for your repository only.

You can add parameters into your pipelines. They are quite verbose on Pipelines and PipelineRuns; but it’s hard to think up front of every parameterisation that may be required by downstream repositories. e.g. changing any image; changing any command line argument in any step, adding/changing any environment variables or volumes? How about adding extra steps before/after a particular step? It can soon get very complex and results in very complex pipelines that are hard to understand and use.

Another option is if you need to change a pipeline file you just copy the entire file or create a fork. But then you end up with 100s of copies or forks of pipelines that are hard to synchronise and manage. You end using ancient image versions or older approaches in some repositories which leads to maintenance nightmare. How do you roll out security updates to images in all those repositories, copies and forks?

Another approach we tried is using a tool like kpt to share YAML files across git repositories and then upgrade them via git. This does work quite well; though the downside is whenever you upgrade a new version (e.g. we roll out a new pipeline catalog or a new image change to a tool for security reasons) you need to generate a pull request on every git repository to upgrade them and usually you end up with merge conflicts as the tekton YAML is not trivial; even fairly minor local customisations lead to merge conflict hell.

So how can you apply the benefits of GitOps to your cloud native pipelines while also avoiding copy-paste of lots of YAML into all of your repositories, keeping things easy to understand and flexible so any repository can customize things when required but at the same time make it painless to move reliably forward as the pipeline catalogs and images change?

GitOps your pipelines

Now our recommendation on the Jenkins X project is to use GitOps for your pipelines as well as for your source code and deployment configuration:

store your pipelines as declarative YAML files inside each of your git repositories.
use the standard Tekton YAML syntax so that you get IDE support and easy linting

This lets each git repository configure what pipelines are triggered by what events with what pipeline steps.

If you need to edit your pipelines in any repository they are right there in git; it is then easy for each repository to use its own version and configuration if required. This lets pipelines and repositories change over time independently to help you accelerate.

Rather than copy pasting task and step YAML between repositories we can refer to a Task or a Step in a Task as follows:

refer to all the steps in a shared task by using

taskSpec:
  steps:
  - image: uses:sourceURI

refer to a single named step from a shared task

  taskSpec:
    stepTemplate:
      image: uses:sourceURI
    steps:
    - name: mystep

SourceURI notation

The source URI notation is enabled by a special image prefix of uses: on step or if an image on a step is blank and the stepTemplate: has an image prefix of uses:

We borrowed this idea from ko and mink; the idea of using a custom prefix on image URIs.

You can refer to the detailed documentation on how the step inheritence and overriding works.

For a github.com source URI we use the syntax:

- image: uses:owner/repository/pathToFile@version

This references the https://github.com repository for owner/repository and @version can be a git tag, branch or SHA.

If you are not using github.com to host your git repositories you can access a pipeline task or step from your custom git serve use the uses:lighthouse: prefix before owner:

- image: uses:lighthouse:owner/repository/pathToFile@version

We recommend you version everything with GitOps so you know exactly what versions are being used from git.

However you can use @HEAD to reference the latest version.

To use a locked down version based on the version stream of your cluster, you can use @versionStream which means use the git SHA for the repository which is configured in the version stream.

The nice thing about @versionStream is that the pipeline catalog you inherit tasks and steps from is locked down to an exact SHA in the version stream; but it avoids you having to go through every one of your git repositories whenever you upgrade a pipeline catalog.

Reusing Tasks and Steps from Tekton Catalog

The Tekton Catalog git repository defines a ton of Tekton pipelines you can reuse in your pipelines

You can image: uses:sourceURI notation inside any pipeline file in your .lighthouse/jenkins-x/mypipeline.yaml file like this:

steps:
  - image: uses:tektoncd/catalog/task/git-clone/0.2/git-clone.yaml@HEAD

This will then include the steps from the git-clone.yaml file

It’s not just the Tekton Catalog - you can use this same approach to reuse Tasks or steps from any git repository of your choosing; such as the Jenkins X Pipeline catalog

How it looks

So here is an example release pipeline generated via the Jenkins X Pipeline catalog if you create a JavaScript quickstart

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: release
spec:
  pipelineSpec:
    tasks:
    - name: from-build-pack
      taskSpec:
        stepTemplate:
          env:
          - name: NPM_CONFIG_USERCONFIG
            value: /tekton/home/npm/.npmrc
          image: uses:jenkins-x/jx3-pipeline-catalog/tasks/javascript/release.yaml@versionStream
          name: ""
          resources:
            requests:
              cpu: 400m
              memory: 512Mi
          volumeMounts:
          - mountPath: /tekton/home/npm
            name: npmrc
          workingDir: /workspace/source
        steps:
        - image: uses:jenkins-x/jx3-pipeline-catalog/tasks/git-clone/git-clone.yaml@versionStream
          name: ""
        - name: next-version
        - name: jx-variables
        - name: build-npm-install
        - name: build-npm-test
        - name: check-registry
        - name: build-container-build
        - name: promote-changelog
        - name: promote-helm-release
        - name: promote-jx-promote
        volumes:
        - name: npmrc
          secret:
            optional: true
            secretName: npmrc
  serviceAccountName: tekton-bot
  timeout: 240h0m0s

You can see it mounts an npm secret for using npm package management and specifies CPU and memory requirements. It then is using the uses: notation to inherit a bunch of steps from the jenkins-x/jx3-pipeline-catalog/tasks/javascript/release.yaml as well as sharing the jenkins-x/jx3-pipeline-catalog/tasks/git-clone/git-clone.yaml task

Also notice we don’t have to copy and paste the exact details of the images, commands, arguments, environment variables and volume mounts required for each step; we can just reference them via Git. Also each pipeline in each repository can reference different versions if required.

Customizing an inherited step

You can edit the step in your IDE and add any custom properties such as command, args, env, script or volumeMount - those values then override the inherited step.

e.g. you can then change any command line, add an environment variable or add a new volume mount without copy pasting the whole step. e.g. we change the script value of the jx-variables step below:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
spec:
  pipelineSpec:
    tasks:
    - taskSpec:
        stepTemplate:
          image: uses:jenkins-x/jx3-pipeline-catalog/tasks/javascript/release.yaml@versionStream
        steps:
        - name: jx-variables
          script: |
            #!/usr/bin/env sh
            echo my replacement command script goes here

Any extra properties in the steps are used to override the underlying uses step.

Inlining a pipeline step locally

If you want to edit a step that is inherited from a pipeline catalog just run the jx pipeline override command from a clone of your repository.

jx pipeline override

This will then prompt you to pick which pipeline and step that’s inherited via the image: uses:sourceURI notation. When chosen the step will be inlined into your local file so you can edit any of the properties.

You can use the git compare to see the changes and remove any properties you don’t wish to override.

Viewing the effective pipeline

To see the actual Tekton pipeline that would be executed from your local source directory you can run the jx pipeline effective command:

jx pipeline effective

If you want to open the effective pipeline in your editor, such as VS Code you can do:

jx pipeline effective -e code

If you use Intellij or any of JetBrains other IDEs you can do the following if you have enabled the idea command line tool:

jx pipeline effective -e idea

If you want to always view an effective pipeline in your editor then define the JX_EDITOR environment variable…

export JX_EDITOR="code"

# now we will always open effective pipelines inside VS Code
jx pipeline effective

Summary

We’ve been on our own digital transformation journey in the world of pipelines and used many different approaches over the years to manage many pipelines across many repositories.

A few months ago we moved to the above GitOps approach for our cloud native pipelines and we are absolutely loving it!

Its super easy to:

share pipelines across all of your git repositories without copy/paste
easily customise pipelines in any project and be able to easily understand what the local changes are and roll them back if required
upgrade pipelines across your repositories in a consistent way as you upgrade your images, applications and cluster via GitOps so that new versions of pipeline catalogs are upgraded once they pass the system tests.

If you are thinking about using cloud native pipelines with Tekton please try it out and see what you think. We’d love to hear your feedback

Blog: Jenkins X 3 - February 2021 LTS

Mon, 01 Feb 2021 00:00:00 +0000

This is the first LTS release for Jenkins X 3.x. We are still in the Beta release and the leadup to GA includes ensuring the process for LTS monthly releases is validated and working well. This first releases focuses on:

community feedback following the Beta release
general helm chart upgrades
improved developer UX when editing Tekton pipelines here
support for in-repo and shared Tekton pipeline libraries including git URI support, e.g.

uses:jenkins-x/jx3-pipeline-catalog/packs/javascript/.lighthouse/jenkins-x/pullrequest.yaml@v1.2.3`

more documentation and examples can be found here

What’s the difference?

Because Jenkins X uses GitOps we can see the git diff of changes that will be brought in with a cluster upgrade. Here is the Pull Request that has been verified for February LTS release. https://github.com/jenkins-x/jx3-lts-versions/pull/209/files

Anything else to be aware of?

Included in the release is a switch of the NGINX Helm chart from the old Helm stable registry. It was discussed on the community slack that some users on EKS and not using a custom domain had to change the domain in their cluster jx-requirements.yml file.

The change of Chart repository meant the old resources were removed and new ones added, resulting in a new Kubernetes LoadBalancer service was created, resulting in a new external IP address. You may need to update the domain in your jx-requirements.yml. To get the external IP run:

kubectl get service -n nginx

Note the external ip address and update your cluster git repository, jx-requirements.yml:

domain: $EXTERNAL_IP_FROM_ABOVE.nip.io

git commit
git push
jx admin logs

And you should see the new ingress rules created when the boot job finishes:

kubectl get ing -n jx

What’s next?

March LTS release aims to switch the incluster boot jobs to use https://carvel.dev/kapp/ rather than using kubectl to apply and prune Kubernetes resources. kapp is a dependency aware approach to installing Kubernetes resources, this aims to help avoid timing issues when installing resources before certain services like cert-manager are fully up and running.

Blog: Jenkins X 3.x walkthroughs

Tue, 26 Jan 2021 00:00:00 +0000

Jenkins X 3.x is now looking ahead towards a GA release, with that we are producing walkthroughs for key areas to help users not only get started but get the most out of Jenkins X.

To kick this off we are going to start with 9 videos that we’ll follow up with more dedicated blogs over the coming weeks. The complete playlist can be found here however the blog below gives a more context for each one.

There are a few key areas we are focusing on here:

Intro + high level architecture

Starting off with a very quick introduction including what to expect from the walkthrough series.

Jenkins X 3.x has focussed on clearer lines of separation, making the architecture significantly more pluggable, extensible and maintainable. With better tooling including UIs and more reliable guard rails for installations and upgrades. Jenkins X 3 also minimises abstractions and wrapping; so it promotes the direct use of open source projects like Helm, Helmfile and Tekton.

Installation and setup

Infrastructure and provisioning

Decoupling the management of Cloud infrastructure away from Jenkins X to tools that are better suited for the job. Jenkins X has started with Terraform and this manages all the cloud resources needed by Jenkins X

Kubernetes cluster
Cloud Service Accounts
IAM bindings
Storage buckets

Over time Jenkins X plans to support other tools (aided by the Kubernetes Cluster API) users in the Kubernetes ecosystem leverage such as crossplane.io, Google Config Connector, AWS Controller etc. These make use of cloud resources declared as custom resources with a Kubernetes operator managing CRUD activities for them.

Once the cloud infrastructure is created, self provisioning happens using GitOps and incluster installation. No more flakey client side installs.

TLS and DNS

A large number of deployments require inbound traffic, whether that is for access to their applications, websites, REST endpoints, Webhook handlers there is often a need for Ingress HTTP traffic into a cluster and to ensure communication is secure.

To achieve this there are two common efforts needed

DNS - configuring a custom owned domain and using DNS to route traffic to endpoints TLS - providing end to end security for web traffic on the internet

In fact many web services do not accept working with insecure endpoints and others require manual override to accept the risk before being able to use the service.

Jenkins X uses two OSS projects to automatically manage DNS (External DNS) and handle the management of TLS certificates (Cert-manager)

Once a domain is owned, External DNS will work with cloud providers to create A records that route traffic from the internet to users clusters. Cert-manager will react to a request from Jenkins X to verify a cluster owns a domain and will issue a wildcard TLS certificate using Lets Encrypt that is used for all Ingress into the cluster. Cert-manager will also handle certificate renewals. This is all handled automatically following the setup of Jenkins X.

Using Jenkins X

GitOps

Initially started out as an implementation detail of Jenkins X but has evolved much more into an administrators typical workflow. Managing installs, upgrades and rollbacks via Git provides approvals, reviews, traceability, RBAC in the same way we manage code. This is the backbone of Jenkins X and provides us with the peace of mind for disaster recovery.

Jenkins X ships with a git operator which is responsible for applying generated Kubernetes resources which live in the cluster Git repository. Every application and configuration for the cluster is in this repository.

Health

In any Kubernetes installation there can be a lot of microservices each with the responsibility to provide functionality that is needed by the overall system. Understanding when things go wrong and the impact of these issues can be difficult to evaluate. It is also useful to have a status page of sorts to quickly check the health of your system.

Jenkins X uses Kuberhealthy and a lot of custom health checks to periodically report on the health of a Jenkins X installation. These custom health checks are easy to extend in any language, already the Jenkins X community has been contributing their own.

There is a CLI which can be used to query the health report as well and a UI.

The health statuses can be easily integrated into users operational alerting systems too.

Extending pipelines

Jenkins X uses vanilla Tekton pipeline resources and has deprecated the v2 jenkins-x.yml.

We will show a demo shortly of working with Tekton pipelines and inheriting shared Tasks but for now we can see it is easy using Lighthouse to trigger shared Pipelines from a git repository. The demo uses the old favorite from the Jenkins project Chuck Norris, only here we are invoking a cloud native pipeline to comment with the joke on a Pull Request.

Secrets

Secrets management stores (e.g. AWS Secrets Manager, Hashicorp Vault, Google Secrets Manager) have gained popularity from both Administrators and Developers. Having a single source of truth for a secret is extremely useful especially when obtaining and changing values, some solutions even offer automatic secret rotation.

With GitOps as described above we need a way to inject real secrets into a cluster rather than storing them in Git.

Jenkins X uses External Secrets from which runs as a Kubernetes controller, using a Kubernetes Custom Resource it knows how to automatically map values from a Secret Management Store into Kubernetes Secrets. This makes it easy for applications to leverage GitOps while keeping the benefits of using a secrets management solution.

Upgrades

Jenkins X now uses GitOps for the whole cluster, one repository keeps all applications and configurations that should be applied to a cluster. This cluster repository also includes a copy of the version stream mentioned below. Jenkins X uses a tool called kpt which reliable syncrosises released version configuration into their own repository that can then be committed and applied to your cluster.

Version Streams

The world of continuous delivery can bring challenges. One of the biggest challenges Jenkins X itself had while embracing CD was how people handle receiving constant change via releases.

Some people wanted to live on the bleeding edge, receiving fixes, improvements and new features as fast as possible so they can help provide feedback and continually improve. Others had more stable requirements which expect more mature features along with corresponding complete documentation that gave better confidence an upgrade will not cause any adverse effects.

Jenkins X has had the concept of version streams which allows the project to collect a number of helm charts, CLI, Docker image version changes together, run further automated testing and release together for users to consume in an upgrade. This acts as a quality gate.

For Jenkins X 3 we have extended this to include a second version stream which gets automatically updated via a Pull Request however that is merged on a slower cadence to cater for users that want greater confidence in the release. We call this the Long Term Support (LTS) version stream.

Users can decide which version stream to track or even use a custom one that they maintain themselves.

Cluster recovery

Clusters can be recreated if they move regions, are accidentally deleted or part of intentional housekeeping to continually verify disaster recovery processes.

Because Jenkins X uses Terraform for infrastructure checked into Git and a another Git repository specifically for the cluster, it means we can recreate a cluster and resume all services with very little manual intervention. This video deletes and recovers a cluster on GCP using Jenkins X.

Blog: New features in the pipelines visualizer UI

Mon, 18 Jan 2021 00:00:00 +0000

The Jenkins X Pipelines Visualizer UI has recently received a number of new features, so let’s do a little tour of these new features!

Pipeline View

When viewing a pipeline, the biggest new feature is the collapsed logs. No more hundreds - or thousands - of log lines, we now group the logs per-container (step), which are collapsed by default. Along with the status of the step and its duration, so it’s easier to go to the interesting part of the logs.

Clicking on a log line will expand the logs for this specific container. You can also use the “Toggle Steps” button to expand/collapse the logs for all the steps at once.

While we’re talking about the logs, you can notice the 2 new buttons:

View raw logs
Download raw logs

On top of the logs, we now display some information about the pipeline:

the pipeline meta information: name, context, build, and a link to see the raw YAML representation of the pipeline
the pipeline status: status, started/finished date/time, and duration
the pipeline source: git repository, pull request or branch, commit SHA, author
the pipeline stages, with links to see the timeline of the steps in each stage. You can also click on the “Show Timeline” button to view the pipeline timeline with all stages and steps.

The pipeline timeline has been improved to include all the steps for all stages, but it is currently hidden by default - to avoid using too much space. Clicking on a stage will bring you to the steps, and clicking on a step will bring you to the logs for this step.

Note that for a pipeline which includes a deployment to a Preview Environment, the UI will also display a link to the application’s URL in that specific Preview Environment.

Homepage

The homepage got some love too, with:

a few stats about the pipelines: top statuses, repositories, authors and durations - with links to filter the pipelines
direct links to the git repositories and pull requests
the Jenkins X logo
and a favicon

Roadmap

We started this project at v0, and we believe that now it has enough features to be a v1!

On our roadmap - without any specific order - we have:

#73 live refresh of a running pipeline - for now only the logs are updated live, not the meta information of the pipeline (status, stages/steps timings)
#42 support local timezone - for now everything is in UTC
improve the support for archived pipelines: load pipelines archived in the long-term storage

Contributing

Thanks to all the contributors!

All contributions are welcomed, the source code is: github.com/jenkins-x/jx-pipelines-visualizer.

Blog: Jenkins X 3.x beta is here!

Wed, 09 Dec 2020 00:00:00 +0000

I’m super excited to announce the 3.0 beta of Jenkins X! Christmas has come early this year!

the main documentation of the changes are:

the new architecture with modular plugins and improved extension points
what has changed since 3.x started
how 3.x compares to 2.x

but here’s a brief summary of the differences:

User Changes

As a user the high level UX of Jenkins X is similar:

automated Continuous Delivery pipelines for using tekton for your repositories with automatic promotion between your environments
pull requests on your repositories create separate Preview Environments where your team can review your changes and give fast feedback before your changes are approved and merged into the main trunk.

New features

we now default to vanilla tekton YAML for defining pipelines while accelerating your tekton with tekton catalog
we include an open source dashboard for visualising pipelines and logs which you can invoke via:

jx dash

Platform Changes

we now use helm (3.x) and helmfile along with optionally kustomize in a GitOps style to define and configure both Jenkins X itself, your tools and applications in any namespace
support multi cluster out of the box so you can keep Staging and Production in separate clusters to your development cluster where your pipelines run, you create and release immutable container images and other artifacts.
to setup or upgrade Jenkins X we use terraform to setup your cloud resources on Azure, Amazon or Google while also supporting on-premises, minkube and OpenShift - see the Admin Guides for more detail
- the actual installation of kubernetes resources takes place using the git operator so it runs reliably inside the cluster itself
we default to using Kubernetes External Secrets to manage all secrets for Jenkins X itself, development tools and your applications too.
- This means we can support various secret backends such as Alibaba Cloud KMS Secret Manager, Amazon Secret Manager, Azure Key Vault, Hashicorp Vault or GCP Secret Manager
- It also means we can then check in all kubernetes resources and custom resources directly into git (apart from Kubernetes Secrets) so that it super easy to version, review and reason about your kubernetes resources in a GitOps way.
built in TLS and DNS support along with Heath reporting and visualising via kuberhealthy
we now have an LTS distribution which lets you switch to a much more slower cadence of releases of Jenkins X

In general Jenkins X 3.x is now much simpler and more flexible. It supports lots more platforms than before and should be easy to extend and configure for other platforms too.

Getting started

If you have never tried 3.x before then please follow the Admin Guide to get Jenkins X installed on your cloud provider, on-premises kubernetes cluster or minikube.

If you previously tried the 3.x alpha then the migration instructions are here.

For folks on older 2.x versions of Jenkins X please see the 2.x migration instructions

Once your cluster has been installed or migrated then check out the User Guide on how to develop software continuously with Jenkins X.

Final thoughts

Please give Jenkins X a try and let us know what you think or raise an issue. All feedback highly appreciated - particularly how we can keep improving to help you develop faster.

I hope you have a great break over the holiday season and 2021 is a little better and more fun than 2020!

For any questions and feedback please reach out on slack https://jayex.io/community/#slack

Blog: Jenkins X 3.x - beta is close!

Fri, 04 Dec 2020 00:00:00 +0000

It has been ‘all hands on deck’ in recent months with the focus on Jenkins X 3 alpha. First off a huge thankyou to everyone involved. The OSS community spirit has really shone through what has been a very difficult year for everyone. Knowing that people from all over the world come and help each other, share banter and work at all hours of the day to help build out a true open source cloud native continuous delivery solution for developers - it’s quite fantastic to see and amazing to be apart of.

As a result of all this hard work the Beta is iminent so this is a good opportunity to thank all involved so far and to outline what to expect in the coming days.

While we’ve been in the Alpha phase it has provided us with the opportunity to deprecate and remove APIs, commands and obsolete features that existed in v2. This means we will not have any code dependency on the v2 codebase and so going forward v3 will be easier to maintain without the tech debt.

With that, we aim to make a big push and roll out a few last changes in preparation for Beta, here’s a couple you will notice if you are already on the Alpha. We recommend taking time to understand these, and avoid upgrading for a few days so that changes can be handled in one go, as there will be a constant stream of larger updates happening:

jx requirements - this is the yaml file used to describe install needs for Jenkins X, until now there have been options available that were unsupported, confusing and in some cases did nothing. These have now all been removed and the structure of the file has changed to be CRD like including an API version. Upon upgrade jx gitops upgrade will migrate your jx-requirements.yml and push the changes back to the cluster gitops repository. You should not see any errors however there are some pipeline steps that need to be updated in order to work with the new structure, this is covered next.
pipeline catalog - currently Jenkins X 3.x defaults to in-repo pipelines where your Tekton resources are managed by kpt, these will need to be updated to work with the new or old requirements described above. This means you will also need to run jx gitops upgrade in each your application git repos. In the near future Jenkins X 3.x will make it easy to work with shared pipeline libraries / catalog repositories, referencing them via URL rather than in-repo. This is so that users have the flexibility to choose which option is best for your use case. In-repo pipelines and shared pipeline libraries have pros and cons for each, mainly about maintaining changes. But for now, you will need to upgrade each application git repository.
nested helmfiles - this is a great feature from Chris Mellard which will be merged very soon. The idea is that your cluster git repository will have support for multiple helmfiles, ones that will contain core Jenkins X charts and maintained by kpt via the jx gitops upgrade process, and others where you can add your own charts rather than extending the single helmfile that is there today.
nginx - for the alpha we have upgraded all the core helm charts we ship in a base Jenkins X installation, the last to land is Nginx thanks to Ankit! This is not expected to cause issues however the change does involve moving to a different chart, there is a risk that there may be some very small window of downtime for the ingress controller so webhooks fired during the upgrade may fail to be delivered.

Balancing upgrades with continuous delivery brings many challenges. However with Kubernetes, GitOps, Jenkins X version streams (including the monthly LTS coming soon) and other tools like kpt makes the process more transparent and provides a greater level of high availability.

Jenkins X automatically upgrades its own infrastructure with every version stream release and we haven’t experienced any disruption during the alpha period so far. We aim to continue this, but when we know there will be disruption we endeavour to inform and explain why ahead of time.

Keep an eye out for announcements of the Beta and thanks again to all involved in the community and we are very much looking forward to welcoming new people too. We are very excited for the Beta release and given the feedback so far we understand users are too.

For any questions and feedback please reach out on slack https://jayex.io/community/#slack

Blog: Accelerate your Tekton with Jenkins X

Wed, 11 Nov 2020 00:00:00 +0000

WARNING: This post refers to the unmaintained Jenkins X version 1 or 2

This post is outdated and is kept only out of historical interest. Some broken links have been removed.

One of the goals of Jenkins X has always been to help accelerate and automate Continuous Delivery so that developers can focus on delivering value to their customers; either by creating that new microservice or adding features to an existing project and not writing and managing pipelines.

Pipeline engines like Jenkins and Tekton are awesome - they can do anything! But they start as a blank sheet of paper where you have to fill in all the details of how to compile your code, test it, verify it, tag it, release, distribute and delivery it to production. Figuring all that stuff out can take a huge amount of time to create and maintain. This gets even more complex as we are all creating more and more microservices each with their own pipelines making more and more things to create and manage.

We want to be able to reuse pipelines and tasks to get work done. But at the same time we want flexibility; not all applications are the same and sometimes things need to be changed on a per team or application basis.

Version 2.x

In Jenkins X 2.x we went with a jenkins-x.xml approach to pipelines which let you inherit pipelines from reuable pipeline library and then use a composition DSL above Tekton which lets you add/remove/replace steps.

e.g. to use the javascript pipeline library but override a step you could use:

buildPack: javascript
pipelineConfig:
  pipelines:
    overrides:
      - pipeline: release
        name: helm-release
        step: 
          image: busybox
          sh: echo "this command is replaced"

This was a pretty good approach; it lets us reuse common pipelines in a shared git repository and let’s reuse a composition DSL.

However we’ve found that this approach as a few downsides:

we have to create and maintain a DSL above Tekton which adds complexity and can be a leaky abstraction
- e.g. the DSL does not yet support all of the semantics of Tekton yet such as conditions, runAfter or finally tasks
- tekton moves fast; it’s hard to keep up in a DSL ;)
its complex trying to understand how to make local modifications of pipelines
- particularly if you just want to add an environment variable; modify a command line argument or something
we can’t use IDE tooling for Tekton to edit/visualise pipelines
we can’t reuse Tekton Catalog tasks

Vision

We want developers to have reusable pipelines for all their applications lifecycles:

continuous integration
verification, testing, linting
releasing, packaging, promoting, deploying

We also want to:

automate the generation of pipelines so most of the time developers don’t need to care about pipelines
reuse pipelines across applications
- usually all, say, spring boot microservices tend to be built and released in the same way
make it easy for developers to view/edit the pipelines if required
- maintain those changes over time as everything in the cloud native ecosystem is changing all the time

From a technical perspective:

we believe Tekton is currently the best cloud native standard way to represent pipelines and tasks for Continuous Delivery and we want that to be the primary DSL for developers and tools
we want to work with the Tekton Catalog so its easy to share tasks among teams

Version 3.x

For Jenkins X 3.x we really wanted to move closer to this vision: to accelerate the adoption of Tekton and help give developers Tekton super powers.

Re-reading Brian Grant’s document on Declarative application management in Kubernetes really got us thinking about this problem of how to reuse complex YAML files for pipelines and tasks while also allowing local modifications while also avoiding a complex leaky DSL for composition.

Then we tried out kpt (pronounced kept) and everything fell into place pretty quickly.

Using Tekton in your repository

When you create a new quickstart or import a repository into Jenkins X 3.x you get a new folder: .lighthouse/jenkins-x added to your source code which contains the Tekton pipeline files you need for your application.

So for a typical application the .lighthouse/jenkins-x folder will contain:

release.yaml the Tekton PipelineRun for releasing your application
pullrequest.yaml the Tekton PipelineRun for perform continuous integration testing, verification and the creation of a Preview Environment for your proposed changes before they merge to the main branch
triggers.yaml to define the lighthouse TriggerConfig which defines the ChatOps and triggering configuration via a spec field which defines presubmits and postsubmits (i.e. Pull Request and Release triggers).

See the Pipeline Catalog documentation for more details on how this works and go to the reference guide if you want to dive into the details.

As a developer you can mostly ignore the .lighthouse folder if you don’t care about how the pipelines work. If you are interested you can look inside.

If you need to modify anything, just open the Tekton files in your IDE and modify them. No complex DSL to understand other than Tekton itself. Then the changes will be used when you submit your local changes via a Pull Request (for the pull request pipeline) or they get merged to the main branch (for release pipeline changes).

To handle change going forward from upstream pipeline catalogs while preserving any local modifications we use a generic update mechanism on all git repositories which is powered by kpt

Reusing Tekton Catalog Tasks

The Tekton Catalog contains a ton of reusable Tekton Tasks for doing all kinds of things in the Continuous Delivery landscape with a variety of tools.

We want to make it super easy for you to reuse any of them easily in your project.

So now the new jx pipeline import command can be used to import Task resources from the Tekton Catalog so you can use them inside your project.

Here’s a demo of this in action:

The tekton Task resources are copied into your .lighthouse directory in a folder using kpt so that you can modify things locally if you need to and can upgrade your local copy with upstream changes.

This lets you work with shared resources from the Tekton community and, when required, modify them to suit and manage them easily over time.

Tekton makes it super easy to share Task resources between different Pipeline instances. Though there is a current limitation where splitting a Pipeline into multiple reusable Task instances results in the pipeline being split among multiple Pod resources; which means to share state between the Tasks you need to use a Persistent Volume for each pipeline run which can be a bit of an overhead.

For example: you may think it’s a nice idea to have a reusable Task to git clone your source code then use it with your other Task to run your tests. It turns out that can be quite expensive infrastructure wise; as it means your cluster will end up making a Persistent Volume for each pipeline invocation so that the git clone pod can clone git and store the state on the PV so that your real Task pod can start and mount the same volume to see the contents of git. Its much easier to just share the git clone steps in each Task; so that there’s no need for the PV; just git clone in each separate Task directly.

So for cases where you want to reuse a collection of steps inside Task resources we added an annotation in lighthouse so that we can import steps from a URL to avoid the copy/paste.

e.g. in our pipeline catalog we use this approach to share the git clone Task steps such as this example:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  annotations:        
    # lets share the git clone tasks as the initial steps in this Task
    lighthouse.jenkins-x.io/prependStepsURL: https://raw.githubusercontent.com/jenkins-x/jx3-pipeline-catalog/005e78cf69b643862344397a635736a51dd1bd89/tasks/git-clone/git-clone.yaml
spec:
  ...

Hopefully we can migrate to a standard tekton based approach if this issue is resolved.

Custom Pipeline Catalogs

Tekton Catalog is an awesome way to reuse Tasks but it doesn’t help when trying to reuse complete PipelineRun and Pipeline resources across projects and repositories while also being able to modify them as needed on a per team or repository basis.

Jenkins X 3.x comes with its own default pipeline catalog for different languages, tools and frameworks. This catalog contains reusable steps, Tasks and Pipelines you can use on any project.

It’s easy for you to fork this catalog to make changes for your team or share between teams in your company. You can make as many catalogs as you like and put whichever catalogs you want in the extensions/pipeline-catalogs.yaml file of your cluster git repository of your Jenkins X 3.x install. For more detail there’s the configuration reference here.

Then when developers create a new quickstart or import a repository developers will be asked to pick the catalog they want from your list if there is more than one, or the configured catalog is silently used.

This gives you complete freedom to configure things at a global, team or repository level while also making it easy to share changes across projects, teams and companies.

Conclusion

We are super excited about the combination of Jenkins X 3.x, Tekton, Tekton Catalog and ChatOps.

We hope that you can use the above capabilities to solve your Continuous Delivery needs and over time you can take your Pipelines and make your own Pipeline Catalogs and share them with folks inside and outside of your company.

Hopefully this can help us all accelerate our Tekton pipelines and catalogs towards more continuous delivery awesome with flexible reusable tasks and pipelines! If you want to give this a try check out Jenkins X 3.x

Blog: Hacktoberfest

Wed, 23 Sep 2020 00:00:00 +0000

We are excited to announce that Jenkins X will be participating in Hacktoberfest again this year! Hacktoberfest is a month-long global celebration of open source software.

From October 1 to October 31, submit four pull requests to qualify for the limited edition Hacktoberfest shirt. All backgrounds and skill levels are encouraged to participate in Hacktoberfest and join a global community of open source contributors.

Learn more about Hacktoberfest and sign up here.

Contribute to Jenkins X

We welcome your contributions to the Jenkins X project!

Issues labelled “hacktoberfest” generally indicate good first issues. However, all pull requests will count towards your Hacktoberfest challenge. Jenkins X welcomes contributors to both:

Contribute to `jx` source code

There are plenty of open issues, and we welcome your help in making Jenkins X even more awesome.

Jenkins X is written largely in Go, but you don’t need to be an expert to contribute! If you are new to the project, search for issues labelled “good-first-issue”. Our Contributing Guide has advice for getting started with contributing to Jenkins X.

Contribute to the docs

We welcome your help in improving the Jenkins X documenation. If you see areas of the documentation that need fixing or augmentation please raise a pull request. Our guide for Contributing to the Documentation has advice for getting started with contributing to the Jenkins X docs.

Ask us questions

We’re happy to help if you have any questions. Talk to us on our slack channels, which are part of the Kubernetes slack. Join Kubernetes slack here and find us on our channels:

#jenkins-x-dev for developers of Jenkins X
#jenkins-x-user for users of Jenkins X

We also have online office hours, where we talk about new developments in Jenkins X and you are welcome to ask us questions. We meet for office hours every other Tuesday at 15:00 UTC (See your timezone here).

Next office hours are on 6 October. Join us on this meeting link. See the Jenkins X calendar for events that you are welcome to join.

Find out more about becoming involved in the Jenkins X community here.

We look forward to seeing you in open source, fixing all the things!

image by Ashley McNamara, creative commons license

Blog: New UI to visualize your pipelines and logs

Wed, 23 Sep 2020 00:00:00 +0000

Welcome to the Jenkins X Pipelines Visualizer: a new open-source read-only UI for Jenkins X, with a very specific goal and scope: visualize the pipelines and logs.

This project was started at Dailymotion and quickly shared with the Jenkins X community.

Why a new UI?

There is already the Octant-based UI, so why a new UI?

The main reason is that Octant “is an application and is intended as a single client tool and at this time there are no plans to support hosted versions of Octant” - see this thread on the Octant github repository for more information and details.

So while Octant answers to a lot of use-cases, there is one for which it is not suited: quickly printing the build logs on a browser, for a specific pipeline. We want to be able to click on a link from a Pull/Merge Request, and get the pipeline logs. This is the specific use-case covered by the Pipelines Visualizer.

Features

We want to keep it small, focused, and fast. It’s a read-only UI, so there won’t be “actions” to trigger a pipeline - because it can already be done using “chatops” commands in the Pull Request for example.

But there are a few interesting features already:

first, it’s very fast to get the logs. Much faster than the old JXUI.
it can retrieve the logs from pipelines that have been garbage-collected - if you configure the URL of the buckets where the logs are stored.
it has URLs compatible with the old JXUI - so it’s very easy to replace the old JXUI with this new UI and keep all the links working.

Roadmap

This project was shared very early with the community, after just a few hours of work. So our short-term goal is to improve the UI - make it beautiful.

Demo

We did a demo of jx-pipelines-visualizer at the last office hours:

Next steps

Check out the jx-pipelines-visualizer github repository if you want to install it in your cluster - there is a Helm Chart which can be added to your Jenkins X Dev Environment.

And any contributions are welcomed - either create an issue or pull request in the project’s github repository, or come in the #jenkins-x-dev Slack Channel.

Value
image.parentRepository	ghcr.io/jenkins-x
image.repository	ghcr.io/jenkins-x/lighthouse
env.JX_DEFAULT_IMAGE	ghcr.io/jenkins-x/builder-maven:2.1.155-778-patch3