If you have come to read this article, you probably already know what Continuous Integration (CI) means, and how projects can benefit tremendously from adopting this practice.
But if you don’t, here’s a quick explainer: with CI we refer to the practice of automating the integration of code changes from different contributors, possibly several times a day. Given this definition, one might think that simply relying on a distributed version-control system like git should be enough, but that’s not quite the case. Indeed, the version-control system is just a piece of the puzzle, and the key here is integration.
Frequent changes can easily break things, and if they are integrated carelessly, they can lead to disastrous results. Of course, one could always ensure that things are working as expected by checking manually, but that would require time (inevitably halting the integration process), which in turn would defy the concept of continuous.
That is why, to truly achieve CI, we need to rely on additional tools that can help guaranteeing safeness and correctness automatically.
In this article we will explore exactly that, by re-creating a simplified version of the CI infrastructure that we have put in place for the PYNQ project. If you follow through the steps, by the end you should be able to automate the building of the PYNQ v2.5 source distribution. To achieve that, we are also going to build all the FPGA binaries (bitstreams and microblaze executables) calling Xilinx tools (in particular Vivado) inside Jenkins.
But first, let’s take a quick overview of the PYNQ CI infrastructure and the underlying technologies.
The PYNQ CI infrastructure: Jenkins & Docker FTW!
To build our CI infrastructure, we have relied on a few key technologies.
Here’s a breakdown:
- Jenkins: the tool we use to define, run and manage our CI jobs. Jenkins architecture is “Master+Agent” (as explained here), where the Master is in charge of orchestration to act as the user end-point, and the Agents perform the actual work. These agents can be distributed across many servers, and even be spawned on-demand as ephemeral containers, thanks to the Jenkins plugin that provides integration with Docker.
- Docker: it is the supporting technology for the whole infrastructure. Everything runs as a Docker container, even the Jenkins Master itself. Portability and ease of scalability are the main reasons why we chose to put Docker into the mix. Containers are taking over the (server) world, and it would be surprising if you did not hear about them at least once in the last years. However, if that’s the case (and even if you need a good refresher), you are encouraged to read more about containers and the underlying concepts in the Docker Getting Started Guide.
- Docker Swarm: our CI infrastructure consists of a cluster of machines onto which we deploy our build jobs. Swarm allows for seamless deployment of containers on any of these machines, in a transparent way. It is responsible for load-balancing and tracking of the status of all running containers, across all the machines that have joined the Swarm.
Hands-on: setting up your CI system
In this section, we are going to create a simplified version of the PYNQ CI infrastructure, relying on the same technologies presented earlier. In order to be able to reproduce what is going to be shown here, you will need to meet the following requirements:
- Have one or more machines available that can reach each other on the network, with Ubuntu or RHEL/CentOS as operating system. One machine is actually sufficient, but if you have more available, it is recommended to use them if you want to make proper usage of Docker Swarm.
- On each machine, you will need to install Docker. To do so, simply follow the instructions available on the official documentation.
- Similarly, on each machine that you are going to use you will need to install Vivado, specifically the 2019.1 version. Instead of including Vivado directly into the Docker images for our Jenkins Agents, for our configuration we chose to install Vivado on the host machine and mount it as a Docker volume. We follow this approach to contain the disk space requirements to run each container.
Before we proceed, an important disclaimer : assume that the system we are going to build will not be secure. We could not account for security while writing this article as it would require many more considerations and mechanisms to put in place. Take what is going to be presented as a base, and educate yourself about the security implications of setting up such a system before you reuse these steps for your production environment. But if you are just testing things out, you should be fine. You have been warned!
Running Jenkins as a Docker container
The first thing we are going to do is to instantiate the Jenkins Master. On the machine you want the Jenkins Master to be running, open up a terminal and write:
docker run -p 8080:8080 -p 50000:50000 --name=jenkins-master jenkins/jenkins:lts
This will pull the official Jenkins LTS image from the Docker Hub and start the container with the name jenkins-master.
You may have noticed that if you launch the container in this way, the Jenkins log will be directed to stdout. However, in a real-world scenario you probably wouldn’t want that. Luckily, Docker offers the possibility of launching a container in detached mode using the -d flag.
Secondly, if for some reason you restart the Docker daemon (for instance if the host machine gets restarted), the Jenkins container will remain stopped and you will have to start it manually. Again, Docker offers a handy option flag for the run command to avoid this to happen, and that is --restart=always.
Let’s try that. Stop the current container (since it is not in detached mode, pressing CTRL+C in the terminal where is running should be enough), and then type:
docker rm jenkins-master
docker run -d --restart=always -p 8080:8080 -p 50000:50000 --name=jenkins-master jenkins/jenkins:lts
The container should be now up and running, but in detached mode. Let’s verify by running the docker ps command. You should see something like this:
And if you try to restart the Docker daemon (Or equivalently restart the entire host machine) and then run docker ps as shown below:
sudo service docker restart && docker ps
you will see that the container has been restarted accordingly (look at the STATUS column):
Cool isn’t it? But that’s not all. Before we move onto the setup of the Jenkins Master, let’s look at a little more complex example. More precisely, now that you have detached the docker container, what if you want to still access the log, and preserve it regardless of the container state (even if it is removed)? And similarly, what if you want to make your configuration and installed plugins persistent? To achieve that (and much more!) we will require to use two things: a Dockerfile and Docker volumes.
The Dockerfile is simply a recipe for Docker images. It is a text file that describes the instructions to create an image, to be fed to the Docker build command. Open your favorite text editor, and copy and paste the following text:
FROM jenkins/jenkins:lts
# Switch to root
USER root
# Create log and cache directories and set correct permissions
RUN mkdir /var/log/jenkins
RUN mkdir /var/cache/jenkins
RUN chown -R jenkins:jenkins /var/log/jenkins
RUN chown -R jenkins:jenkins /var/cache/jenkins
# Set some useful Jenkins env vars, like configuring where to write the logfile, and setting a custom webroot location
ENV JENKINS_OPTS="--logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war"
# Install Jenkins plugins
RUN /usr/local/bin/install-plugins.sh git github docker-plugin
# Switch to user jenkins
USER jenkins
Now save the content as Dockerfile. Here we basically start FROM the official Jenkins LTS image, and apply a few customizations. We first change the USER to root to be able to create directories and change permissions. We then manually create custom log and cache paths that we pass to the JENKINS_OPTS ENVironment variable. Finally, we install the required plugins, and switch back the USER to the default user jenkins. For more details on the Dockerfile syntax, please look at the official Dockerfile reference.
We are almost there, we just need to build the custom Docker image using the freshly-made Dockerfile, and run a container using that image. Open a shell, cd where you have saved the Dockerfile and then type:
docker rm jenkins-master
docker build -t jks .
docker run -d --restart=always -p 8080:8080 -p 50000:50000 --name=jenkins-master -v jks-data:/var/jenkins_home -v jks-log:/var/log/jenkins jks
Good job! All it remains now is to complete the Jenkins Master setup using its Wizard. But before that, you may have noticed two new arguments (-v) passed to the run command. The -v flag allows to specify a Docker volume. Volumes can be either used to mount directories from the host, or to create storage buckets that can be used to persistently store data. In this instance, we are doing the latter. We are creating two Docker-managed volumes jks-data and jks-log to respectively store configuration and log, even if the container is destroyed. Indeed if you type docker volume ls in a shell, you will see this two volumes listed:
In case you remove the running container, and you reference again in the future the volumes jks-data or jks-log, Docker will find the existing instances and use them, instead of creating new ones (unless you have also removed the volumes). This may be useful for instance if you need to make changes to the Docker image and deploy a revised container, but you don’t want to lose your current configuration. However, to access the data stored in a volume, this has to be bound to an existing container. If that is the case, even if the container is stopped, you will be able to access its content, and consequently also of the mounted volumes. For instance, to access the log even when the container is stopped, you can use the Docker cp command:
docker cp jenkins-master:/var/log/jenkins/jenkins.log jenkins.log
less jenkins.log
Let’s now move onto the Jenkins setup Wizard. Open your favorite browser and go to: http:\\localhost:8080 (if the machine hosting the Jenkins Master is located remotely, you will need to replace localhost with the IP address of that machine). You will land on this page:
Which means you will need to unlock Jenkins in order to use it. Let’s recover the requested password using the Docker exec command. In a shell type:
docker exec jenkins-master cat /var/jenkins_home/secrets/initialAdminPassword
Copy and paste the retrieved password where asked, and then click “continue”.
On the next page, you can just click on “Install suggested plugins”, we won’t need anything else for this project. Wait for it to finish, and you will then be redirected to the admin creation page:
Simply fill the form and click “Save and Continue”.
After that, you will be asked to specify the Jenkins URL. For this project, you can leave the default value. Click on “Save and Finish”, and then on “Start using Jenkins!”. You will be now redirected to the Jenkins homepage (after logging in with the credentials you just specified):
You are now good to go! We can move on to setting up the Docker Swarm.
Docker Swarm for distributed builds
Now that we have the Jenkins Master up and running, it is time to set-up the Docker Swarm to distribute builds on multiple hosts. Since the current version of the plugin supports only Docker Swarm Standalone while the newer Docker Engine Swarm Mode is not yet supported, that is what we are going to set-up. You can find more info about the Docker plugin on the official wiki.
Decide now which one of your machines will be designated as the Swarm manager. We are going to assume it will be the machine where the Jenkins Master is running, but in reality the choice does not really matter.
Enable remote API for Docker
The first step that needs to be performed on all machines that will join the Swarm is to enable the remote API for Docker. In order to do so, you will need to edit the file /etc/docker/daemon.json (create it if it does not exists) and insert this line:
{
"hosts": ["fd://", "tcp://0.0.0.0:2375", "unix:///var/run/docker.sock"]
}
This will bind the Docker daemon to both a Unix socket and a TCP port. After you have saved the file, you will need to reload the Docker daemon for the change to be effective. Open up a shell on the machine and type:
sudo service docker restart
This should do. On the machine where the Jenkins Master is running, if you now type docker ps in a shell, you should still be able to see the container running, even after restarting the daemon. This is thanks to the --restart=always flag that we passed to the docker run command, as explained earlier.
Start the discovery service
You will need a discovery backend for Docker Swarm Standalone to work. This will allow to dynamically manage the machines within your cluster. For this example project, we will be using Consul, and (of course!) we will be running the service as a Docker container.
On the designated manager machine, open a shell and type:
MANAGER_IP=$(hostname --all-ip-addresses | awk '{print $1}')
docker run -d --restart=always -p 8500:8500 --hostname=consul --name=consul-discovery progrium/consul -server -bootstrap -advertise=$MANAGER_IP
echo $MANAGER_IP
The IP that will be printed in the terminal is the <manager-ip>. Please note it down and substitute the occurrences of this string in the following commands that reference to it.
Start the Docker manager
Let’s now start the Swarm manager on the manager machine. Again, open up a shell and type:
MANAGER_IP=$(hostname --all-ip-addresses | awk '{print $1}')
docker run -d --restart=always -p 4000:4000 --name=smarm-manager swarm manage -H :4000 --advertise=$MANAGER_IP:4000 consul://$MANAGER_IP:8500
Add machines to the Swarm
On all machines that will need to join the Docker Swarm, start a Swarm node typing the following in a shell:
MANAGER_IP=<manager-ip>
HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}')
docker run -d --restart=always --name=smarm-node swarm join --advertise=$HOST_IP:2375 consul://$MANAGER_IP:8500
Final check
Now check that everything is up and running. Open a shell on any of the machines that you have used to setup the Swarm and type:
docker -H tcp://<manager-ip>:4000 info
This should print out a recap of the Swarm configuration, including a list of the machines that have currently joined.
What about adding new machines later?
If you need to enlarge your Swarm adding new machines at a later stage, you will just need to enable the remote API for Docker and Add the machine to the swarm. And that’s it! This change will be transparent to Jenkins and you will have immediately more resources available to instantiate ephemeral Jenkins Agents. No additional setup required.
Putting it all together: set up a Docker Jenkins Agent
We are now at the final steps of the setup part. Here we will create the Docker Jenkins Agent and configure the Jenkins Master to use it.
Docker Registry
The first thing you will need to do is to create a private Docker Registry to store your Agents images. On the manager machine, open a shell and run:
docker run -d --restart=always -p 5000:5000 --name=private-registry -v ./config/registry:/etc/docker/registry:ro -v registry-data:/var/lib/registry registry
hostname --all-ip-addresses | awk '{print $1}'
This will start a private Registry (as always, as a Docker container) and print out what will be the Registry IP, that is the <manager-ip>. To allow all the Swarm nodes to access the Registry, edit again the /etc/docker/daemon.json file on all machines in the Swarm and put this line:
{
"insecure-registries" : ["<manager-ip>:5000"]
}
Then restart the Docker daemon. Remember that all your containers will be automatically restarted since we added the --restart=always flag to all of them.
sudo service docker restart
Creating the Jenkins Agent Docker image
We now need to create the Docker image that will be used to spawn our ephemeral Agent. Similarly for what we have done for the Jenkins Master, let’s create a Dockerfile, this time with the following content:
FROM ubuntu:16.04
# Install Useful Packages
RUN apt-get -y update && \
apt-get -y install openjdk-8-jre \
git \
python3 \
python3-pip \
wget \
libtool \
build-essential \
automake \
zip \
unzip \
lsb-release \
locales \
xterm \
gcc-multilib \
sudo && \
rm -rf /var/lib/apt/lists/*
# /bin/sh set to /bin/bash
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
ARG user=jenkins
ARG group=jenkins
ARG uid=1000
ARG gid=1000
ENV JENKINS_HOME /home/${user}
# Jenkins is run with user `jenkins`, uid = 1000
RUN groupadd -g ${gid} ${group} \
&& useradd -d "$JENKINS_HOME" -u ${uid} -g ${gid} -m -s /bin/bash ${user}
RUN chown -R ${user}:${user} /home/${user}
# Add the jenkins user to sudoers
RUN echo "${user} ALL=(ALL) NOPASSWD:ALL" >> etc/sudoers
This will create a Docker image that should be sufficient to build our source distribution, and is general enough to carry other basic Jenkins jobs. Feel free to modify it to your liking if you want more packages or specific configurations.
Now let’s build the Docker image and upload it to the Registry we previously created using the Docker push command (remember to replace <manager-ip> with the correct value):
docker build -t pynq-sdist-builder .
docker tag pynq-sdist-builder <manager-ip>:5000/pynq-sdist-builder
docker push <manager-ip>:5000/pynq-sdist-builder
Setting up the Docker plugin on Jenkins
Open the Jenkins GUI using the browser as you did previously. On the left, click on “Manage Jenkins” and then on the first available option named “Configure Jenkins”. In the configuration page, scroll down to the bottom, until you see the “Cloud” section:
Click on “Add a new cloud”, and then on “Docker”. After you have done that, click on “Docker Cloud details” to expand the related configurations.
In the “Docker Host URI” put the Docker Swarm address tcp://<manager-ip>:4000, then click on “Test Connection”. You should see the connection test passing and showing something likeVersion = swarm/X.Y.Z, API Version = X.YZ as shown in the image above. Remember to tick the “Enabled” checkbox and then on “Apply” to save the changes.
Adding the Docker Agent to Jenkins and allow it to access the host Vivado install.
While still being in the “Cloud” section of the Jenkins configuration page, click on “Docker Agent templates” to add a new Docker Agent. Specify a “Label”, like pynq-sdist-builder. This will be required to reference the correct Agent when creating the Jenkins job. Then, tick the “Enabled” checkbox to enable the Agent template. In “Docker Image” provide the address to access the target Docker image, which is going to be <manager-ip>:5000/pynq-sdist-builder. Under “Remote File System Root” put /home/jenkins, select Only build jobs with label expressions matching this node for “Usage”, and type jenkins for “User”. Remember also to tick the “Remove volumes” checkbox so that the volumes associated with the container are removed when it is destroyed. This is optional but will keep disk usage on your machines contained. Lastly, change the “Pull strategy” to Pull once and update latest. You can look at the image below for reference:
We are forgetting one last thing though, we need make Vivado visible to the container by mounting it as a volume. Click on “Container settings” and under Volumes, insert the following <Vivado-host-install-path>:/opt/Xilinx/Vivado:ro . The Vivado install path is usually /opt/Xilinx/Vivado . What we are telling Jenkins to do here is to mount the path on the host where Vivado is installed to the path /opt/Xilinx/Vivado of the container, in read-only ( ro ) mode.
Click on Save.
Congratulations! You are now ready to run your first Jenkins job!
Creating a Jenkins Pipeline: build the PYNQ FPGA binaries and source distribution
In this last part of the article, we are going to explore how to define a simple Jenkins Declarative Pipeline to rebuild the PYNQ v2.5 source distribution. In order to do so, we are going to need to also run Vivado (through the build.sh script included in the PYNQ repo) inside the Jenkins Pipeline, to build the required FPGA binaries (bitstreams and Microblaze executables).
Let’s get straight into this. In Jenkins, click on “New Item” on the left menu.
In the following page, select “Pipeline”, give it a name and click “OK”. We’ll use PYNQ_v2.5_sdist .
In the configuration page that is presented next, tick on the “GitHub project” and write down the repository link: https://github.com/Xilinx/PYNQ
Under the “Build Triggers” section, let’s create a trigger. Let’s use the “Poll SCM” option and specify @daily. This will check every day for changes on the GitHub repo, and trigger a build if there are any. Keep in mind that this is just a basic configuration to give you an example, but there are many more complex combinations for triggering builds in Jenkins, some even relying on additional plugins.
Finally, in the “Pipeline” section, let’s specify the actual Jenkins Declarative Pipeline we are going to execute. Make sure that under “Definition” the “Pipeline script” option is selected, and copy and paste the following in the “Script” text-box:
pipeline {
agent {
label "pynq-sdist-builder"
}
stages {
stage("Checkout repository"){
steps {
git branch: "image_v2.5", url: "https://github.com/Xilinx/PYNQ"
}
}
stage("Build"){
steps {
sh '''
source /opt/Xilinx/Vivado/2019.1/settings64.sh
./build.sh
python3 setup.py sdist
'''
archiveArtifacts artifacts: "dist/pynq-2.5.tar.gz", onlyIfSuccessful: true
}
}
}
}
The other option for “Definition” would be “Pipeline script from SCM”, but in that case we are going to need to rely on a Jenkinsfile, configured with our Pipeline, that should be hosted on the target repository. In this case however, we don’t have control over the repository we are using, so we are going to specify the Pipeline directly in Jenkins.
Click on Save. You are now ready to launch your first Jenkins build! But let’s briefly review what the Pipeline will do before that. Keep in mind that this is going to be just a quick look at it. If you want more details on how to write a Jenkins Pipeline, you are encouraged to read more on the Pipeline Syntax page. First, we define the Agent that will be used thanks to the “label” argument, that will correspond to the label we have specified previously while creating our Docker Agent template. Then, we create two stages for our Pipeline. In the first (the stage named “Checkout repository”), we checkout the target repository with the correct branch. In the second (the stage named “Build”), we actually build the source distribution. In it we first source the Vivado settings64.sh , then we run the build.sh script to build the PYNQ FPGA binaries. This script is used in our sdbuild flow and takes care of building all the FPGA binaries required to create the final PYNQ source distribution. After that is completed, we package everything using the sdist command, and we archive the produced tarball as build artifact.
The only step left is to click on “Build now” on the left menu and wait for it to finish. Once done, you will be able see the source distribution saved as artifact in the Jenkins Pipeline dashboard. If you want to download it, you can simply click on it.
And that concludes this article! If you have any further questions, don’t hesitate to use our forum discuss.pynq.io to ask them!