Continuous Integration

On the vEcoli GitHub website, there is a badge next to certain commits that is either a red “X” or a green checkmark. Clicking on this badge reveals a set of continuous integration (CI) tests that are automatically run to ensure the model continues to work as expected. Some of these tests are run entirely through a GitHub service called GitHub Actions. Others are run via a Jenkins instance that is constantly running on the Sherlock cluster.

GitHub Actions

The tests that are run through GitHub Actions are configured by the files in .github/workflows. Some of these “tests” can be more accurately described as maintenance tasks, like publishing the updated documentation website. These tasks are:

  • docs_deploy.yml: deploy-docs job updates the documentation website every time the master branch changes (push, PR merge)

  • docs_test.yml: test-docs job ensures that the documentation builds properly (Sphinx) on PRs and changes to master

  • pr_tests.yml: tests that run on PRs and changes to master

    • Reproducibility ensures that two runs of the ParCa and simulation with the same settings produce the same output

    • two-gens runs a two-generation workflow on glucose minimal media

  • pytest.yml: code quality and functionality tests that run on PRs and changes to master

    • Pytest runs pytest (all functions with test_ prefix)

    • Mypy checks types with mypy

    • Lint checks code style and formatting with ruff

Tip

If you are still actively working on a pull request, add [skip ci] on a new line in your commit message to skip the CI tests.

Logs from these tests can be viewed on the GitHub website and you are required to get all of these tests passing before merging a PR.

When you submit a pull request (PR), a bot will comment with a table showing the current code coverage of the pytest suite. As of December 2024, coverage is less than 30% and even that is misleading because many of the tests are “migration” tests that compare vEcoli to a snapshot of the original whole cell model. These tests will be removed in the future as vEcoli is further developed. Additionally, these tests do not tell us whether the code is working as intended, only that it is working the same way it worked in the original model. Ideally, we would like to increase test coverage by adding unit tests which actually test edge cases and ensure the code does what it is supposed to do.

To that end, we would appreciate if you added tests as part of your pull requests. This could be as simple as a Python function with the test_ prefix that ensures the code added or modified in your PR works as intended using a few test cases.

Jenkins

Longer running CI tests that are not compatible with the free tier of GitHub Actions are run using Jenkins on Sherlock. For information about this setup (e.g. for troubleshooting or to replicate it on another cluster), please refer to Jenkins Setup.

Each test has an associated Jenkinsfile in runscripts/jenkins/Jenkinsfile that, in essence, runs one or more workflows using the JSON configuration files located in runscripts/jenkins/configs. These tests are designed to run daily (cron) on the latest commit of the master branch.

The logs for failed tests are publicly viewable on the repository website by clicking on the relevant commit status badges. Upon doing so, you will be brought to a page with logs and a link to View more details on Jenkins vEcoli. This link requires access to the Covert Lab’s Sherlock partition.

Connecting to Jenkins

First, connect to Sherlock and run the following command:

squeue -p mcovert -o '%.10i %.9P %.50j %.8u %.2t %.10M %.3C %.6D %.20R %k'

Tip

Create an alias sqp for the command above by adding alias sqp="{insert command here}" to your ~/.bashrc.

Look for a job called jenkins_new. Under the COMMENT column, there should be a login node of the format shXX-lnXX. Close your SSH connection and run the following command, replacing as appropriate:

ssh {username}@{login node}.sherlock.stanford.edu -L 8080:localhost:8080

With that SSH connection open, the View more details on Jenkins vEcoli link will pull up the details of the failing job in the Jenkins interface.

Additionally, you can open the Jenkins web interface by going to http://localhost:8080 in your web browser. From here, you can see the status of all jobs, start and stop jobs, add and modify jobs, and do other administrative tasks.

Note

Jenkins is set-up as a persistent job on Sherlock that automatically resubmits itself every ~5 days. If this fails, it can be restarted by navigating to $GROUP_HOME/jenkins_vecoli/slurm_logs and running sbatch --mail-user={your email here} ../jenkins_vecoli.sh.

Modifying Tests

Any modifications to the existing Jenkinsfiles in runscripts/jenkins/Jenkinsfile will modify the behavior of the corresponding tests. To add a new test, you will need to create a new Jenkinsfile and (with administrator Jenkins privileges) add a new multibranch pipeline job (see below).

Adding New Jenkins Jobs

First, create a new branch and push a commit to GitHub with your new Jenkinsfile. Refer to the existing Jenkinsfiles in runscripts/jenkins/Jenkinsfile for examples.

From the main Jenkins dashboard, click New Item in the left sidebar and select Multibranch Pipeline.

Under Branch Sources:

  1. Select GitHub.

  2. Select the GitHub App credential added in Jenkins Setup.

  3. Enter the vEcoli repository URL.

Under Behaviors:

  1. Add the Filter by name (with wildcards) behavior and set Include to master. To test the pipeline, you can temporarily add the name of your new branch, then save the pipeline. Jenkins should recognize the Jenkinsfile on your branch and trigger the pipeline (including setting GitHub commit statuses). Make sure to remove your branch from this section, and save the pipeline again when you are done testing.

  2. Add the Status Checks Properties behavior, give it an informative name, and tick Skip GitHub Branch Source notifications.

Under Build Configuration:

  1. Replace Jenkinsfile with the path to the Jenkinsfile for the pipeline relative to the root of the repository (e.g. runscripts/jenkins/Jenkinsfile/anaerobic).

Click Save to create the pipeline, scan the repository for branches that match the filter and contain the Jenkinsfile, and trigger the pipeline as appropriate.

Jenkins Setup

Note

This section is intended for people who want to set up their own Jenkins instance on a non-Sherlock cluster or for troubleshooting purposes. Members of the Covert Lab should already have a functioning Jenkins instance on Sherlock.

The following describes the steps taken to set up Jenkins on Sherlock to run long continuous integration tests on the master branch of vEcoli.

Request an interactive session on Sherlock, taking note of the login node. Once the interactive session is started, run the following command to forward the port used by Jenkins to the login node:

ssh -nNT {username}@{login node} -R 8080:localhost:8080 &

In this same session, download the latest WAR file from the Jenkins website, load the Java and fontconfig modules, then run Jenkins:

wget https://get.jenkins.io/war-stable/latest/jenkins.war
module load java/17.0.4 fontconfig
JENKINS_HOME=$GROUP_HOME/jenkins_vecoli java -jar jenkins.war --httpPort=8080

In a new terminal, open a new SSH connection to the previously noted login node with port forwarding:

ssh {username}@{login node}.sherlock.stanford.edu -L 8080:localhost:8080

On a local machine, open a web browser and navigate to localhost:8080. Proceed with the post-installation setup wizard (see Jenkins documentation).

Manually select the following basic plugins to install: Folders, OWASP Markup Formatter, Build Timeout, Credentials Binding, Timestamper, Workspace Cleanup, Pipeline, GitHub Branch Source, Pipeline: GitHub Groovy Libraries, Pipeline Graph View, Git, GitHub, Matrix Authorization, Email Extension, Mailer, and Dark Theme.

Create an admin user with a username and password of your choice, and keep the default web URL of localhost:8080. After setup is complete, click on Manage Jenkins in the left sidebar, then Plugins. Click Available Plugins in the left sidebar, then search for and install the GitHub Checks plugin.

Follow the linked instructions to create a GitHub App for the Covert Lab organization, install it on the vEcoli repository, and add it as a credential in Jenkins.

Stop the Jenkins server by pressing Ctrl+C in the terminal where it is running. Then, move the jenkins.war file to the $GROUP_HOME/jenkins_vecoli directory. Copy runscripts/jenkins/jenkins_vecoli.sh to the same directory.

Finally, create a directory called slurm_logs in $GROUP_HOME/jenkins_vecoli and cd into it. From here, launch Jenkins with sbatch --mail-user={your email here} ../jenkins_vecoli.sh. This will queue a persistent Jenkins job that should run indefinitely, resubmitting itself every 5 days. The stdout and stderr from these jobs will be written to the directory in which you ran the sbatch command. Remember to run sbatch in slurm_logs to keep all logs in a consistent location accessible to all members of the lab. You will get an email if any of these jobs fail, in which case you should review the most recent logs and resubmit with sbatch.