Automatic testing

X-HEEP includes a script to perform automatic tests on your modifications. In addition, it also has a CI setup that checks the code by simulating all the existing applications and handles publishing new X-HEEP releases.

Simulation script

The testing script (test/test_apps/test_apps.py) can be used to perform local tests. For quick debugging, you can check the global variables in the script such as the BLACKLIST and WHITELIST.

You can run it with the following command:

make test

This will output the results in the terminal and in the test/test_apps/test_apps.log file.

Additionally, you can check only the compilation of the applications with the following command:

make test TEST_FLAGS=--compile-only

This script is also integrated in the CI workflow described in the following section.

Github CIs

The project’s Continuous Integration (CI) is managed through GitHub Actions. The workflows are defined in the .github/workflows directory. The main CI workflow is ci.yml, which is triggered on every push and pull request to the main branch.

CI Workflow (ci.yml)

This workflow ensures the stability and integrity of the codebase by running a series of checks, compilations, and simulations.

Triggers:

  • Push to any branch (push: branches: [ "**" ]).

  • Pull request to the main branch (pull_request: branches: [ "main" ]).

Jobs:

  1. determine-image-tag:

    • Purpose: Determines the Docker image tag to be used by subsequent jobs.

    • Details: It checks the Git history for the most recent tag. If no tag is found in the current branch, it looks for one in the main branch. If no tags are found at all, it defaults to latest. This ensures that the CI always uses a relevant toolchain version.

  2. compile-apps:

    • Purpose: Compiles all software applications with both GCC and Clang to ensure they build correctly.

    • Dependencies: Depends on determine-image-tag to select the correct Docker image.

    • Environment: Runs inside the ghcr.io/x-heep/x-heep/x-heep-toolchain Docker container.

    • Steps:

      • Generates the MCU configuration using make mcu-gen X_HEEP_CFG=configs/ci.hjson.

      • Executes test/test_apps/test_apps.py with the --compile-only flag to build all applications, without simulating them. This is done to offer a quick feedback about the apps’ integrity, before their runtime behaviour is checked in RTL simulation.

  3. simulate-apps:

    • Purpose: Runs Verilator RTL simulations for all applications (except the blacklisted ones) to verify their runtime behavior.

    • Condition: This job only runs on pull requests to main.

    • Dependencies: Depends on determine-image-tag.

    • Environment: Runs inside the x-heep-toolchain Docker container.

    • Steps:

      • Generates the MCU configuration using make mcu-gen X_HEEP_CFG=configs/ci.hjson.

      • Executes test/test_apps/test_apps.py to compile and simulate all applications.

  4. lint:

    • Purpose: Checks that all auto-generated hardware files are up-to-date and have been formatted .

    • Dependencies: Depends on determine-image-tag.

    • Environment: Runs inside the x-heep-toolchain Docker container.

    • Steps:

      • Runs make mcu-gen to regenerate all hardware files.

      • Uses util/git-diff.py to check for any differences between the working directory and the git HEAD. The job fails if any differences are found.

  5. gen-peripherals:

    • Purpose: Tests the Python-based peripheral generation scripts and templates.

    • Dependencies: Depends on determine-image-tag.

    • Environment: Runs inside the x-heep-toolchain Docker container.

    • Steps:

      • Runs make clean-all to ensure a clean state.

      • Executes test/test_x_heep_gen/test_peripherals.py.

  6. check-vendor:

    • Purpose: Verifies that all third-party vendored dependencies are up-to-date.

    • Environment: Runs inside a ubuntu-latest VM.

    • Steps:

      • Installs Python dependencies.

      • Runs the util/vendor.py script for all .vendor.hjson files to re-vendor all dependencies.

      • Uses util/git-diff.py to check for any differences, ensuring that any changes to vendored repositories are properly committed.

  7. black-formatter:

    • Purpose: Checks that all Python code adheres to the black formatting standard.

    • Environment: Runs inside a ubuntu-latest VM.

    • Steps:

      • Uses the psf/black GitHub Action to check the formatting of all relevant Python files.

Release Workflows

The project includes a robust, automated process for creating and publishing releases. This is handled by two GitHub Actions workflows: create-release.yml and publish-release.yml. This system ensures that every release is consistently built, tested, and published with its corresponding toolchain and Docker image.

Create X-HEEP Release Workflow (create-release.yml)

This workflow prepares a new release. It is a comprehensive process that builds the toolchain, packages it, creates a draft release, builds a Docker container, and opens a version bump pull request. It’s designed to be triggered manually when a new release is needed.

Trigger:

  • Manual dispatch (workflow_dispatch) from the GitHub Actions tab.

  • Inputs:

    • llvm_version: The LLVM version tag to build (default: llvmorg-19.1.4).

    • gcc_version: The GCC version tag to build (default: 2023.01.03).

    • release_tag: The tag for the new GitHub release (e.g., v1.0.0).

Note

This workflow is intended for major releases that:

  • Introduce support for new tools

  • Bump existing tools to newer versions

  • Modify the CI workflows

  • Represent a significant update in general

Minor bug fixes or feature improvements may not require/justify a full new release.

Jobs:

  1. check-changes:

    • Checks previous release tag.

    • Downloads tool-versions.env from the previous release to compare GCC, LLVM, Verilator, and Verible versions.

    • Checks for changes in Docker-related files (util/docker/, util/conda_environment.yml, util/python-requirements.txt, docs/python-requirements.txt).

    • Sets outputs rebuild_toolchain and rebuild_docker to avoid unnecessary rebuilds.

  2. prepare-release:

    • Creates a new release branch (release/<release_tag>).

    • Updates the version in core-v-mini-mcu.core and the toolchain version in util/docker/dockerfile.

    • Commits and pushes the changes to the new branch.

    • Creates a draft GitHub release.

  3. build-and-upload-toolchain:

    • Conditional: Runs the build only if rebuild_toolchain is true.

    • Builds the RISC-V GCC and Clang/LLVM toolchains from the sources specified in the workflow inputs.

    • Packages the compiled toolchains into a .tar.gz file.

    • If rebuild_toolchain is false, it downloads the toolchain asset from the previous release.

    • Uploads the toolchain tarball as an asset to the draft GitHub release.

    • Uploads a tool-versions.env file containing version metadata.

  4. build-docker:

    • Conditional: Runs the build only if rebuild_docker is true.

    • Downloads the toolchain asset from the draft release.

    • Builds the x-heep-toolchain Docker image, injecting the new toolchain.

    • Pushes the new Docker image to the GitHub Container Registry (GHCR) with the release tag.

    • If rebuild_docker is false, it retags the previous release’s Docker image with the new tag.

  5. create-version-pr:

    • Creates a new pull request to merge the release branch back into main. This PR contains the version bumps.

  6. cleanup-on-failure:

    • This job runs only if any of the previous jobs fail.

    • It automatically cleans up by deleting the draft release, the release tag, the remote release branch, and the pushed Docker image from GHCR. This prevents leftovers from partial, broken releases.

Publish Release Workflow (publish-release.yml)

This workflow finalizes the release process. It is triggered automatically after the version bump PR (created by the create-release.yml workflow) is merged into the main branch.

Trigger:

  • A pull request from a release/* branch is merged into main.

Jobs:

  1. publish-release:

    • Identifies the release tag from the merged branch name.

    • Converts the corresponding draft release into a public release.

    • Deletes the now-merged remote release branch to keep the repository clean.

  2. tag-latest:

    • After the release is published, this job pulls the newly released Docker image from GHCR.

    • It then re-tags this image with the latest tag and pushes it.

    • This ensures that the main ci.yml workflow will use the most up-to-date toolchain for future runs on the main branch.

Compare mcu-gen runs

The test/test_x_heep_gen/compare_mcu_gen.py script is a utility to compare the outputs of mcu-gen between the current branch and main. This can be useful to manually check if changes in the configuration or in the MCU-Gen code have an effect on the generated files.

You can run it with the following command:

make compare-mcu-gen

This will generate the mcu-gen outputs in the _mcu_gen_current and _mcu_gen_main directories inside test/test_x_heep_gen, and then list the files that differ between them. You can check the differences to see if they are expected or if they indicate an unintended change in the generated files.