From 99648914400685392377abc98f29a0a90a9b48c2 Mon Sep 17 00:00:00 2001 From: Robert Hensing Date: Thu, 16 Feb 2023 17:17:32 +0100 Subject: [PATCH] nixos/doc: Add Developing the Test Driver --- .../developing-the-test-driver.chapter.md | 43 +++++++++++++++++++ nixos/doc/manual/development/development.md | 1 + 2 files changed, 44 insertions(+) create mode 100644 nixos/doc/manual/development/developing-the-test-driver.chapter.md diff --git a/nixos/doc/manual/development/developing-the-test-driver.chapter.md b/nixos/doc/manual/development/developing-the-test-driver.chapter.md new file mode 100644 index 000000000000..4b70fe00af47 --- /dev/null +++ b/nixos/doc/manual/development/developing-the-test-driver.chapter.md @@ -0,0 +1,43 @@ + +# Developing the NixOS Test Driver {#chap-developing-the-test-driver} + +The NixOS test framework is a project of its own. + +It consists of roughly the following components: + + - `nixos/lib/test-driver`: The Python framework that sets up the test and runs the [`testScript`](#test-opt-testScript) + - `nixos/lib/testing`: The Nix code responsible for the wiring, written using the (NixOS) Module System. + +These components are exposed publicly through: + + - `nixos/lib/default.nix`: The public interface that exposes the `nixos/lib/testing` entrypoint. + - `flake.nix`: Exposes the `lib.nixos`, including the public test interface. + +Beyond the test driver itself, its integration into NixOS and Nixpkgs is important. + + - `pkgs/top-level/all-packages.nix`: Defines the `nixosTests` attribute, used + by the package `tests` attributes and OfBorg. + - `nixos/release.nix`: Defines the `tests` attribute built by Hydra, independently, but analogous to `nixosTests` + - `nixos/release-combined.nix`: Defines which tests are channel blockers. + +Finally, we have legacy entrypoints that users should move away from, but are cared for on a best effort basis. +These include `pkgs.nixosTest`, `testing-python.nix` and `make-test-python.nix`. + +## Testing changes to the test framework {#sec-test-the-test-framework} + +When making significant changes to the test framework, we run the tests on Hydra, to avoid disrupting the larger NixOS project. + +For this, we use the `python-test-refactoring` branch in the `NixOS/nixpkgs` repository, and its [corresponding Hydra jobset](https://hydra.nixos.org/jobset/nixos/python-test-refactoring). +This branch is used as a pointer, and not as a feature branch. + +1. Rebase the PR onto a recent, good evaluation of `nixos-unstable` +2. Create a baseline evaluation by force-pushing this revision of `nixos-unstable` to `python-test-refactoring`. +3. Note the evaluation number (we'll call it ``) +4. Push the PR to `python-test-refactoring` and evaluate the PR on Hydra +5. Create a comparison URL by navigating to the latest build of the PR and adding to the URL `?compare=`. This is not necessary for the evaluation that comes right after the baseline. + +Review the removed tests and newly failed tests using the constructed URL; otherwise you will accidentally compare iterations of the PR instead of changes to the PR base. + +As we currently have some flaky tests, newly failing tests are expected, but should be reviewed to make sure that + - The number of failures did not increase significantly. + - All failures that do occur can reasonably be assumed to fail for a different reason than the changes. diff --git a/nixos/doc/manual/development/development.md b/nixos/doc/manual/development/development.md index 6a0dd091b129..76f405c3b29c 100644 --- a/nixos/doc/manual/development/development.md +++ b/nixos/doc/manual/development/development.md @@ -10,5 +10,6 @@ bootspec.chapter.md what-happens-during-a-system-switch.chapter.md writing-documentation.chapter.md nixos-tests.chapter.md +developing-the-test-driver.chapter.md testing-installer.chapter.md ```