Testing

Every recipe has a pytest module beside it, and the repository has checks of its own. This chapter is how to run them, how to add one, and what CI runs.

Levels#

Running them#

pip install pytest
pytest tests/ -m static                                  # seconds, no xlings
pytest tests/ -m "static or isolation"                   # what every pull request runs first
pytest tests/c/test_cmake.py -m "static or isolation" -v # one package

L1, L3 and L4 call xlings and install for real. Run them only with an isolated XLINGS_HOME, never against your own home:

XLINGS_HOME="$(mktemp -d)" pytest tests/c/test_cmake.py -m "lifecycle or verify" -v

Adding a test#

The module mirrors the recipe: pkgs/n/my-package.lua → tests/n/test_my_package.py (- becomes _).

"""mypackage"""
import pytest
from tests.lib.xpkg_parser import parse_xpkg
from tests.lib.assertions import (
    assert_required_fields, assert_valid_spec, assert_valid_type,
    assert_no_typos, assert_no_exec_xvm, assert_no_bashrc_modification,
    assert_no_direct_path_modification, assert_uses_new_api,
    assert_xim_add_succeeds, assert_install_succeeds,
    assert_command_output, assert_xvm_registered,
)
from tests.lib.platform_utils import skip_if_not

PKG = "mypackage"
PKG_FILE = "pkgs/m/mypackage.lua"


@pytest.fixture(scope="module")
def meta():
    return parse_xpkg(PKG_FILE)


class TestStatic:
    @pytest.mark.static
    def test_required_fields(self, meta):
        assert_required_fields(meta)

    @pytest.mark.static
    def test_valid_spec(self, meta):
        assert_valid_spec(meta)

    @pytest.mark.static
    def test_valid_type(self, meta):
        assert_valid_type(meta)

    @pytest.mark.static
    def test_no_typos(self):
        assert_no_typos(PKG_FILE)


class TestIndex:
    @pytest.mark.index
    def test_xim_add(self):
        assert_xim_add_succeeds(PKG_FILE)


class TestIsolation:
    @pytest.mark.isolation
    def test_no_exec_xvm(self):
        assert_no_exec_xvm(PKG_FILE)

    @pytest.mark.isolation
    def test_no_bashrc(self):
        assert_no_bashrc_modification(PKG_FILE)

    @pytest.mark.isolation
    def test_no_path_modification(self):
        assert_no_direct_path_modification(PKG_FILE)

    @pytest.mark.isolation
    def test_new_api(self):
        assert_uses_new_api(PKG_FILE)


class TestLifecycle:
    @pytest.mark.lifecycle
    @skip_if_not("linux")
    def test_install(self):
        assert_install_succeeds(PKG)


class TestVerify:
    @pytest.mark.verify
    @skip_if_not("linux")
    def test_version(self):
        assert_command_output("mypackage --version", contains="mypackage")

    @pytest.mark.verify
    @skip_if_not("linux")
    def test_xvm_registered(self):
        assert_xvm_registered("mypackage")

• An alias package (ref) asserts meta.is_ref and little else.
• A platform-specific test is marked @skip_if_not("windows").
• A known problem is @pytest.mark.xfail(reason=...), which does not block CI.
• A verify test does real work where it can (a compiler compiles, a VM boots); --version does not show a broken library path.

Shared modules#

Repository-wide checks#

Some rules span every recipe, and each has its own test or script:

What CI runs#

L3 and L4 run in CI only for what a change touched and in the weekly sweep: a full install of every recipe on every platform is not a per-push budget.