CI/CD Workflows

Overview

BLVM uses GitHub Actions for continuous integration and deployment. All workflows run on self-hosted Linux x64 runners to ensure security and deterministic builds.

Release channels: stable artifacts from main (versioned GitHub Releases + GHCR tags); rolling nightly from develop (nightly tag, ghcr.io/btcdecoded/blvm:nightly). See Release process: Release channels.

What Happens When You Push Code

On Push to Any Branch

When you push code to any branch, the following workflows may trigger:

  1. CI Workflow - Runs tests, linting, and build verification
  2. Coverage Workflow - Calculates test coverage
  3. Security Workflow - Runs security checks (if configured)

On push to main

When the release job runs (not every push: respect [skip_release] and workflow filters), blvm/.github/workflows/ci.yml may:

  1. Bump Cargo.toml patch version and tag vX.Y.Z
  2. Build Linux x86_64, Linux aarch64, and Windows release binaries
  3. Publish crates.io sets per repo policy (coordinated release scripts)
  4. Upload GitHub Release artifacts and dispatch blvm-release to website and commons-website (install page at btcdecoded.org/install). The blvm-docs book install chapter is timeless and deploys only on blvm-docs pushes.

See Release process for artifact variants and feature matrices.

On push to develop

The same ci.yml runs nightly-release on develop: rolling nightly tag, prerelease assets, and ghcr.io/btcdecoded/blvm:nightly, after optional develop crate-set publish/verify.

Repository-Specific CI Workflows

blvm-consensus

Workflows:

  • ci.yml - Runs test suite, linting, and build verification
  • coverage.yml - Calculates test coverage

What Runs:

  • Unit tests
  • Integration tests
  • Property-based tests
  • BLVM Specification Lock (check-drift then verify on self-hosted runners; required where #[spec_locked] is enabled)
  • Code formatting check (cargo fmt --check)
  • Linting check (cargo clippy)

blvm-protocol

Workflows:

  • ci.yml - Runs test suite and build verification
  • coverage.yml - Calculates test coverage

What Runs:

  • Unit tests
  • Integration tests
  • Protocol compatibility tests
  • BLVM Specification Lock (where #[spec_locked] is enabled)
  • Build verification

blvm-node

Workflows:

  • ci.yml - Runs test suite and build verification
  • coverage.yml - Calculates test coverage

What Runs:

  • Unit tests
  • Integration tests
  • Node functionality tests
  • Network protocol tests
  • BLVM Specification Lock (where #[spec_locked] is enabled)
  • Build verification

blvm (Main Repository)

Workflows:

  • ci.yml - Runs tests across all components
  • coverage.yml - Aggregates coverage from all repos
  • release.yml - Official release workflow
  • prerelease.yml - Prerelease workflow
  • nightly-prerelease.yml - Scheduled nightly builds

Reusable Workflows (blvm-commons)

The blvm-commons repository provides reusable workflows that other repositories call:

verify_consensus.yml

Purpose: Reusable workflow for multi-repo checkouts: runs tests and BLVM Specification Lock when the blvm-spec-lock input is true (mirrors per-crate Verify jobs).

Inputs:

  • repo - Repository name
  • ref - Git reference (branch/tag)
  • blvm-spec-lock - Boolean to run spec-lock (check-drift + verify)

What It Does:

  • Checks out the repository
  • Runs test suite
  • Runs BLVM Specification Lock when enabled (same commands as Formal Verification)
  • Reports results

build_lib.yml

Purpose: Deterministic library build with artifact hashing

Inputs:

  • repo - Repository name
  • ref - Git reference
  • package - Cargo package name
  • features - Feature flags to enable
  • verify_deterministic - Optional: rebuild and compare hashes

What It Does:

  • Builds the library with cargo build --locked --release
  • Hashes outputs to SHA256SUMS
  • Optionally verifies deterministic builds (rebuild and compare)

build_docker.yml

Purpose: Builds Docker images

Inputs:

  • repo - Repository name
  • ref - Git reference
  • tag - Docker image tag
  • image_name - Docker image name
  • push - Boolean to push to registry

What It Does:

  • Builds Docker image
  • Optionally pushes to registry

Workflow Dependencies and Ordering

Builds follow Cargo’s dependency graph (simplified view):

1. blvm-primitives: foundation types/crypto shared by consensus & protocol
 ↓
2. blvm-consensus: depends on primitives
 ↓
3. blvm-protocol: depends on consensus + primitives
 ↓
4. blvm-node: depends on protocol + consensus
 ↓
5. blvm (CLI): depends on blvm-node

blvm-sdk: depends on blvm-protocol + blvm-consensus (and optionally blvm-node); not a separate “no-deps” lane
 ↓
blvm-commons: depends on blvm-sdk + blvm-protocol

Security Gates: Consensus verification (tests + BLVM Specification Lock where #[spec_locked] is enabled) must pass before downstream builds proceed. Details: Formal Verification.

Self-Hosted Runners

All workflows run on self-hosted Linux x64 runners:

  • Security: Code never leaves our infrastructure
  • Performance: Faster builds, no rate limits
  • Deterministic: Consistent build environment
  • Labels: Optional labels (rust, docker, blvm-spec-lock) optimize job assignment (note: label uses lowercase for technical compatibility)

Runner Policy:

  • All jobs run on [self-hosted, Linux, X64, builds] runners (workflows may also accept related label variants)
  • Workflows handle installation as fallback if labeled runners unavailable
  • Repos should restrict Actions to self-hosted in settings

Deterministic Builds

All builds use deterministic build practices:

  • Locked builds: When a lockfile is part of the workflow, cargo build --locked (or equivalent) resolves dependencies exactly as locked. Root Cargo.lock policy differs by repository, see each crate’s Contributing.
  • Toolchain Pinning: Per-repo rust-toolchain.toml defines exact Rust version
  • Artifact Hashing: All outputs hashed to SHA256SUMS
  • Verification: Optional deterministic verification (rebuild and compare hashes)

Interpreting CI Results

✅ Success

All checks pass:

  • ✅ Tests pass
  • ✅ Linting passes
  • ✅ Build succeeds
  • ✅ Coverage meets threshold

Action: Your PR is ready for review (subject to governance requirements).

❌ Test Failures

One or more tests fail:

  • Check the test output in the workflow logs
  • Look for error messages and stack traces
  • Run tests locally to reproduce: cargo test

Common Causes:

  • Logic errors in your code
  • Test environment differences
  • Flaky tests (timing issues)

❌ Linting Failures

Code style or quality issues:

  • Formatting: Run cargo fmt locally
  • Clippy warnings: Run cargo clippy -- -D warnings and fix issues

Action: Fix locally and push again.

❌ Build Failures

Code doesn't compile:

  • Check compiler errors in workflow logs
  • Build locally: cargo build
  • Check for missing dependencies or version conflicts

Action: Fix compilation errors and push again.

⚠️ Coverage Below Threshold

Test coverage is below the required threshold:

  • Add more tests to cover untested code
  • Check coverage report to see what's missing

Action: Add tests to increase coverage.

Debugging CI Failures

1. Check Workflow Logs

Click on the failed check in your PR to see detailed logs:

  • Expand failed job
  • Look for error messages
  • Check which step failed

2. Reproduce Locally

Run the same commands locally:

# Run tests
cargo test

# Check formatting
cargo fmt --check

# Run clippy
cargo clippy -- -D warnings

# Build
cargo build --release

3. Check for Environment Differences

CI runs in a clean environment:

  • Dependencies are fresh
  • No local configuration
  • Specific Rust toolchain version

Solution: Use rust-toolchain.toml to pin Rust version.

4. Common Issues

Issue: Tests pass locally but fail in CI

  • Cause: Timing issues, environment differences
  • Solution: Make tests more robust, check for race conditions

Issue: Build works locally but fails in CI

  • Cause: Dependency version mismatch, different lockfile, or different feature set
  • Solution: Follow that repository’s Contributing and .github/workflows/. Many BLVM crates do not commit a root Cargo.lock (see .gitignore); CI may still run cargo … --locked when it generates or checks in a lockfile for that job. When a lockfile is present for your workflow, keep local resolution aligned (e.g. regenerate with cargo generate-lockfile where the project documents it).

Issue: Coverage calculation fails

  • Cause: Coverage tool issues
  • Solution: Check coverage tool version, ensure tests run successfully

Workflow Status Checks

PRs require all status checks to pass before merging:

  • Required Checks: Must pass (configured per repository)
  • Optional Checks: Can fail but won't block merge
  • Status: Shown in PR checks section

Note: Even if all checks pass, PRs still require:

  • Maintainer signatures (see PR Process)
  • Review period to elapse

Best Practices

Before Pushing

  1. Run tests locally: cargo test
  2. Check formatting: cargo fmt
  3. Run clippy: cargo clippy -- -D warnings
  4. Build: cargo build --release

During Development

  1. Push frequently: Small commits are easier to debug
  2. Check CI early: Don't wait until PR is "done"
  3. Fix issues immediately: Don't let failures accumulate

When CI Fails

  1. Don't panic: CI failures are normal during development
  2. Read the logs: Errors name the failing step or crate
  3. Reproduce locally: Fix the issue, then push again
  4. Ask for help: If stuck, ask in discussions or PR comments

Workflow Configuration

Workflows are configured in .github/workflows/ in each repository:

  • Trigger conditions: When workflows run
  • Job definitions: What each job does
  • Runner requirements: Which runners to use
  • Dependencies: Job ordering

Note: Workflows in blvm-commons are reusable and called by other repositories via workflow_call.

Workflow Optimization

Caching Strategies

For self-hosted runners, local caching can provide significant performance improvements:

Local Caching System

Using /tmp/runner-cache with rsync can be much faster than GitHub Actions cache for self-hosted runners (measure on your hardware):

  • No API rate limits: Local filesystem access
  • Faster restore: rsync is much faster than GitHub cache API
  • Works offline: Once cached, no network needed
  • Preserves symlinks: Better than GitHub cache for complex builds

Shared Setup Jobs

Use a single setup job that all other jobs depend on:

  • Checkout dependencies once: Avoid redundant checkouts
  • Generate cache keys once: Share keys via job outputs
  • Parallel execution: Other jobs can run in parallel after setup

Cross-Repo Build Artifact Caching

Cache target/ directories for dependencies across workflow runs:

  • Don't rebuild dependencies: Cache blvm-consensus and blvm-protocol build artifacts
  • Faster incremental builds: Only rebuild what changed
  • Shared across repos: Same cache can be used by multiple repositories

Cache Key Strategy

Use deterministic cache keys based on:

  • Dependency inputs: often a Cargo.lock hash when the workflow commits or restores one; otherwise manifest hashes (Cargo.toml / workspace definition) plus toolchain
  • Rust toolchain version (for toolchain changes)
  • Combined key: ${DEPS_KEY}-${TOOLCHAIN}

Disk Space Management

For long-running runners, implement cache cleanup:

  • Automatic cleanup: Remove caches older than N days
  • Keep recent caches: Maintain last N cache entries
  • Emergency cleanup: Check disk space and clean if >80% full

BLVM repositories: scheduled workflows such as cleanup-runner-disk.yml (e.g. blvm, blvm-node, blvm-sdk) prune stale trees under /tmp/runner-cache/… (namespaced per repo where configured). Node CI documents BLVM_RUNNER_BUILD_ROOT / sibling purge behavior in blvm-node/.github/workflows/ci.yml: keep per-job build roots off shared parent directories unrelated to that workflow.

Performance Improvements

With proper caching optimization:

  • Dependency checkout: ~30s (once in setup job)
  • Cache restore: ~5s per job (local cache vs ~20s for GitHub cache)
  • Dependency build: ~30s (cached artifacts vs ~5min without cache)
  • Total overhead: ~2min vs ~35min without optimization

Estimated speedup: measure setup overhead locally on self-hosted runners (varies by disk and cache state).

Additional Resources