Background

For a while now I've wanted a tool to help in a number of areas:

• Running tests and building docs when I push to Git repositories
• Reconfiguring Guix machines when I update the configuration in Git
• Orchestrating testing for Guix patches/branches/Pull Requests, fixing some of the issues with the QA Frontpage

Many years ago I used Jenkins, I even remember trying to write some Groovy code for some reason.

When working on automated patch testing for Guix, I did use Laminar, and that worked. The simplicity was nice, but the job chaining wasn't a great experience when trying to structure more complicated tasks. I haven't got around to trying the Laminar configuration from Guix Forge yet, but it looks like the perfect way to configure Laminar.

I did have a go at using Forgejo actions, but the YAML wasn't satisfying.

Another tool I did come across years ago though was Concourse. Unfortunately this leans heavily in to YAML and image based containers, but it provided a way of structuring more complicated tasks, and displaying this to make monitoring what was going on easier.

So, for a while I've been thinking about what a tool for generic pipelines would look like if you leaned in to Guile, Guix and Spritely Goblins.

Introducing Guile Automate

Guile Automate provides the structure for writing pipelines with Guile. The steps and pipeline itself are record values. The body of a step is a Guix G-expression, since this code is staged to run later when the pipeline is run. Steps can reference the outputs of other steps, plus parameters and resources. Parameters are values provided when a pipeline run is triggered, and resources are values that sit outside of the pipeline and manage its interaction with external things, like the network, filesystem and Guix daemon.

Spritely Goblins is used within Guile Automate, and it's also the mechanism through which steps interact and resources work.

Also included is a component called the "pumphouse". This stores the history of the pipeline runs, the log output and a web interface to view all of this plus the live activity of the pipelines.

You can see Guile Automate's own pipelines on automate.cbaines.net and if you follow the source link on the trunk pipeline page, you'll see how this is defined.

I've put this together over the past two weeks. Similar to Safsaf I worked on this while travelling. I think there's something about this where I feel less pressure to continue grinding away on existing projects and look at something new.

This has also been written using Claude Code, which has some interesting consequences for the software design/architecture process. I'm comfortable designing around Guile, Guix and Fibers, but Spritely Goblins is an unknown. Having something working almost immediately has meant that iterating on how to set out the pipelines has been much easier.

What next?

This is very much a prototype at the moment, but I'd like Guile Automate to become a great generic tool for building and running pipelines. My use cases are around continuous integration and continuous deployment type tasks, but I'd love to hear other tasks where this might be helpful.

Getting the pipeline structure and functionality right is really important. I've already made a bunch of tweaks, but I've been testing this by building pipelines so it's quite possible there's still more to do to perfect the structure. If you do try using it, I'd love to hear how that works out.

Then there's the code quality. I don't subscribe to this notion of not looking at LLM generated code. I've already done a lot of refactoring about how the code is structured and how specific bits work, but there's a lot more to do. The steps do provide isolation, but this needs reviewing to understand the security properties. The Goblins code is also new to me, so I'm excited about figuring out how that fits and what needs fixing or improving.

You can find the source on forge.cbaines.net, logging in to raise issues and pull requests requires a Codeberg account but please contact me if that's an issue. Please also contact me with any thoughts, comments or questions you have.