Guile Knots is a library providing higher-level patterns and building blocks for programming with Guile Fibers.

I started developing it back in 2024, extracting code I'd written for the Guix Data Service and Guix Build Coordinator, making it possible to use this code in other services like the Bffe, Nar Herder and QA Frontpage.

I started using Guile Fibers first for the Guix Data Service all the way back in 2019, and that was just because I copied the initial code from the Mumi project.

There have definitely been some problems I've hit along the way, but using Fibers seems like a good fit for the software I've been writing in the past few years, and the functionality in Guile Knots is an essential part of that.

To call out just some of the included functionality, the resource pools are a versatile tool, which can be used for database connections or limiting parallelism to give just a couple of examples.

(let ((connection-pool
       (make-resource-pool
        (lambda ()
          (open-postgresql-connection
           "web"
           postgresql-statement-timeout))
        (floor (/ postgresql-connections 2))
        #:name "web"
        #:idle-seconds 30
        #:destructor
        (lambda (conn)
          (close-postgresql-connection conn "web"))
        #:add-resources-parallelism
        (floor (/ postgresql-connections 2))
        #:default-max-waiters (floor (/ postgresql-connections 2))
        #:default-checkout-timeout (/ postgresql-statement-timeout
                                      1000))))

The thread pools are useful when threads are required, useful for interacting with SQLite for example.

(let ((reader-thread-pool
       (make-fixed-size-thread-pool
        (min (max (current-processor-count)
                  32)
             128)
        #:thread-initializer
        (lambda ()
          (let ((db
                 (db-open database-file #:write? #f)))
            (sqlite-exec db "PRAGMA busy_timeout = 5000;")
            (list db)))
        #:thread-destructor
        (lambda (db)
          (sqlite-close db))
        #:thread-lifetime 50000
        #:name "db read")))

Using suspendable ports you can do reliable I/O, and Guile Knots includes with-port-timeouts which supports timeouts both for plain suspendable ports and for fibers.

(with-port-timeouts
 (lambda ()
   (http-get ...))
 #:timeout 20)

The parallelism procedures and promises make it easier and safer to take advantage of the parallelism that fibers open up, and the exception and backtrace handling through print-backtrace-and-exception/knots helps to address the issues that this causes for debugging.

(fibers-let
    ((package-derivations
      (with-resource-from-pool (connection-pool) conn
        (package-derivations-for-branch conn
                                        (string->number repository-id)
                                        branch-name
                                        system
                                        target
                                        package-name)))
     (build-server-urls
      (call-with-resource-from-pool (connection-pool)
       select-build-server-urls-by-id)))

Then there's the web server. One key motivation for this was to have a web server implementation which didn't read the request body into memory. Not doing this makes it possible to write a web server that receives requests with large amounts of data (e.g. file uploads), which was a key requirement in the Guix Build Coordinator. The knots web server expects to be started from fibers, rather than calling run-fibers itself, which makes it possible to run multiple servers on different ports as well as doing initial setup in fibers before starting the server.

I've also worked on support for chunked transfer encoding for both request and response bodies, encoding handling improvements, buffering improvements and reliability improvements.

Thanks to the great Guile Documenta, there's documentation available online for Guile Knots.

If you have any comments or questions, please reach out to me via email at mail@cbaines.net, raise an issue on Forgejo, or contact me on the Fediverse.