Defining Large, Functional Runtime Configurations for Racket Programs

My less-trivial Racket programs need a runtime configuration built from many sources. This article covers a unified, functional, and opinionated approach to modeling runtime configurations using Racket parameters. This might not be useful to you if you have a small handful of settings in your program, but it is indispensible for programs with hundreds or thousands of settings and related tests.

I'll start in the context of command line interfaces, then extend the code using runtime configuration files and environment variables. I'll end with a complete example, a design exercise, and a small library for you to copy and adapt for your project.

Delaying Dynamic Bindings

In Racket, a parameter is a dynamic binding. That is, a binding determined at runtime. Since our goal is to define a runtime configuration, we can use parameters to bind values used to control our program. Parameters are also useful for creating tests because of Racket's parameterize form, which maintains specifics bindings for parameters within the form.

(define foo (make-parameter 0))

(foo) ; 0
(foo 1) ; Rebind to 1, imperatively.
(foo) ; 1

; The output of this procedure changes with foo
(define (add-2) (+ 2 (foo)))

(add-2) ; 3
(parameterize ([foo 5]) (add-2)) ; 7
(add-2) ; 3 again, because we left the parameterization

; 9
(parameterize ([foo 0])
  (+ (add-2)
     (parameterize ([foo 5])
       (add-2))))

Loosely speaking, a parameterization is the state for every parameter at some moment in a Racket program. When I say “runtime configuration”, I am referring to a Racket parameterization.

Parameters alone don't make managing runtime configurations easy, because we are dealing with several eligible values. Do we bind to a value from a command line flag, a runtime configuration file, an environment variable, or a hard-coded default? Precedence rules help, and we'll get there in a moment, but the most pressing issue is that we probably won't have all of our possible values available to make a decision until some point in the future of our runtime.

One way to make this work is to defer when we create a dynamic binding. We can create an anonymous procedure to represent this easily enough.

(define (defer-dynamic-binding param value)
  (λ (continue)
    (parameterize ([param value])
      (continue))))

This procedure returns another procedure that binds a parameter to a prescribed value, before using a callback. In that sense, it's just a planned use of parameterize.

(define foo (make-parameter 0))
(define (add-2) (+ 2 (foo)))
(define call-with-eight (defer-dynamic-binding foo 8))
(call-with-eight add-2) ; 10

This particular form is important for a couple of reasons. First, we would not want to use parameterize in something like a flag handler for parse-command-line or command-line, because program control is not in a helpful place.

#lang racket

(module+ main
  (define value (make-parameter "not set"))

  (command-line
   #:once-each
   [("-v")
    user-value
    "set custom value"
    (parameterize ([value user-value])
      ; uhhhhh, what do I do here?
      (void))]
   #:args () (displayln (value)))) ; always prints "not set"

Some people handle this by using imperative code to re-bind a configuration value like so:

#lang racket

(module+ main
  (define value (make-parameter "not set"))

  (command-line
   #:once-each
   [("-v")
    user-value
    "set custom value"
    (value user-value)]

   ; can print "not set" or something else
   #:args () (displayln (value))))

This works, but it's not a clean solution. You cannot rely on the parameters to stay bound to a reasonable default when running tests on this code. You'd need to use parameterize or more imperative code to compensate. I ask you to trust me when I say that's a pain in the neck.

With defer-dynamic-binding, this isn't a problem because the actual act of binding the parameter can be saved for the relevant part of the program.

Interestingly enough, command-line is no longer suitable for our program because it does not give you a binding to data gathered from user-defined flags. parse-command-line does, so we'll use an equivalent call to it instead.

#lang racket

(module+ main
  (define value (make-parameter "not set"))

  (parse-command-line
   "my-program"
   (current-command-line-arguments)
   `((once-each
      [("-v")
       ,(λ (flag user-value) (value user-value))
       ("user-value" "set custom value")]))
   (λ (deferred-bindings)
     (displayln (value)))
   null))

This program behaves the same way, but we now have an argument I called deferred-bindings in the procedure that handles our program's processing steps. As per the documentation for parse-command-line, it is bound to a list of values accumulated from command line flags. In this imperative example, that list will either be empty because no flags are set, or a list containing only a (void) because (value user-value) is an expression evaluated for a side-effect—meaning it returns (void).

Now let's incorporate a deferred dynamic binding to remove the imperative code.

#lang racket

(define (defer-dynamic-binding param value)
  (λ (continue)
    (parameterize ([param value])
      (continue))))

(module+ main
  (define value (make-parameter "not set"))

  (define (show)
    (displayln (value)))

  (parse-command-line
   "my-program"
   (current-command-line-arguments)
   `((once-each
      [("-v")
       ,(λ (flag user-value)
          (defer-dynamic-binding value user-value))
       ("user-value" "set custom value")]))
   (λ (deferred-bindings)
     (if (null? deferred-bindings)
         (show)
         ((car deferred-bindings) show)))
   null))

This program behaves the same way as the imperative example, except now we use deferred-bindings to change parameterizations. If a deferred dynamic binding exists, we actually apply that binding and then show the value.

The code is harder to read, but easier to test. You can create a list of deferred bindings as a pure function of command line arguments, with no persistent change in shared state between calls.

Applying Multiple Dynamic Bindings

This program is obviously limited to just one binding. Let's define a way to apply all of our deferred bindings at once.

(define (call-with-runtime-configuration fs continue)
  ((foldl (λ (next accumulated)
            (λ () (next accumulated)))
           continue
           (reverse fs))))

call-with-runtime-configuration is just a patterned way to call multiple procedures. It assumes that each procedure in fs accepts one argument, but continue accepts no arguments. Remember that the anonymous procedure returned from defer-dynamic-binding accepts one procedure to call in the context of a parameterize form. The net effect is that you can use call-with-runtime-configuration like parameterize, such that one callback procedure runs in the dynamic context of many other procedures. We'll cover why we use reverse in a moment.

Let's take it from the top, except now we'll use two parameters.

#lang racket

(define (defer-dynamic-binding param value)
  (λ (continue)
    (parameterize ([param value])
      (continue))))

(define (call-with-runtime-configuration fs continue)
  ((foldl (λ (next accumulated)
            (λ () (next accumulated)))
           continue
           fs)))

(module+ main
  (define foo (make-parameter "foo not set"))
  (define bar (make-parameter "bar not set"))

  (define (show)
    (displayln (foo))
    (displayln (bar)))

  (parse-command-line
   "my-program"
   (current-command-line-arguments)
   `((once-each
      [("-f")
       ,(λ (flag user-foo)
          (defer-dynamic-binding foo user-foo))
       ("foo-value" "set foo")]
      [("-b")
       ,(λ (flag user-bar)
          (defer-dynamic-binding bar user-bar))
       ("bar-value" "set bar")]))
   (λ (deferred-bindings)
     (call-with-runtime-configuration deferred-bindings show))
   null))

This program does what we want, in the sense that setting the flag changes what gets printed. But now we have a system for managing parameters that doesn't have to deal with a command line interface at all, and that can be extended with different sources by adding their deferred bindings to the deferred-bindings list. The bindings that appear later in the list override those that appear earlier in the list.

(call-with-runtime-configuration
 (append from-envvars from-rcfile from-cli)
 ...)

In this section we covered a way to load a runtime configuration at a relevant time using a command line interface. But there's more to do. We need to add better validation, and more sources of configuration.

Making Smarter Configurations

We have a way to defer loading the runtime configuration, but we don't have any protection against loading the wrong value into a parameter. I'll use guard procedures because they allow control over the final value bound to a parameter. Since configuration sources will likely produce strings, some level of type coercion is appropriate.

(require racket/tcp)

(define (expect-listen-port v)
  (cond [(listen-port-number? v) v]
        [(string? v) (expect-listen-port (string->number v))]
        [else (raise-user-error 'port
                                "~v is not a valid listen port number"
                                v)]))

(define port (make-parameter 80 expect-listen-port))

This plays well with a corollary of the earlier program: Any error in loading a runtime configuration can now be raised from call-with-runtime-configuration. This helps us trace errors back to our configuration sources from a common code path. The act of deferring bindings won't raise validation errors, but a parameter guard will complain the moment we try to actually put the wrong value in its parameter.

We're not done. The parameters we've been creating are bound to identifiers in a namespace for which we don't have a reference. We can get that reference, but it's easier to organize our parameters in a dictionary value type like a hash table.

(define parameters
  (hasheq 'port (make-parameter 80 expect-listen-port)))

This makes it easy to dump the current configuration to a file.

(define parameters
  (hasheq 'port (make-parameter 80 expect-listen-port)))

(define (dump-runtime-configuration parameters)
  (for/hash ([(key param) (in-dict parameters)])
    (values key (param))))

(call-with-runtime-configuration deferred-bindings
 (λ () (writeln (dump-runtime-configuration parameters))))

Adding a Configuration File

To make things more interesting, we will include a runtime configuration file (“rcfile”) as a source for values. We'll also cover how to provide those values in addition to command-line flags.

Many files in your Racket installation actually use Racket literal values for configuration, which is handy. We can write a text file containing just an associative list...

((foo . 1)
 (bar . 2))

...or a hash table...

#hash((foo . 1)
      (bar . 2))

...and read either one into memory like this:

#lang racket

(define rc (file->value "rcfile"))

(dict-ref rc 'foo) ; 1
(dict-ref rc 'bar) ; 2
(dict-ref rc 'does-not-exist 3) ; 3

This works because associative lists and hash tables both count as dictionary types. We can read the file and compare the keys with those in our parameter dictionary. This helps whenever we extend our configuration using sources that act as dictionaries.

(define (defer-dynamic-bindings/dict parameter-dict config-dict)
  (for/fold ([rc null])
            ([(key val) (in-dict config-dict)])
    (if (dict-has-key? parameter-dict key)
        (cons (defer-dynamic-binding
                (dict-ref parameter-dict key)
                val)
              rc)
        rc)))

We can now amend our program as follows:

#lang racket

(require racket/tcp)

(define (expect-listen-port v)
  (cond [(listen-port-number? v) v]
        [(string? v) (expect-listen-port (string->number v))]
        [else (raise-user-error 'port
                                "~v is not a valid listen port number"
                                v)]))


(define (defer-dynamic-binding param value)
  (λ (continue)
    (parameterize ([param value])
      (continue))))


(define (defer-dynamic-bindings/dict parameter-dict config-dict)
  (for/fold ([rc null])
            ([(key val) (in-dict config-dict)])
    (if (dict-has-key? parameter-dict key)
        (cons (defer-dynamic-binding
                (dict-ref parameter-dict key)
                val)
              rc)
        rc)))


(define (call-with-runtime-configuration fs continue)
  ((foldl (λ (next accumulated)
            (λ () (next accumulated)))
           continue
           (reverse fs))))


(module+ main
  (define parameters
    (hasheq 'port (make-parameter 80 expect-listen-port)))

  (parse-command-line
   "my-program"
   (current-command-line-arguments)

   ; Flag table
   `((once-each
      [("-p" "--port")
       ,(λ (flag user-port)
          (defer-dynamic-binding
            (dict-ref parameters 'port)
            user-port))
       ("port" "Port used to listen for connections")]))

   ; Main program
   (λ (from-cli)
     (define from-rcfile
       (if (file-exists? "rcfile")
           (defer-dynamic-bindings/dict
            parameters
            (file->value "rcfile"))
           null))

     (call-with-runtime-configuration
      (append from-rcfile from-cli) ; flags can override rcfile
      (λ () (displayln ((dict-ref parameters 'port))))))

   null))

Notice that in this version of the program, the bindings from the rcfile are applied before the bindings applied by the command-line interface.

This why we used reverse in call-with-runtime-configuration: It makes the arguments more readable. The dynamic bindings that appear after all others get the final say, which means they are actually the first to wrap the main logic of our program when used in foldl.

If the rcfile and a command line flag both specify a port, the command line flag's value wins. To change the precedence rules, just rearrange from-rcfile and from-cli.

Adding Environment Variables

Environment variables will challenge our design in new ways. Let's start by converting an environment variable set into a hash table:

(define (envvars->hash env)
  (for/fold ([acc (hasheq)])
            ([name (in-list (environment-variables-names env))])
    (hash-set acc
              (string->symbol (bytes->string/utf-8 name))
              (environment-variables-ref env name))))

This allows us to extend our runtime configuration further.

(define parameters
  (hasheq 'port 8080 'listen-addresses "127.0.0.1"))

(define from-rcfile
  (defer-dynamic-bindings/dict
   parameters
   (file->value "rcfile")))

(define from-envvars
  (defer-dynamic-bindings/dict
   parameters
   (envvars->hash (current-environment-variables))))

(call-with-runtime-configuration
 (append from-envvars from-rcfile)
 (λ () (dump-runtime-configuration parameters)))

Unfortunately, our abstraction is starting to leak. We don't want to couple our system's environment variables to a bunch of collidable runtime configuration symbols. If we use a program-specific prefix like MY_PROGRAM_, then we have a new namespace with no relationship to the keys used to look up parameters.

We can address this by changing our keys to be less collidable.

(define parameters
  (hasheq 'port
          (make-parameter 80 expect-listen-port)))

(define parameters
  (hasheq 'MY_PROGRAM_PORT
          (make-parameter 80 expect-listen-port)))

This only mitigates the risk of collisions with environment variable names, but we can now modify the program to allow canonical names from all sources. This way, precedence rules decide what value wins.

$ echo '((MY_PROGRAM_PORT . 100))' > rcfile
$ MY_PROGRAM_PORT=8080 racket program.rkt --MY_PROGRAM_PORT 90

Outside of the names, we now have more moving parts in our configuration values. In our command line flags, each configuration value is a string that may have been modified by shell features. In environment variables, we have values that start as byte strings affected by the user's locale. Finally, the rcfiles we've covered are assumed to hold Racket literals. This means that envvars and CLI flags start as strings that need to become Racket values, whereas rcfiles need no work beyond reading. The guard procedures help with this, but what is the best way to normalize configuration values?

Designing Around Shifting Value Types

We've created a configuration space that can be controlled by a CLI, an rcfile, and environment variables. The problem is that each one of these sources can have their own syntax or conventions for usability reasons.

Have you ever seen command line flags like -gf, where -f turns something on, but -g turns something off? How about environment variables or rcfiles where a boolean true appears as the strings "on" or "yes"?

Of course you have.

I can't tell you how to handle this human element for your programs, and that is the design exercise I leave to you.

What I can tell you is that I write a lot of libraries for Racket programmers, so I set it up where some settings that start as a string must be readable Racket literals. My configuration for booleans might look something like this:

$ echo '((MY_PROGRAM_BOOLEAN . #t))' > rcfile
$ MY_PROGRAM_BOOLEAN='#f' racket program.rkt --MY_PROGRAM_BOOLEAN '#t'

I allow short command line flags, but they must come with Racket literals for arguments. This is unconventional because short command line flags for booleans normally omit arguments, so that they can be combined with other short flags.

$ echo '((MY_PROGRAM_BOOLEAN . #t))' > rcfile
$ MY_PROGRAM_BOOLEAN='#f' racket program.rkt -b '#t'

This wouldn't be the case for my smaller programs because this would be a pain to type all of the time, but I sometimes make this sacrifice for consistency, predictability, and flexibility reasons.

The good news is that every source of configuration for such a program can completely define a runtime configuration with the exact values that would be bound in memory. The bad news is that the user must know what a Racket literal is, and they need to type a value after every flag. But so long as my audience for a program is a Racket programmer, I suspect I can get away with that.

Now that you have a better idea of how to add sources to a list of deferred bindings, you can spend more time thinking about what conventions make sense for your audience.

Conclusion

I've shown you how to defer dynamic bindings in Racket, so that you can build runtime configurations for easily testable programs without unwelcome side-effects. We've moved on to review how rcfiles and environment variables can extend this system.

If you want to use the procedures we've discussed as a library, please copy the below module code for use in your projects and modify it as you see fit.

#lang racket/base

(require racket/contract
         racket/dict)

(provide
 (contract-out
  [dump-runtime-configuration
   (-> dict? hash?)]
  [defer-dynamic-binding
   (-> parameter? any/c (-> (-> any) any))]
  [defer-dynamic-bindings/dict
   (-> dict? dict? (listof (-> (-> any) any)))]
  [call-with-runtime-configuration
   (-> (listof (-> (-> any) any)) (-> any) any)]))


(define (dump-runtime-configuration parameters)
  (for/hasheq ([(key param) (in-dict parameters)])
    (values key (param))))


(define (defer-dynamic-binding param value)
  (λ (continue)
    (parameterize ([param value])
      (continue))))


(define (defer-dynamic-bindings/dict parameter-dict config-dict)
  (for/fold ([rc null])
            ([(key val) (in-dict config-dict)])
    (if (dict-has-key? parameter-dict key)
        (cons (defer-dynamic-binding
                (dict-ref parameter-dict key)
                val)
              rc)
        rc)))


(define (call-with-runtime-configuration fs continue)
  ((foldl (λ (next accumulated)
            (λ () (next accumulated)))
           continue
           (reverse fs))))


(module+ test
  (require rackunit)

  (define a (make-parameter 0))
  (define b (make-parameter 1))
  (define c (make-parameter 2))
  (define h (hasheq 'a a 'b b 'c c))
  (test-equal? "dump-runtime-configuration"
               (dump-runtime-configuration h)
               (hasheq 'a 0 'b 1 'c 2))

  (test-case "defer-dynamic-binding"
    (define (add-2) (+ 2 (a)))
    (check-= (add-2) 2 0)
    (check-= ((defer-dynamic-binding a 7) add-2) 9 0))

  (test-equal? "call-with-runtime-configuration + defer-dynamic-bindings/dict"
               (call-with-runtime-configuration
                (defer-dynamic-bindings/dict h (hasheq 'x "should not appear" 'b 10))
                (λ () (dump-runtime-configuration h)))
               (hasheq 'a 0 'b 10 'c 2)))