Racket-Vulkan FFI Ready for First Tests

Previously racket-vulkan-types-done.htmlI wrote about C type declarations generated in Racket from the Vulkan Registry. By the time the article was done I still had work to do.

The project is now in a state where a generated FFI module can be required by an end-user to write a Vulkan application.

Will it work? Yes. On the first try? Lol, no. On all platforms? With luck. I cast a wide net but am targeting Linux and Windows first.

Here I continue my notes for processing vk.xml using Racket, with special focus on platform specifics. You’ll understand my vote of confidence afterwards.

Platform Type Haunt

Despite the efforts from my last update, I was not totally done with data types. Platforms came to haunt me in time for Halloween.

How Long is a Long Long?!?

Here’s a selection of some constants from vk.xml that are in an tag, but do not constitute a C enumerated type:

<enum value="(~0U)" ...>
<enum value="(~0ULL)" ...>
<enum value="1" ...>
<enum value="0" ...>
<enum value="(~0U-1)" ...>
<enum value="(~0U-2)" ...>
<enum value="256" ...>

There are C numeric literals in the value attributes. I need to translate these to Racket values so that users can write them as they would in an equivalent C program.

Racket has this super cool number type system and all, but no obvious way to parse the (~0...) literals.

In C, ~0 is the bitwise complement of 0, which is a bitstring of 1s. The parentheses are there to enforce order of operations with surrounding C code, since these values would just drop in somewhere with a #define. The -1 and -2 are just “minus 1” and “minus 2”, so nothing odd there.

The letters? Those scare me.

The capital U means “unsigned”, and the L means “long”. LL means “long long”. These tell us know how many bits we’re talking about, but only in the abstract.

The C99 standard (§5.2.4.2.1) defines longs as being at least 32 bits, and defines long longs as being at least 64 bits. I can’t just say how many bits to use, because that’s for a client system to decide.

To make matters worse, Racket’s bitwise-not doesn’t let you configure bits or if the output is signed. (bitwise-not 0) is -1 and if you don’t like it you can go get your socks wet.

integer-bytes->integer is the only procedure I see that can work with this scenario, since it lets you control the number of bits and the sign. It maxes out at 64-bits though, so I’m not sure if there’s any plans to have integer-bytes->integer cover cases of “at least 64” bits for long longs.

Unlike the other cases, I ended up writing a mini-parser for vk.xml’s XML-attribute-embedded C. Pay close attention to the quote depth:

(define (compute-~0-declaration literal)
(define sub-op/match (regexp-match* #px"-\\d" literal))
(define sub-op (if (empty? sub-op/match)
                   0
                   (abs (string->number (car sub-op/match)))))

`(- (integer-bytes->integer
      (make-bytes
        (ctype-sizeof ,(if (string-contains? literal "LL") _llong _long))
        255)
      ,(string-contains? literal "U"))
    ,sub-op))

I apply make-bytes to the size of the expressed type, assuming each byte is 255—that is, all 1s—because the procedure is for complements of 0 in the first place.

So, (~0ULL-7) produces the following code that runs on your machine when you (require vulkan/unsafe):

(- (integer-bytes->integer (make-bytes (ctype-sizeof _llong) #t) 7)

All of these moving parts handle a total of nine relevant values in vk.xml to date. Based on the C standard, this going to break on the system where (ctype-sizeof _llong) returns something greater than 8.

Am I done? Can I write a compute pipeline yet? Please?

Third-Party Blind Spots

In my last article I wrote about how I “symbolically” declared types to get around Racket being unable to declare recursive C types within the forms provided by ffi/unsafe.

That worked until I started declaring functions to look up from the Vulkan binary. After trying to run my generated code it complains about VisualID not being defined. That was a struct member from an extension, so let’s assess the damage.

<param><type>VisualID</type> <name>visualID</name></param>

Yeah, an X11 type, so wha—ooooh noooo.

Racket’s ffi/unsafe module insists that I have a complete type declaration to look up a function of a certain signature. Since I don’t have the original type definition, AND the parameter is not a pointer, I can’t declare the function for this.

But I’m not generating X11 bindings, and I can’t start recursively visiting C headers now! Not without a C parser and the scope increase that brings.

This problem can occur as many times as there are elements of form that are NOT used as pointers in Vulkan APIs. I count fourteen cases as of my mirror of vk.xml. Not terrible, but I don’t have a future-proof solution.

So I opted to seek out the headers, read them, and write the FFI bindings by hand.

I got spooked by the apparant system specifics in the other headers. From X.h:

#ifndef _XSERVER64
// ...
typedef unsigned long VisualID;
#else
// ...
typedef CARD32 VisualID;
#endif

Dante’s Inferno immediately flashed across my mind.

Dinanzi a me non fuor cose create se non etterne, e io etterno duro. Lasciate ogne speranza, voi ch’intrate’.

I have to figure out Racket code to alias some basic type based on knowledge of X11’s code, which vk.xml doesn’t care about because Vulkan can just drop an #include and call it a day.

And I still can’t prescribe a type up front. I need to base the type definition on specifics of the client system. I cannot use (get-ffi-obj) to look up specifics of a typedef.

This is going to take forever, isn’t it?

Actually, no. Through all the preprocessor directives and typedefs, this took only a few minutes on Startpage to resolve. For now. It’s still scary that vk.xml doesn’t include platform type hints, but it turns out that I probably can’t expect them to do so. Two of the types are actually behind an NDA for the sake of Stadia. So, they’re void* as far as I know or care. A Stadia developer probably isn’t going to use Racket anytime soon, but I can’t publish a correction for hobbyists. If I let signatories provide the types themselves, then I have to sanitize their code and—forget it, its just going to be void* until someone twists my arm.

The completionist in me is still screaming for a C parser to chew on all headers/libraries for each relevant platform, and have an elegant interlaced system to embed relevant parsed data into client-side code that finally picks a Racket-C type.

The pragmatist in me—born from overexposure to enterprise software—says to admit defeat and hand-write casework for each type.

The pragmatist won. No time to figure out if he was a shoulder angel, assuming he cared if I knew. Thankfully this resolved quickly enough after the initial anxiety.

Feature Idea: Platform Queries

After some time in Wonderland I am now seeing two general ways to deliver bindings from vk.xml:

  1. Dump every last one of them without concern for platform, version, or deprecation notices all over your nice shoes and tell you to go play.

  2. Start from a selection of and elements, collect the names they declare, and produce bindings for only the names discovered from there.

I opted for the former initially out of ignorance of how and would play into my program. Now, I keep doing so for short-term convenience. This means that even on the Fedora 30 system on which I’m writing this article, it will have access to bindings for Win32, Android, X (+ Xlib, XRandR), GGP, IOS, Metal, Fuchsia, Nintendo Vi, MacOS, Wayland and Google Games. The bloat isn’t terrible enough to make me hate myself, but still present. I’m okay if the pile comes up to your ankles, but no higher. I have a reputation to uphold!

Problem is, when Racket tries to look up certain symbols from a Vulkan library, it’s going to fail. Thankfully, ffi/unsafe/define offers a brilliant make-not-available procedure that will only raise an error if a developer tries to actually use something that doesn’t exist in the Vulkan library.

Without this feature, I would have to refactor heavily. This is, however, not an excuse to put off implementing the second option indefinitely. It makes sense now to generate several FFI modules depending on configuration profiles. A hypothetical raco command could function as follows:

$ raco vulkan generate-ffi \
--feature "VK_VERSION_1_0" \
--extension "VK_KHR_display_swapchain" \
--platform "android" > android/unsafe.rkt

Yeah… that’s cool. I should do this later.

Feature Idea: Published X-Expression Registry

I can use vk.xml to generate validation logic, which is what makes it more helpful than the C headers alone.

So let’s take a look at the, uh, LaTeX?

<member len="latexmath:[\textrm{codeSize} \over 4]"
        altlen="codeSize / 4">
const <type>uint32_t</type>*            <name>pCode</name></member>

I can’t complain about mixed code in general. I mean, I use Racket.

But I had to do a double take here. I was really scared I had to go down another rabbit hole.

Thankfully the altlen attribute is present to show the same expression using C, so people like me don’t have to handle XML, C, and LaTeX to generate validation code. I think we’re looking at the compromise from an angry meeting.

Still, I’m uneasy since this further legitimizes an XML/C hybrid as an acceptable means to write a API specification of importance to our industry. It only takes a cursory glance at the Vulkan-Docs repository to see why an official shift to a new registry format would make heads roll. And yes, I bothered to ask.

From @oddhack’s answer in the linked thread, I think Vulkan adoption would pick up speed if volunteers maintain unofficial registry formats for different purposes. KhronosGroups’ toolchain is obviously the primary consumer, which biases the XML to express content like the above element.

How much value would racket-vulkan contribute to the Lisp community at large if it dumped a curated, unambiguous S-expression of vk.xml’s contents for other Lisps to munch on? Doing this is not in scope for racket-vulkan, but I could feasibly spin off a Vulkan registry package in Racket dedicated to reformatting vk.xml.

Next Steps

Now we get to the good stuff: Writing a Vulkan application in Racket. I found a nice example by @Erkaman on GitHub that renders the Mandelbrot set. I intend to port that over and use it as a baseline to introduce much deserved sugar, like letting the Racket garbage collector manage Vulkan values, and letting ffi/unsafe/define check error return codes for me. Those two changes alone would slash the size of Vulkan apps.

Problem is, I’ve never worked with the Racket garbage collector like that before. Might take me a minute.

Conclusion

This has been a roller coaster. So many details and nuances came from one file! I’ve covered the remaining challenges in generating a valid FFI for Vulkan that can be used by a client, and how we’re now in a state to start testing.

The Racket package index is still choking on an earlier failed build, but I invite you to take a look at the source code and contact me with any questions or feedback.

With any luck, we might be able to make video games with this thing one day.