Racket-Vulkan FFI Ready for First Tests
Previously racket-vulkan-types-done.htmlI wrote about C type declarations generated in Racket from the Vulkan Registry. By the time the article was done I still had work to do.
The project is now in a state where a generated FFI module can be required by an end-user to write a Vulkan application.
Will it work? Yes. On the first try? Lol, no. On all platforms? With luck. I cast a wide net but am targeting Linux and Windows first.
Here I continue my notes for processing vk.xml
using Racket,
with special focus on platform specifics. You’ll understand my vote of
confidence afterwards.
Platform Type Haunt
Despite the efforts from my last update, I was not totally done with data types. Platforms came to haunt me in time for Halloween.
How Long is a Long Long?!?
Here’s a selection of some constants from vk.xml
that are in an
tag, but do not constitute a C enumerated type:
<enum value="(~0U)" ...>
<enum value="(~0ULL)" ...>
<enum value="1" ...>
<enum value="0" ...>
<enum value="(~0U-1)" ...>
<enum value="(~0U-2)" ...>
<enum value="256" ...>
There are C numeric literals in the value
attributes. I need to
translate these to Racket values so that users can write them as they
would in an equivalent C program.
Racket has this super cool number type system and all, but no obvious
way to parse the (~0...)
literals.
In C, ~0
is the bitwise complement of 0, which is a bitstring
of 1s. The parentheses are there to enforce order of operations with
surrounding C code, since these values would just drop in somewhere
with a #define
. The -1
and -2
are just “minus 1”
and “minus 2”, so nothing odd there.
The letters? Those scare me.
The capital U
means “unsigned”, and the L
means
“long”. LL
means “long long”. These tell us know how many bits
we’re talking about, but only in the abstract.
The C99 standard (§5.2.4.2.1) defines longs as being at least 32 bits, and defines long longs as being at least 64 bits. I can’t just say how many bits to use, because that’s for a client system to decide.
To make matters worse, Racket’s bitwise-not
doesn’t let you
configure bits or if the output is signed. (bitwise-not 0)
is
-1
and if you don’t like it you can go get your socks wet.
integer-bytes->integer
is the only procedure I see that can
work with this scenario, since it lets you control the number of bits
and the sign. It maxes out at 64-bits though, so I’m not sure if
there’s any plans to have integer-bytes->integer
cover cases of
“at least 64” bits for long longs.
Unlike the other cases, I ended up writing a mini-parser for
vk.xml
’s XML-attribute-embedded C. Pay close attention to the
quote depth:
(define (compute-~0-declaration literal)
(define sub-op/match (regexp-match* #px"-\\d" literal))
(define sub-op (if (empty? sub-op/match)
0
(abs (string->number (car sub-op/match)))))
`(- (integer-bytes->integer
(make-bytes
(ctype-sizeof ,(if (string-contains? literal "LL") _llong _long))
255)
,(string-contains? literal "U"))
,sub-op))
I apply make-bytes
to the size of the expressed type, assuming
each byte is 255
—that is, all 1s—because the procedure is for
complements of 0
in the first place.
So, (~0ULL-7)
produces the following code that runs on
your machine when you (require vulkan/unsafe)
:
(- (integer-bytes->integer (make-bytes (ctype-sizeof _llong) #t) 7)
All of these moving parts handle a total of nine relevant values in
vk.xml
to date. Based on the C standard, this going to break on
the system where (ctype-sizeof _llong)
returns something
greater than 8
.
Am I done? Can I write a compute pipeline yet? Please?
Third-Party Blind Spots
In my last article I wrote about how I “symbolically” declared types
to get around Racket being unable to declare recursive C types within
the forms provided by ffi/unsafe
.
That worked until I started declaring functions to look up from the
Vulkan binary. After trying to run my generated code it complains
about VisualID
not being defined. That was a struct member from
an extension, so let’s assess the damage.
<param><type>VisualID</type> <name>visualID</name></param>
Yeah, an X11 type, so wha—ooooh noooo.
Racket’s ffi/unsafe
module insists that I have a complete type
declaration to look up a function of a certain signature. Since I
don’t have the original type definition, AND the parameter is not a
pointer, I can’t declare the function for this.
But I’m not generating X11 bindings, and I can’t start recursively visiting C headers now! Not without a C parser and the scope increase that brings.
This problem can occur as many times as there are elements of form
that are NOT used as
pointers in Vulkan APIs. I count fourteen cases as of my mirror of
vk.xml
. Not terrible, but I don’t have a future-proof solution.
So I opted to seek out the headers, read them, and write the FFI bindings by hand.
I got spooked by the apparant system specifics in the other
headers. From X.h
:
#ifndef _XSERVER64
// ...
typedef unsigned long VisualID;
#else
// ...
typedef CARD32 VisualID;
#endif
Dante’s Inferno immediately flashed across my mind.
Dinanzi a me non fuor cose create se non etterne, e io etterno duro. Lasciate ogne speranza, voi ch’intrate’.
I have to figure out Racket code to alias some basic type based on
knowledge of X11’s code, which vk.xml
doesn’t care about
because Vulkan can just drop an #include
and call it a day.
And I still can’t prescribe a type up front. I need to base the type
definition on specifics of the client system. I cannot use
(get-ffi-obj)
to look up specifics of a typedef
.
This is going to take forever, isn’t it?
Actually, no. Through all the preprocessor directives and typedefs,
this took only a few minutes on
Startpage to
resolve. For now. It’s still scary that vk.xml
doesn’t include
platform type hints, but it turns out that I probably can’t expect
them to do so. Two of the types are actually
behind
an NDA for the sake of Stadia. So, they’re void*
as far as I
know or care. A Stadia developer probably isn’t going to use Racket
anytime soon, but I can’t publish a correction for hobbyists. If I let
signatories provide the types themselves, then I have to sanitize
their code and—forget it, its just going to be void*
until
someone twists my arm.
The completionist in me is still screaming for a C parser to chew on all headers/libraries for each relevant platform, and have an elegant interlaced system to embed relevant parsed data into client-side code that finally picks a Racket-C type.
The pragmatist in me—born from overexposure to enterprise software—says to admit defeat and hand-write casework for each type.
The pragmatist won. No time to figure out if he was a shoulder angel, assuming he cared if I knew. Thankfully this resolved quickly enough after the initial anxiety.
Feature Idea: Platform Queries
After some time in Wonderland I am now seeing two general ways to
deliver bindings from vk.xml
:
Dump every last one of them without concern for platform, version, or deprecation notices all over your nice shoes and tell you to go play.
Start from a selection of
and
elements, collect the names they declare, and produce bindings for only the names discovered from there.
I opted for the former initially out of ignorance of how
and
would play into my
program. Now, I keep doing so for short-term convenience. This means
that even on the Fedora 30 system on which I’m writing this article,
it will have access to bindings for Win32, Android, X (+ Xlib,
XRandR), GGP, IOS, Metal, Fuchsia, Nintendo Vi, MacOS, Wayland and
Google Games. The bloat isn’t terrible enough to make me hate myself,
but still present. I’m okay if the pile comes up to your ankles, but
no higher. I have a reputation to uphold!
Problem is, when Racket tries to look up certain symbols from a Vulkan
library, it’s going to fail. Thankfully, ffi/unsafe/define
offers a brilliant make-not-available
procedure that will only
raise an error if a developer tries to actually use something
that doesn’t exist in the Vulkan library.
Without this feature, I would have to refactor heavily. This is, however, not an excuse to put off implementing the second option indefinitely. It makes sense now to generate several FFI modules depending on configuration profiles. A hypothetical raco command could function as follows:
$ raco vulkan generate-ffi \
--feature "VK_VERSION_1_0" \
--extension "VK_KHR_display_swapchain" \
--platform "android" > android/unsafe.rkt
Yeah… that’s cool. I should do this later.
Feature Idea: Published X-Expression Registry
I can use vk.xml
to generate validation logic, which is what
makes it more helpful than the C headers alone.
So let’s take a look at the, uh, LaTeX?
<member len="latexmath:[\textrm{codeSize} \over 4]"
altlen="codeSize / 4">
const <type>uint32_t</type>* <name>pCode</name></member>
I can’t complain about mixed code in general. I mean, I use Racket.
But I had to do a double take here. I was really scared I had to go down another rabbit hole.
Thankfully the altlen
attribute is present to show the same
expression using C, so people like me don’t have to handle XML, C,
and LaTeX to generate validation code. I think we’re looking
at the compromise from an angry meeting.
Still, I’m uneasy since this further legitimizes an XML/C hybrid as an acceptable means to write a API specification of importance to our industry. It only takes a cursory glance at the Vulkan-Docs repository to see why an official shift to a new registry format would make heads roll. And yes, I bothered to ask.
From @oddhack’s answer in the linked thread, I think Vulkan adoption
would pick up speed if volunteers maintain unofficial registry formats
for different purposes. KhronosGroups’ toolchain is obviously the
primary consumer, which biases the XML to express content like the
above
element.
How much value would racket-vulkan
contribute to the Lisp
community at large if it dumped a curated, unambiguous S-expression of
vk.xml
’s contents for other Lisps to munch on? Doing this is
not in scope for racket-vulkan
, but I could feasibly spin off a
Vulkan registry package in Racket dedicated to reformatting
vk.xml
.
Next Steps
Now we get to the good stuff: Writing a Vulkan application in
Racket. I found a
nice
example by @Erkaman on GitHub that renders the Mandelbrot set. I
intend to port that over and use it as a baseline to introduce much
deserved sugar, like letting the Racket garbage collector manage
Vulkan values, and letting ffi/unsafe/define
check error return
codes for me. Those two changes alone would slash the size of Vulkan
apps.
Problem is, I’ve never worked with the Racket garbage collector like that before. Might take me a minute.
Conclusion
This has been a roller coaster. So many details and nuances came from one file! I’ve covered the remaining challenges in generating a valid FFI for Vulkan that can be used by a client, and how we’re now in a state to start testing.
The Racket package index is still choking on an earlier failed build, but I invite you to take a look at the source code and contact me with any questions or feedback.
With any luck, we might be able to make video games with this thing one day.