Writing Procedural Macros in Rust

Qumulo’s software has been written entirely in C for a while. In recent years we’ve been flirting with Rust and the many benefits it has to offer – I’ve written about this previously here.

Recently we’ve been working on writing our own LDAP client, and part of that has been writing our own ASN.1 BER serialization library. We’ve been able to provide an idiomatic way of generating serialization thanks to an often overlooked killer feature of Rust: macros.

Macros have been an element of many programming languages long before Rust was around. They tend to get evaluated before the normal compilation phase, and sometimes by a completely separate program (called a preprocessor). They are used to generate more non-macro code to be compiled by the normal compilation phase. Here at Qumulo we are quite familiar with macros from C, although C macros are very different from macros in Rust. C macros operate on principles of text-parsing whereas Rust macros operate on tokens.

// C macro
#define LOCATION() (“file: ” __FILE__)

// Rust macro
macro_rules! location {
    () => {
        concat!("file: ", file!())
    }
}

In fact there are two different types of macros in Rust: ones that are defined and run similar to C macros (shown above), and another kind called procedural macros. These macros are written in Rust itself (instead of a macro language). They are compiled into plugins (dynamic libraries) and run by the compiler when compiling. This means we can write significantly more complicated macros with ease. They power popular libraries like serde or rocket, and are the kind of macros we’ve used in our serialization library. They come in a couple types, I’m going to focus on derive procedural macros, which are generally used to automatically implement a trait for a type.

trait Hello {
    fn hello();
}

// Foobar implements the Hello trait now, and we didn’t have to write 
// an impl block!
#[derive(Hello)]
struct FooBar;

At the core, a procedural macro is just a function that takes a token stream as input, and gives a token stream as output. They tend to have two distinct phases that I’ll call a parse phase and a generate phase.

Since we are just given a stream of tokens, if we want to get any structural information about the tokens, we first have to parse them. This can be done using syn. During the parse phase, tokens are parsed into syn types, and then into any intermediate types of the macro.

During the generate phase, the types from the parse phase are leveraged to generate new code. The quote library can be used to do some templating and generate a token stream. Generally with derive procedural macros, they are outputting code that implements a trait.

use proc_macro::TokenStream;
use syn::{parse_macro_input, DeriveInput};
use quote::quote;

#[proc_macro_derive(Hello)]
pub fn derive_hello(input: TokenStream) -> TokenStream { 
    // Parse Phase
    let derive_input = parse_macro_input!(input as DeriveInput);
    let ident = &derive_input.ident;
    let name = derive_input.ident.to_string();
    
    // Generate Phase
    (quote! {
        impl Hello for #ident {
            fn hello() {
                println!("Hello from {}", #name);
            }
        }
    }).into()
}

Derive procedural macros can save a lot of work. Rather than implementing the same trait for many different types, we can write a macro that implements the trait for any type.

use ber::{Encode, Decode, DefaultIdentifier};

#[derive(Encode, PartialEq, Debug, DefaultIdentifier, Decode)]                                      
struct StructPrimitives {                                                                           
    x: u32,                                                                                         
    is_true: bool,                                                                                  
    negative: i32,                                                                                  
}                

#[test]
fn encode() {
    let s = StructPrimitives { x: 42, is_true: true, negative: -42 };
    
    // We have an encode method thanks to macros!
    let mut encoded = vec![];
    s.encode(&mut encoded).unwrap();
}

Our serialization library is able to encode the struct StructPrimitives even though we didn’t explicitly write the implementation of Encode. Thanks to the procedural macro, the library is able to write it itself.

Support by the compiler for procedural macros, as well as the great accompanying libraries like syn and quote, make writing them in Rust smooth. Procedural macros are one of the many reasons why I am happy we’ve chosen Rust. We’ve already leveraged them for a variety of uses, and I’m sure we’ll keep finding new uses.

Also! Mark Your Calendars! Please join us at Qumulo on Tuesday, Aug. 13 at 6:30 p.m. for the monthly Seattle Rust Meetup. Food and drink will be provided!

Qumulo’s Collin Wallace will be discussing the following: “Qumulo had a large C codebase with a bit of magic to allow for methods, interfaces, and a limited form of traits/generics. Now we have a mixed C+Rust codebase, and we’ve been picky that new Rust code be idiomatic *without* feeling out-of-place to the C code it links against. In this talk I’ll explore the path we used to achieve this, which leverages procedural macros, compiler plugins, auto-generated C headers, and some thoughtful approaches to testing.”

Contact us here if you’d like to set up a meeting or request a demo. And subscribe to our blog for more helpful best practices and resources!