Recent posts

Introducing pmutil

DongYoon Kang

swc uses proc macros extensively. I use a procedural macro if possible to reduce boilerplates and to avoid rewriting boilerplates when something is changed. Behind the scene, there's a hero crate called pmutil. It provides useful stuffs for proc macros.

Features#

smart_quote!#

It's a rustfmt-friendly quasi quotter. It exploits the fact that rustfmt can format function-call-like macros.

It takes two arguments. The first argument is to list variables, and should be wrapped with Vars { }. You can think this as constructing a struct named Vars. It's really the same, as shorthand notation is supported. It means, you can pass

// type is not required in real code
let i: Ident = {
// ..
};
let v = {
// ..
};
smart_quote!(Vars {
i,
TypeName: &v,
}, {
// ignore second argument at now
});

Currently, Vars{} is required even if no variable is passed in. This restriction may be removed in the future.

The second argument is a template, wrapped with a group ((), [], {}). In most cases, wrapping with { and } is best.

smart_quote!(Vars {
i,
mtd_name: &method,
TypeName: &input.ident,
}, {
impl TypeName {
pub const fn mtd_name(&self) -> usize{
i
}
}
})

where the method is ident named len, i is an arbitrary value (we use 10 at here) and TypeName is the name of DeriveInput (we use Enum at here), expands to

impl Enum {
pub const fn len(&self) -> usize {
10
}
}

As rustfmt is happy with

  • function-call-like macro invokation
  • a block containing multiple items

you can do something like without breaking rustfmt.

smart_quote!(Vars {
is_body,
mtd_name: &method,
TypeName: &input.ident,
}, {
impl TypeName {
pub const fn mtd_name(&self) -> usize {
i
}
}
impl TypeName {
pub fn is(&self) -> bool {
is_body
}
}
})

Naming convention#

I recommend to follow the naming convention of template argument. It means, identifier of a type should be passed in PascalCase although it is passed as a field initialization.

Quote.parse() and error reporting#

You might think quasi quoting is enough and parsing quoted tokens to ast type is just waisting compile time. It's true if your project is small. But debugging proc macro is hard. I made 10+ proc macros and you should trust me. It's really hard.

Let's start with a simple wrong code.

use syn::Expr;
let expr: Expr = Quote::new(def_site::<Span>())
.quote_with(smart_quote!(
Vars {
},
{
impl Type {
}
}
))
.parse();

As you generated a wrong token (impl block is not an expression), Quote.parse() will panic, reports the line of smart_quote! invocations and tell you to rerun after setting an environment variable named DBG_DUMP to 1. When you run with it, Quote.parse() will print the tokens you quoted.

error: custom attribute panicked
--> ecmascript/codegen/src/decl.rs:7:5
|
7 | #[emitter]
| ^^^^^^^^^^
|
= help: message: Quote::parse() failed.
Note: quasi quotting was invoked from:
ecmascript/codegen/macros/src/lib.rs:54:25
Error from syn: expected expression
>>>>>
To get code failed to parse,
please set environment variable `DBG_DUMP` and run in again
<<<<<
error: aborting due to previous error
error: could not compile `swc_ecma_codegen`.

Yes, I broke code to show the message. With DBG_DUMP=1,

error: custom attribute panicked
--> ecmascript/codegen/src/decl.rs:7:5
|
7 | #[emitter]
| ^^^^^^^^^^
|
= help: message: Quote::parse() failed.
Note: quasi quotting was invoked from:
ecmascript/codegen/macros/src/lib.rs:54:25
Error from syn: expected expression
>>>>>
impl Type { }
<<<<<
error: aborting due to previous error
error: could not compile `swc_ecma_codegen`.

You can see the code, which is clearly not an expression, and fix it.

ToTokensExt#

If you have an experience of writing a proc macro, you may know proc_macro::TokenStream and proc_macro2::TokenStream interop badly, as name is equal to each other. ToTokensExt helps you.

It provides a .dump() method, which returns proc_macro2::TokenStream without bringing the name into scope.

// more imports
use proc_macro::TokenStream;
#[proc_macro_derive(Is, attributes(is))]
pub fn is(input: TokenStream) -> TokenStream {
let input: DeriveInput = syn::parse(input).expect("failed to parse derive input");
let generics: Generics = input.generics.clone();
let items = match input.data {
Data::Enum(e) => expand(e),
_ => panic!("`Is` can be applied only on enums"),
};
ItemImpl {
attrs: vec![],
defaultness: None,
unsafety: None,
impl_token: Default::default(),
generics: Default::default(),
trait_: None,
self_ty: Box::new(Type::Path(TypePath {
qself: None,
path: Path::from(input.ident),
})),
brace_token: Default::default(),
items,
}
.with_generics(generics)
.dump()
.into()
}

The method is also useful for debugging as proc_macro2::TokenStream implements Display while ast types from syn do not implement it.

Macros built with pmutil#

Lots of macros are built over pmutil.

If you want a macro below to be a standalone crate, please ping me on slack (slackin for swc) or file an issue at github.

(This is one of the reasons why I'm writing this section)

FromVariant (swc)#

#[derive(Debug, Clone, FromVariant)]
pub enum Expr {
Lit(Lit),
Ident(Ident),
}
// Generated impls:
impl From<Lit> for Expr {}
impl From<Ident> for Expr {}

It uses pmutil like

// In real code, ItemImpl is inferred
let from_impl: ItemImpl = Quote::new_call_site()
.quote_with(smart_quote!(
Vars {
VariantType: field.ty,
Variant: variant_name,
Type: &ident,
},
{
impl From<VariantType> for Type {
fn from(v: VariantType) -> Self {
Type::Variant(v)
}
}
}
))
.parse();

where VariantType is replaced with the type of the only field of the variant, Variant is replaced with the name of the variant, and Type is replaced with the name of the enum itself.

ast_node (swc)#

Alias for #[derive(Spanned, Fold, Clone, Debug, PartialEq)] for a struct and #[derive(Spanned, Fold, Clone, Debug, PartialEq, FromVariant)] for an enum.

Fold, Visit (swc)#

#[derive(Fold)]
pub struct Ident {
pub span: Span,
pub sym: String,
}
#[derive(Fold)]
pub struct Lit {
pub span: Span,
pub value: String,
}
#[derive(Fold)]
pub enum Expr {
Ident(ident),
Lit(Str)
}
// Generated impls:
impl ::swc_common::FoldWith for Ident {
// fn fold_children()
}
impl ::swc_common::VisitWith for Ident {
// fn visit_children()
}
impl ::swc_common::FoldWith for Lit {
// fn fold_children()
}
impl ::swc_common::VisitWith for Lit {
// fn visit_children()
}
impl ::swc_common::FoldWith for Expr {
// fn fold_children()
}
impl ::swc_common::VisitWith for Expr {
// fn visit_children()
}

With specialization, which requires nightly, it becomes an automagical weapon.

You can collect all identifiers in ast node with 20 lines of code however it is stored (unless you are using #[fold(ignore)]).

struct IdentCollector {
ids: Vec<Ident>
}
impl Visit<Ident> for IdentCollector {
fn visit(&mut self, i: &Ident) {
self.ids.push(i.clone());
}
}
pub fn find_ids<T: VisitWith<IdentCollector>>(node: &T) -> Vec<Ident> {
let mut v = IdentCollector { ids: vec![] };
node.visit_with(&mut v);
v.ids
}
assert_eq!(find_ids(box vec![Expr::Ident(i1), Expr::Lit(lit), Expr::Ident(i2)]) , vec![i1, i2]);

Spanned (swc)#

#[derive(Spanned)]
pub struct Ident {
pub span: Span,
pub symbol: String,
}
#[derive(Spanned)]
pub struct Lit {
pub span: Span,
pub value: String,
}
#[derive(Spanned)]
pub enum Expr {
Lit(Lit),
Ident(Ident)
}
// Generated impls:
impl ::swc_common::Spannned for Ident {}
impl ::swc_common::Spannned for Lit {}
impl ::swc_common::Spannned for Expr {}

enum_kind (swc)#

#[macro_use]
extern crate enum_kind;
/// You can split attributes if you want.
#[derive(Kind)]
#[kind(functions(is_a = "bool", is_b = "bool"))]
#[kind(functions(is_a_or_b = "bool", num = "u8"))]
pub enum E {
#[kind(is_a, is_a_or_b, num = "1")]
A,
/// You can split attributes if you want.
#[kind(is_b)]
#[kind(is_a_or_b)]
#[kind(num = "2")]
B(u8),
/// Default value of bool is false if not specified and true if specified.
///
/// Both struct like variant and tuple like variant are supported.
#[kind(num = "3")]
C {},
}
assert!(E::A.is_a() && E::A.is_a_or_b() && !E::A.is_b());
assert_eq!(E::A.num(), 1);
assert!(!E::B(0).is_a() && E::B(0).is_a_or_b() && E::B(0).is_b());
assert_eq!(E::B(0).num(), 2);
assert!(!E::C {}.is_a() && !E::C {}.is_a_or_b() && !E::C {}.is_b());
assert_eq!(E::C {}.num(), 3);

string_enum (swc)#

#[derive(StringEnum)]
pub enum Tokens {
/// `a`
A,
/// `bar`
B,
}
// Generated:
//
// pub fn as_str(&self) -> &'static str
// impl serde::Serilaize
// impl serde::Deserilaize
// impl FromStr
// impl Debug
// impl Display
assert_eq!(Tokens::A.as_str(), "a");
assert_eq!(Tokens::B.as_str(), "bar");
assert_eq!(Tokens::A.to_string(), "a");
assert_eq!(format!("{:?}", Tokens::A), format!("{:?}", "a"));

swc_ecma_parser_macros (swc)#

I'll show you the code and then tell how it works.

#[parser]
impl<'a, I: Tokens> Parser<'a, I> {
fn parse_yield_expr(&mut self) -> PResult<'a, Box<Expr>> {
let start = cur_pos!();
assert_and_bump!("yield");
debug_assert!(self.ctx().in_generator);
// Spec says
// YieldExpression cannot be used within the FormalParameters of a generator
// function because any expressions that are part of FormalParameters are
// evaluated before the resulting generator object is in a resumable state.
if self.ctx().in_parameters {
syntax_error!(self.input.prev_span(), SyntaxError::YieldParamInGen)
}
if is!(';') || (!is!('*') && !cur!(false).map(Token::starts_expr).unwrap_or(true)) {
Ok(Box::new(Expr::Yield(YieldExpr {
span: span!(start),
arg: None,
delegate: false,
})))
} else {
let has_star = eat!('*');
let arg = self.parse_assignment_expr()?;
Ok(Box::new(Expr::Yield(YieldExpr {
span: span!(start),
arg: Some(arg),
delegate: has_star,
})))
}
}
}

Did you get it? As Input is stored in Parser, you should pass an instance of Parser into macro, so code to consume a token should be

expect!(self, "yield");

But #[parser] injects self to the code automatically. It knows the list of macros that accept Parser as a first argument and inject self into the macro call.

swc_ecma_codegen_macros (swc)#

impl Emitter {
#[emitter]
pub fn emit_module(&mut self, m: &Module) -> Result {
// ..
}
}

becomes

impl Emitter {
pub fn emit_module(&mut self, m: &Module) -> Result {
// ..
}
}
impl Node for Module {
fn emit_with(&self, emitter: &mut Emitter) -> Result {
emitter.emit_module(self)
}
}

Thus emit! works for any emittable types. emit! is defined as

macro_rules! emit {
($emitter:expr, $e:expr) => {{
crate::Node::emit_with(&$e, $emitter)?;
}};
}

As the macro creates an impl block from a method, it reduced the amount of code drastically. (Writing impl Node for T every time is cumbersome.)

is-macro#

is-macro is a proc macro to generate methods for enum to allow using it like Option / Result.

use is_macro::Is;
#[derive(Debug, Is)]
pub enum Enum<T> {
A,
B(T),
C(Option<T>),
}
// Generated methods:
//
// is_a(&self), is_b(&self), is_c(&self)
// b(self) -> Option<T>, c(self) -> Option<Option<T>>
// expect_b(self) -> T, expect_c(self) -> Option<T>
//
// Rust's type inference cannot handle this.
assert!(Enum::<()>::A.is_a());
assert_eq!(Enum::B(String::from("foo")).b(), Some(String::from("foo")));
assert_eq!(Enum::B(String::from("foo")).expect_b(), String::from("foo"));

st-map#

This macro adds .iter(), .iter_mut(), .into_iter(), .map(|k, v| {}), .map_values(|v| {}) to a map-like structs. Map-like means that all fields have the same type.

#[derive(Debug, PartialEq, Default, StaticMap)]
struct BrowserData<T: Default> {
chrome: T,
safari: T,
android: T,
}
// Example of using generated methods.
{
// .iter(), .iter_mut(), .into_iter()
let mut data = BrowserData {
chrome: true,
safari: false,
android: true,
};
assert_eq!(
data.iter().collect::<Vec<_>>(),
vec![("chrome", &true), ("safari", &false), ("android", &true),]
);
assert_eq!(
data.iter_mut().collect::<Vec<_>>(),
vec![
("chrome", &mut true),
("safari", &mut false),
("android", &mut true),
]
);
assert_eq!(
data.into_iter().collect::<Vec<_>>(),
vec![("chrome", true), ("safari", false), ("android", true),]
);
}
{
// .map(), .map_values()
let data = BrowserData {
chrome: 20000,
safari: 10000,
..Default::default()
};
assert_eq!(
data.map_value(|v| v > 15000),
BrowserData {
chrome: true,
safari: false,
android: false,
}
);
}