The complete list of short-cuts for specifying a BNF are:

• [ things to parse go here ] : an optional list of things to parse.
• { things to parse go here “keyword” }** : a keyword separated list of things to parse.
• { one-thing “keyword” }* : a keyword separated list of one-thing to parse. This is only used if it is a top-level construct; if it is nested within an optional construct or another list construct, then it is treated as the previous case (two stars).

The values generated for these special constructs can no longer be just a simple return type, like in a standard BNF. Instead, pgen uses a list_t data structure.

For the optional construct, the list is either empty (when nothing was generated), or contains the optional items. To make this list homogenous, the parser generator insists that every return type must be able to be mapped into the default return type using special wrapper functions. These wrapper functions are used to embed any item into the default return type, and the wrapped data is inserted into the list.

For the second construct, the list contains wrapped items with an entry for every non-keyword item in the construct. The third construct is a special-case optimization of the second. In this case, the wrapper is omitted.

The wrapping requirement means that a user must provide wrapper functions for every built-in data type and return type. These functions are named in a specific way. The baseline wrapper functions that must always be provided are:

  ret *prefix_wrap_W_double (double d) { ... }
  ret *prefix_wrap_W_int (int i) { ... }
  ret *prefix_wrap_W_list (list_t *l) { ... }
  ret *prefix_wrap_W_string (char *s) { ... }

If an expression is used, then

ret *prefix_wrap_W_expr (Expr *e) { ... }

is also required. Here ret is the user-specified default return type, prefix is the user specified prefix used by pgen, and W is the walk type.