User:Cscott/Ideas/Wikitext Composition and Extension

I'm drifting increasingly off topic here, but note the separation between "parsing"/tokenization and evaluation/expansion. The contents of the <ref> are opaque to the tokenizer while the contents of the top level template arguments are not, but that turns out to be mostly a syntax issue regarding how delimiters are found and braces matched up. The "value" of {{#list|a|{{#list|b}}}} can be a linked list data structure, it doesn't "have" to be transmitted as a string. Similarly {{#raw:'''b'''}} doesn't have to have <b> tags in it's argument when the parser function gets it. Tokenizing the arguments helps identify delimiters in "human friendly" ways in corner cases -- aka forgetting to add a close }} or close ]] usually gives much nicer results than forgetting a closing </ref> which will happily suck in everything until it finds the next </ref> -- but fundamentally the arguments should be thought of as raw strings from the original source /not/ tokens, at least until the...