markup for pstream token types is fragile
This seems to be the root of the issue I encountered while adding in dictionary macros. The additional
MACRO enumeration value (placed immediately after
WORD) means that the indices of the subsequent
tokenType enumerations increase by one. This made
DOUBLE_SCALAR receive a value of 9.
UOPstream::write(double) this enumeration value is written as a character, which unfortunately corresponds to Tab and thus gets swallowed as a whitespace character. After this, the receiving end hasn't much chance.
I think that we should be invoking
writeToBuffer(char) directly instead of using things like
write(char(token::DOUBLE_SCALAR)) - should be more reliable.