- previously simply read files until the input stream went bad and no
  more lines could be read.  With the more stringent checking of
  values read (commit 0ce7e364) this approach causes problems.

  Use the underlying tokenizer instead to decide about termination.