javascript: Improve support for unterminated statements
Add support for implicit semicolons so many unterminated statements'
end are properly recognized.
The implementation doesn't follow the ECMAScript standard because doing
so requires to recognize precise grammar of all constructs, and the
parser doesn't currently work this way. So instead it uses some
heuristics that should work most of the time and only consider implicit
semicolons where they would be explicitly relevant to avoid most false-
positives. See the extensive comment in `readTokenFull()` for details.
In practice, this mostly fixes handling of files using unterminated
variable assignations like the following:
var v1 = 0
var v2 = 1
// ...
function f1() {
// ...
}
In such situations the parser used not to be able to really tell where
the variable assignation would end and would not recognize any
statement before the next semicolon or closing curly brace at the same
level. In practice, it wouldn't have emitted any tag for this example,
not even `v1` as it generates tags when reaching the statement's end.