disallow keywords as identifiers
commite2f815f8d0d5339f0f672d6c2193c8015871bcb0
authorCatherine Gasnier <catg@fb.com>
Thu, 30 Aug 2018 09:25:47 +0000 (30 02:25 -0700)
committerHhvm Bot <hhvm-bot@users.noreply.github.com>
Thu, 30 Aug 2018 10:07:24 +0000 (30 03:07 -0700)
treec297f46466b4431ecfec80ccd82456f6ba353957
parent019d7089516973200b4293569836ea00ed14ab68
disallow keywords as identifiers

Summary:
In PHP and Hack, some keywords are reserved in the sense that they cannot be used as identifier. For example, control flow keywords like `if` are reserved: when used as a function or class name, an error message like "unexpected token T_IF" is issued by php or HPHPC. On the other hand, other keywords, like `self` or `bool` or `attribute`, can usually be used as an identifier. To be more precise, this is what is allowed where:
* `function <name>() {}`: all non-reserved keywords are allowed for `<name>`
* `class <name> {}`: all non-reserved keywords are allowed for `<name>`, except for types like `bool`, in which case the error is not `Unexpected token T_BOOL` but `bool is reserved`.
* `const <name> = ...`: all keywords are allowed, even reserved ones like `if`
* `function <name>() {}` as a method definition: same, all keywords are allowed
* enums: all keywords are allowed
* as a consequence of these, all keywords, even reserved ones, are allowed after a `$`, `->` or `::` symbol, but in any other place where we used to call `next_token_as_name` (see below), we should actually not allow reserved keywords.

We use to have:
* `next_token` (get next token such that non-keywords words like `blah` have token kind `Name` while keywords like `bool` have the corresponding keyword token kind, like `Bool`)
* `next_token_as_name` (get next token, such that all words, including keywords, have token kind `Name` (but a symbol like `==>` would have token kind `Arrow` obviously))

So we now need a third alternative:
* `next_token_non_reserved_as_name`: get next token such that non-keywords words like `blah`, and non-reserved keywords like `bool`, have token kind `Name` while reserved keywords like `if` have the corresponding keyword token kind, like `If`.

Similarly, we used to have:
* `require_name`: get next token with `next_token` and check if it is a `Name`
* `require_name_allow_keyword`: get next token with `next_token_as_name` and check if it is a `Name` (IOW, just check we don't have a symbol with punctuation like `==>`)

We now need in addition:
* `require_name_allow_non_reserved`: get next token with `next_token_for_identifier` and check that we get a `Name`

Non-reserved keywords are flagged as `allowed_as_identifier` in `Full_fidelity_schema` (suggestions welcome for the name of this flag), so that the generated `Full_fidelity_token_kind.from_string` has additional guards. In terms of performance, this additional guard costs a boolean check each time a non-reserved keyword is lexed. Alternatively, we could simply run a full additional match on tokens each time we call `next_token_for_identifier`. It's not obvious to me which is less costly, nor whether that matters at all actually.

Reviewed By: jamesjwu

Differential Revision: D8768546

fbshipit-source-id: dca0d04debba7a14127ee675911c8c5bba3ecf86
73 files changed:
hphp/hack/src/generate_full_fidelity.ml
hphp/hack/src/naming/naming_special_names.ml
hphp/hack/src/parser/full_fidelity_ast.ml
hphp/hack/src/parser/full_fidelity_declaration_parser.ml
hphp/hack/src/parser/full_fidelity_expression_parser.ml
hphp/hack/src/parser/full_fidelity_lexer.ml
hphp/hack/src/parser/full_fidelity_lexer.mli
hphp/hack/src/parser/full_fidelity_lexer_sig.ml
hphp/hack/src/parser/full_fidelity_parser_errors.ml
hphp/hack/src/parser/full_fidelity_parser_helpers.ml
hphp/hack/src/parser/full_fidelity_statement_parser.ml
hphp/hack/src/parser/full_fidelity_syntax_error.ml
hphp/hack/src/parser/full_fidelity_syntax_error.mli
hphp/hack/src/parser/full_fidelity_token_kind.ml
hphp/hack/src/parser/full_fidelity_type_parser.ml
hphp/hack/src/parser/hack_grammar_descriptor_helper.ml
hphp/hack/src/parser/js/full_fidelity_editable.js
hphp/hack/src/parser/js/full_fidelity_schema.json
hphp/hack/src/parser/schema/full_fidelity_schema.ml
hphp/hack/test/full_fidelity/cases/keyword_as_class_allowed.php [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_class_allowed.php.errors.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_class_allowed.php.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_class_reserved.php [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_class_reserved.php.errors.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_class_reserved.php.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_class_reserved_php.php [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_class_reserved_php.php.errors.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_class_reserved_php.php.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_class_unexpected.php [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_class_unexpected.php.errors.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_class_unexpected.php.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_const.php [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_const.php.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_const_w_ty_spec.php [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_const_w_ty_spec.php.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_enum.php [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_enum.php.errors.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_enum.php.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_enum.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_function_allowed.php [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_function_allowed.php.errors.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_function_allowed.php.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_function_unexpected.php [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_function_unexpected.php.errors.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_function_unexpected.php.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_member.php [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_member.php.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_method.php [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_method.php.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_static_method.php [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/keyword_as_static_method.php.sexp.exp [new file with mode: 0644]
hphp/hack/test/full_fidelity/cases/test_instanceof_arbitrary_expressions.php.errors.exp
hphp/hack/test/full_fidelity/cases/test_object_creation_errors.php.errors.exp
hphp/hack/test/full_fidelity/cases/unset_as_expression.php.errors.exp
hphp/hack/test/full_fidelity/cases/unset_as_expression.php.sexp.exp
hphp/hack/test/integration_ml/test_infer_type.ml
hphp/hack/test/typecheck/array/safe_vector_array/disallow_darray_with_wrong_key_type_where_varray_or_darray_is_required_in_non_strict_mode.php
hphp/hack/test/typecheck/array/safe_vector_array/disallow_darray_with_wrong_key_type_where_varray_or_darray_is_required_in_non_strict_mode.php.exp
hphp/hack/test/typecheck/const_attribute/usage/class.php
hphp/hack/test/typecheck/goto_mid_statement.php.exp
hphp/hack/test/typecheck/inherited_const_redeclared_abstract_in_used_trait.php
hphp/hack/test/typecheck/inherited_const_redeclared_abstract_in_used_trait.php.exp
hphp/hack/test/typecheck/instanceof_naming.php.exp
hphp/hack/test/typecheck/memoize_static2.php
hphp/test/slow/class_type_constant/type_constant2.php.expectregex
hphp/test/slow/inout/bad-class.php.expectregex
hphp/test/slow/null_coalesce_assignment/null_coalesce_assignment.php
hphp/test/slow/parser/hh-reserved-name-array-1.php.expectregex
hphp/test/slow/parser/hh-reserved-name-array-2.php.expectregex
hphp/test/slow/parser/php-reserved-name-array-1.php.expectregex
hphp/test/slow/parser/php-reserved-name-array-2.php.expectregex
hphp/test/slow/parser/semi-reserved-keywords/type_const_bad.php.expectregex
hphp/test/slow/streams/user_flush.php