TokenPolyfill issue on PHP 7.4 platform

[llaville]

Hello,

I've worked today to make a package compatible with both PHP-Parser 4.18 and 5.0

I've detected an issue on (I know old platform PHP 7.4, but this package is still compatible PHP 7). Here are the details of my analysis

When PHP-Parser 5.0.0 is installed with PHP_CodeSniffer (3.9.0 in my situation), I got a TypeError

TypeError: Argument 1 passed to PhpParser\Internal\TokenPolyfill::__construct() must be of the type int, string given, called in /shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/Internal/TokenPolyfill.php on line 202

/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/Internal/TokenPolyfill.php:42
/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/Internal/TokenPolyfill.php:202
/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/Lexer.php:31
/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/Lexer/Emulative.php:71
/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/ParserAbstract.php:182
/shared/backups/forks/symbol-parser/src/Parser/PHP/SymbolNameParser.php:41
/shared/backups/forks/symbol-parser/tests/Integration/Parser/PHP/SymbolNameParserTest.php:42

Version 5.0 is supposed to be compatible with PHP 7.4 min (see https://github.com/nikic/PHP-Parser/blob/v5.0.0/composer.json#L16-L17)

But :

• T_NAME_QUALIFIED https://github.com/nikic/PHP-Parser/blob/v5.0.0/lib/PhpParser/Internal/TokenPolyfill.php#L200 was introduced on PHP 8.0.0beta1
• T_NAME_RELATIVE https://github.com/nikic/PHP-Parser/blob/v5.0.0/lib/PhpParser/Internal/TokenPolyfill.php#L198 was introduced on PHP 8.0.0beta1
• T_NAME_FULLY_QUALIFIED https://github.com/nikic/PHP-Parser/blob/v5.0.0/lib/PhpParser/Internal/TokenPolyfill.php#L196 was introduced on PHP 8.0.0beta1

Reason is PHPCS define these tokens as string.
And TokenPolyfill first argument of class constructor is int

Hope my explains are enough clear ?

[staabm]

there is a related non-released commit on master: ff095c3

maybe this already "fixes" your problem?

also related: #974

[staabm]

@llaville I have opened a issue on the PHPCS repo. maybe you can give more information about your use-case there, so jrfnl can decide whether there is something todo or not.

I just wanted to raise awareness and cannot add more than that.

[llaville]

@staabm Thanks but I don't think it come from PHPCS. I've investigated by debugging code analysed with TokenPolyfill, and there is an issue inside !

[nikic]

I've implemented a check to detect broken libraries like this a while ago (ff095c3) but never released it. It is now released and will hard error early on.

[llaville]

@nikic FYI

BTW

I've put some debugging steps on TokenPolyfill

                    if ($j > $i + 1) {
                        \error_log('@AAA ' . var_export([$i, $j, \T_NS_SEPARATOR, \T_NAME_FULLY_QUALIFIED, \T_NAMESPACE, \T_NAME_RELATIVE, \T_NAME_QUALIFIED], true), 3, '/var/log/php/php-parser.log');
                        if ($id === \T_NS_SEPARATOR) {
                            $id = \T_NAME_FULLY_QUALIFIED;
                        } elseif ($id === \T_NAMESPACE) {
                            $id = \T_NAME_RELATIVE;
                        } else {
                            $id = \T_NAME_QUALIFIED;
                        }
                        if (!\is_int($id)) {
                            \error_log('@BBB ' . var_export(debug_backtrace(), true), 3, '/var/log/php/php-parser.log');
                        }

Code Analysed was

        $code = <<<CODE
        <?php
        declare(strict_types=1);

        namespace Test\Sub;

        class TestClass {
            public function test() {}
        }
        CODE;

When namespace is found

@AAA gave us
[
	0 => 11, 
	1 => 14, 
	2 => 393, 
	3 => 'T_NAME_FULLY_QUALIFIED', 
	4 => 391, 
	5 => -1, 
	6 => 'T_NAME_QUALIFIED'
]

[llaville]

Confirmed by running package unit tests with PHP-Parser 5.0.1

1) ComposerUnused\SymbolParser\Test\Integration\Parser\PHP\SymbolNameParserTest::itShouldParseClasses
Error: Token T_NAME_QUALIFIED has ID of type string, should be int. You may be using a library with broken token emulation

/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/compatibility_tokens.php:30
/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/compatibility_tokens.php:62
/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/Lexer.php:5
/shared/backups/forks/symbol-parser/vendor/composer/ClassLoader.php:578
/shared/backups/forks/symbol-parser/vendor/composer/ClassLoader.php:432
/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/Lexer/Emulative.php:24
/shared/backups/forks/symbol-parser/vendor/composer/ClassLoader.php:578
/shared/backups/forks/symbol-parser/vendor/composer/ClassLoader.php:432
/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/ParserFactory.php:18
/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/ParserFactory.php:32
/shared/backups/forks/symbol-parser/tests/Integration/Parser/PHP/SymbolNameParserTest.php:25

2) ComposerUnused\SymbolParser\Test\Integration\Parser\PHP\SymbolNameParserTest::itShouldParseInterfaces
Error: Undefined constant 'T_NAME_RELATIVE'

/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/ParserAbstract.php:1219
/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/ParserAbstract.php:159
/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/ParserFactory.php:21
/shared/backups/forks/symbol-parser/vendor/nikic/php-parser/lib/PhpParser/ParserFactory.php:32
/shared/backups/forks/symbol-parser/tests/Integration/Parser/PHP/SymbolNameParserTest.php:53

[jrfnl]

I've implemented a check to detect broken libraries like this a while ago (ff095c3) but never released it. It is now released and will hard error early on.

But does that solve the problem ? As far as I can see, that just moves the problem around and doesn't solve anything...

[llaville]

Answer is into official documentation https://github.com/nikic/PHP-Parser/blob/master/doc/component/Lexer.markdown

We can read

So no solution on my use case

[nikic]

@llaville This refers to something completely unrelated. That point is exactly the same on PHP-Parser 4.0.

The problem here is that PHP-Parser and PHP_CodeSniffer are not compatible with each other. You currently cannot use both at the same time.

[llaville]

But the issue still exists even if I removed PHP_CodeSniffer from context !

[nikic]

@jrfnl It doesn't solve the problem, but it provides a clear error message. It's impossible to "solve" this problem in PHP-Parser, because the requirement for integer token IDs is imposed by PHP itself. It's something I could avoid in my own TokenPolyfill on older versions, but on PHP 8 the native PhpToken is used, and it requires integer token IDs as well.

The only way I can avoid this requirement is by not using the global token constants at all, and instead redefining all token constants myself. I considered doing this (specifically to avoid this kind of problem), but opted not to do so, because I thought there was more value in sticking with PHP's own token format. In any case, it's not something I can change anymore.

[llaville]

I've just found something strange. Need to investigate more again.

On my fork version of composer-unused/symbol-parser (https://github.com/llaville/symbol-parser/tree/php-parser-v5) where I've implemented PHP-Parser v5 compatibility, I run unit tests with PHAR version of PHPUnit 9.6 that raise me errors.

But when I run same tests with phpunit installed as a dev dependency, I got no errors.