Integration tests fail massively

[Vinai]

Currently I'm unable to run the integration tests.
Steps to reproduce:

$ cd dev/tools && php tests.php --type=integration

I have to assign 1G memory to PHP to make them run, but even then pretty much every test gives me a stacktrace.
The same happens if I call phpunit directly without the tests.php script.

And its not always the same stacktrace. Some are due to Magento\RecurringPayment missing, but some others seem completely unrelated to the payment methods.
See below at the bottom of this issue for a few examples.

What I don't get is why the integration tests on travis run.

cd dev/tests/integration/; ./../../../vendor/bin/phpunit -c phpunit.xml.dist;

Any idea what I am doing wrong? Was something changed recently in how to run the integration tests?

Here a couple examples of from the exceptions and errors:

Fatal error: Call to a member function getId() on a non-object in m2.dev/htdocs/dev/tests/integration/testsuite/Magento/Catalog/Model/AbstractTest.php on line 125

m2.dev/htdocs/dev/tests/integration/framework/Magento/TestFramework/Annotation/DataFixture.php(159): Magento\TestFramework\Annotation\DataFixture->_applyOneFixtPHP Fatal error:  Call to a member function getId() on a non-object in m2.dev/htdocs/dev/tests/integration/testsuite/Magento/Catalog/Model/AbstractTest.php on line 125

Error in fixture: "m2.dev\/htdocs\/dev\/tests\/integration\/testsuite\/Magento\/Catalog\/_files\/products.php"
exception 'ReflectionException' with message 'Class Magento\RecurringPayment\Model\Product\Attribute\Backend\Recurring does not exist' in m2.dev/htdocs/lib/internal/Magento/Framework/Code/Reader/ClassReader.php:18

Error in fixture: "m2.dev\/htdocs\/dev\/tests\/integration\/testsuite\/Magento\/Catalog\/_files\/products_rollback.php"
exception 'Magento\Eav\Model\Entity\Attribute\Exception' with message 'SQLSTATE[42S22]: Column not found: 1054 Unknown column 'main_table.website_id' in 'where clause', query was: SELECT `main_table`.*, `cp_table`.`type_id` FROM `cataloginventory_stock_item` AS `main_table`
 INNER JOIN `catalog_product_entity` AS `cp_table` ON main_table.product_id = cp_table.entity_id WHERE (`main_table`.`product_id` IN('')) AND (`main_table`.`website_id` = '1')' in /Volumesm2.dev/htdocs/lib/internal/Magento/Framework/ObjectManager/Factory/AbstractFactory.php:98

[Vinai]

Ups, I got it. Refreshing the test database to match the current m2 version helps.

[Vinai]

Ups, too early. That didn't help after all :(

[buskamuza]

Try to cleanup everything before running tests:

1. empty DB
2. cleanup var directory
3. cleanup dev/tests/integration/tmp directory

Maybe DB cleanup doesn't work properly or/and you have old generated files and they are not removed automatically.

[Vinai]

Thanks, one of those 3 almost did the trick. Now 29% of the integration tests execute without errors, even though many fail.

% ./../../../vendor/bin/phpunit -c phpunit.xml.dist
PHPUnit 4.1.0 by Sebastian Bergmann.

Configuration read from /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/dev/tests/integration/phpunit.xml

............SF.F..S..........................................   61 / 3057 (  1%)
....IIIIIII.......FFFFFFFFF.F.F.FFFFFFFFFFFF....FFFFFF...F...  122 / 3057 (  3%)
...I.........................................................  183 / 3057 (  5%)
................SSS.....F.........................F.F.FF.....  244 / 3057 (  7%)
.........................................................SFFF  305 / 3057 (  9%)
EFFFFFFFFFFFFFFFF.F..FFFFFFFF...................S............  366 / 3057 ( 11%)
.............................................................  427 / 3057 ( 13%)
..........................S..................................  488 / 3057 ( 15%)
................................S............................  549 / 3057 ( 17%)
........I....................................................  610 / 3057 ( 19%)
.......................I.....................................  671 / 3057 ( 21%)
.............................................................  732 / 3057 ( 23%)
...................................................S.........  793 / 3057 ( 25%)
.............................................................  854 / 3057 ( 27%)
..........................I..............F....I..............  915 / 3057 ( 29%)
...........PHP Fatal error:  Call to a member function getAssociatedProductIds() on a non-object in /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/dev/tests/integration/testsuite/Magento/ConfigurableProduct/Controller/Adminhtml/ProductTest.php on line 32
PHP Stack trace:
PHP   1. {main}() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/phpunit:0
PHP   2. PHPUnit_TextUI_Command::main() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/phpunit:55
PHP   3. PHPUnit_TextUI_Command->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/Command.php:132
PHP   4. PHPUnit_TextUI_TestRunner->doRun() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/Command.php:179
PHP   5. PHPUnit_Framework_TestSuite->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:426
PHP   6. PHPUnit_Framework_TestSuite->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestSuite.php:675
PHP   7. PHPUnit_Framework_TestCase->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestSuite.php:675
PHP   8. PHPUnit_Framework_TestResult->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:753
PHP   9. PHPUnit_Framework_TestCase->runBare() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestResult.php:686
PHP  10. PHPUnit_Framework_TestCase->runTest() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:817
PHP  11. ReflectionMethod->invokeArgs() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:951
PHP  12. Magento\ConfigurableProduct\Controller\Adminhtml\ProductTest->testSaveActionAssociatedProductIds() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:951

Again I'm wondering how this is different from how travis is executing the integration tests.

[buskamuza]

Hm. I wonder, why PHPUnit says that it reads information from phpunit.xml, while you specify phpunit.xml.dist for it?

[Vinai]

Yes, interesting. I have added a phpunit.xml with the only difference to the phpunit.xml.dist being 1G available memory for running the tests.

% diff -u phpunit.xml.dist phpunit.xml                                                                                                                                    ✭
--- phpunit.xml.dist    2015-01-05 15:44:51.000000000 +0100
+++ phpunit.xml 2015-01-05 23:39:34.000000000 +0100
@@ -38,6 +38,7 @@
         <includePath>testsuite</includePath>
         <ini name="date.timezone" value="America/Los_Angeles"/>
         <ini name="xdebug.max_nesting_level" value="200"/>
+        <ini name="memory_limit" value="1024M"/>
         <!-- Local XML configuration file ('.dist' extension will be added, if the specified file doesn't exist) -->
         <const name="TESTS_INSTALL_CONFIG_FILE" value="etc/install-config-mysql.php"/>
         <!-- Semicolon-separated 'glob' patterns, that match global XML configuration files -->

[maksek]

Hey @Vinai, any luck with tests? They are straight forward, did you specify DB connection, last errors looks like no DB connected.

[Vinai]

Thanks for checking in. Yes, I have the DB connection configured. WIthout it the integration tests abort before the first test is executed.
No change from my last post above.

[maksek]

If you clean all required folders, drop DB and create new empty one. Then launching tests - what last error do you have.

[Vinai]

Here is what I get

% php --version
PHP 5.5.20 (cli) (built: Dec 27 2014 17:36:56)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with uopz v2.0.6, Copyright (c) 2014, by Joe Watkins <krakjoe@php.net>
    with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies
    with Xdebug v2.2.6, Copyright (c) 2002-2014, by Derick Rethans
% pwd
/Volumes/CaseSensitive/Workspace/m2.dev/htdocs/dev/tests/integration
% rm -rf ../../../var/cache ../../../var/generation tmp/*
% echo "DROP DATABASE magento_integration_tests; CREATE DATABASE magento_integration_tests;" | mysql
% ../../../vendor/bin/phpunit -c phpunit.xml
PHPUnit 4.1.0 by Sebastian Bergmann.

Configuration read from /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/dev/tests/integration/phpunit.xml

................SF.F..S......................................   61 / 3059 (  1%)
........IIIIIII.......FFFFFFFFF.F.F.FFFFFFFFFFFF....FFFFFF...  122 / 3059 (  3%)
F......I.....................................................  183 / 3059 (  5%)
....................SSS.....F.........................F.F.FF.  244 / 3059 (  7%)
.............................................................  305 / 3059 (  9%)
SFFFEFFFFFFFFFFFFFFFF.F..FFFFFFFF...................S........  366 / 3059 ( 11%)
.............................................................  427 / 3059 ( 13%)
..............................S..............................  488 / 3059 ( 15%)
....................................S........................  549 / 3059 ( 17%)
............I................................................  610 / 3059 ( 19%)
...........................I.................................  671 / 3059 ( 21%)
.............................................................  732 / 3059 ( 23%)
.......................................................S.....  793 / 3059 ( 25%)
.............................................................  854 / 3059 ( 27%)
..............................I..............F....I..........  915 / 3059 ( 29%)
...............PHP Fatal error:  Call to a member function getAssociatedProductIds() on a non-object in /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/dev/tests/integration/testsuite/Magento/ConfigurableProduct/Controller/Adminhtml/ProductTest.php on line 32
PHP Stack trace:
PHP   1. {main}() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/phpunit:0
PHP   2. PHPUnit_TextUI_Command::main() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/phpunit:55
PHP   3. PHPUnit_TextUI_Command->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/Command.php:132
PHP   4. PHPUnit_TextUI_TestRunner->doRun() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/Command.php:179
PHP   5. PHPUnit_Framework_TestSuite->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:426
PHP   6. PHPUnit_Framework_TestSuite->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestSuite.php:675
PHP   7. PHPUnit_Framework_TestCase->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestSuite.php:675
PHP   8. PHPUnit_Framework_TestResult->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:753
PHP   9. PHPUnit_Framework_TestCase->runBare() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestResult.php:686
PHP  10. PHPUnit_Framework_TestCase->runTest() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:817
PHP  11. ReflectionMethod->invokeArgs() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:951
PHP  12. Magento\ConfigurableProduct\Controller\Adminhtml\ProductTest->testSaveActionAssociatedProductIds() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:951

Fatal error: Call to a member function getAssociatedProductIds() on a non-object in /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/dev/tests/integration/testsuite/Magento/ConfigurableProduct/Controller/Adminhtml/ProductTest.php on line 32

Call Stack:
    0.0014     982168   1. {main}() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/phpunit:0
    0.0096    1235232   2. PHPUnit_TextUI_Command::main() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/phpunit:55
    0.0097    1235864   3. PHPUnit_TextUI_Command->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/Command.php:132
   27.4045   98097192   4. PHPUnit_TextUI_TestRunner->doRun() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/Command.php:179
   27.4103   98184288   5. PHPUnit_Framework_TestSuite->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:426
  811.7143  454923376   6. PHPUnit_Framework_TestSuite->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestSuite.php:675
  811.7826  454137224   7. PHPUnit_Framework_TestCase->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestSuite.php:675
  811.7826  454137632   8. PHPUnit_Framework_TestResult->run() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:753
  812.8949  459494536   9. PHPUnit_Framework_TestCase->runBare() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestResult.php:686
  813.0525  456755376  10. PHPUnit_Framework_TestCase->runTest() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:817
  813.0526  456756256  11. ReflectionMethod->invokeArgs() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:951
  813.0526  456756464  12. Magento\ConfigurableProduct\Controller\Adminhtml\ProductTest->testSaveActionAssociatedProductIds() /Volumes/CaseSensitive/Workspace/m2.dev/htdocs/vendor/phpunit/phpunit/src/Framework/TestCase.php:951


=== Memory Usage System Stats ===
Memory usage (OS):  766.51M (164.66% of 465.50M reported by PHP)
Estimated memory leak:  301.01M (39.27% of used memory)

%

And thats it.

[maksek]

Did you try to skip the test and run without it? Wonder if it is specific test issue. Trying to reproduce.

[maksek]

@Vinai, i have fresh cloned magneto2, cleaned up everything in var/*, and and tests passed well.
Can you try to clean not only var/cache and var/generate, but all children folders in var/ and try again. Seems like environment glitch.