OOM/Timeouts when saving a category with large catalog as all product URLs are regenerated

[devonto]

With a catalog of many thousands of products loaded, saving a category causes long execution times and high memory usage, when it isn't necessarily required. In a production environment, the execution fails with max_execution_time or memory_limit errors.

It appears that the system is trying to regenerate all URLs of products contained within the category and subcategories; even if the url_key is not modified.

Preconditions

1. Magento 2.1.1
2. max_execution_time set to 300
3. memory_limit set to 2G
4. Catalog of 140,000 products
5. Top category which contains subcategories and 40,000+ products

Steps to reproduce

1. Simply "Save" the top category without even making any changes

Expected result

1. No errors
2. A notice of You saved the category should appear

Actual result

1. PHP errors for max_execution_time and/or memory_limit

A workaround is to modify /vendor/magento/module-catalog-url-rewrite/Observer/CategoryProcessUrlRewriteSavingObserver.php temporarily to return; on line 1 of the execute() function to prevent URL rewrite changes.

A suggested performance improvement would be to check if the url_key was modified before running the regeneration of URLs.

This, though, cannot be used if the url_key is updated. At that point, the process should be handled by some background task.

[erikhansen]

One of my 2.1.4 merchants is experiencing this exact issue. They have a category that contains 538 children/descendent categories and those child categories contain a total of 5k+ products. Upon saving the category, the 768M native Magento memory limit is hit. Magento's memory limit should not need to be increased to accommodate saving this category.

When the category is saved, the \Magento\CatalogUrlRewrite\Observer\CategoryUrlPathAutogeneratorObserver::updateUrlPathForChildren observer calls \Magento\CatalogUrlRewrite\Observer\CategoryUrlPathAutogeneratorObserver::updateUrlPathForChildren for all ~583 child categories. This behavior happens of regardless of whether "By Category" is set to "Anchor == Yes" or "Anchor == No".

I profiled the category save using Xdebug and loaded the data in MacCallGrind and have included some screenshots below demonstrating how much of the processing time was used by the method mentioned above:

[orlangur]

@devonto,

A suggested performance improvement would be to check if the url_key was modified before running the regeneration of URLs.

Yeah, AFAIR it worked this way in initial implementation:
https://github.com/magento/magento2/blob/develop/app/code/Magento/CatalogUrlRewrite/Observer/CategoryUrlPathAutogeneratorObserver.php#L61

Maybe, this logic was broken in one of further fixes, like 6c0ac05#diff-33a66f964a57f4cb0f7f47ee7358ed35R106 where some force changes to url_path are introduced.

@erikhansen,

This behavior happens of regardless of whether "By Category" is set to "Anchor == Yes" or "Anchor == No".

Anchor option is responsible for products attachment to category I believe. Another option is responsible for whether URLs of child categories contain parent category url_path or not. The latter can be used to avoid updates of what is not necessary to be updated.

[quisse]

Any update on this?

[dan-advantec]

Is there an update on this please, still an issue in 2.1.7

[woutersamaey]

If I remember correctly, the fix is in the latest develop branch, but Magento is terribly slow to release it to the general public. We have solved this ourselves in a custom built module for our client.

…

On 2 Aug 2017, at 13:15, dan-advantec ***@***.***> wrote: Is there an update on this please, still an issue in 2.1.7 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7874 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEVa5zq427edlpqtqi5N8Jb4MfJISp3ks5sUFpngaJpZM4LQqWu>.

[dan-advantec]

Excellent, thanks @woutersamaey I assume this module isn't released to the general public?

[woutersamaey]

No, this is closed source, but you can search for the solution in the develop branch or my company can do the work for you and re-package the fix in a stand-alone module if you don’t have the means.

…

On 2 Aug 2017, at 13:23, dan-advantec ***@***.***> wrote: Excellent, thanks @woutersamaey <https://github.com/woutersamaey> I assume this module isn't released to the general public? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7874 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEVa5wBPK1PMPP_UisHFJCLcp1uwBZrks5sUFwwgaJpZM4LQqWu>.

[dan-advantec]

No worries @woutersamaey I'll have a look in the dev branch, but thank you for the offer.

[orlangur]

@dan-advantec @woutersamaey please check https://community.magento.com/t5/Magento-DevBlog/Pull-Requests-for-2-1-x-Patch-Releases/ba-p/65630 out. So, having knowledge where a bugfix resides in develop branch the best approach is to create a backport PR to 2.1-develop branch (it will be included in next 2.1.x release, until then you can apply a well-tested fix manually to your codebase).

[mdmujassam]

@dan-advantec you did find any solution ?