High memory usage

[zenas1210]

Q	A
Version	0.17.0

Support Question

Hi,
I've encountered some memory usage issues when using this with phpro/soap-client v4. I'm running out of memory when decoding certain responses, which aren't really big and php-soap/ext-soap-engine with phpro/soap-client v3.4 has no problem with them. Since the SOAP webservice I'm integrating with is private I can't share an end to end example of my problem. Is there any automated test or maybe just a code snippet which would enable me to demonstrate the issue by supplying the WSDL and a somewhat fake response with confidential data removed from it?

[veewee]

We'll at least need a memory-problematic payload and wsdl to reproduce and start improving upon. The data inside it may be fake.

Is this happening during encoding, decoding or both? This might require 2 payloads then: a php objects representation for encoding and an XML payload for decoding.

If that's not possible, you could also do the profiling of the code yourself to see if there are abvious resources eating up memory. We're open for PRs.

[zenas1210]

This happens only during decoding.
From first glance ObjectEncoder::iso seems to be called many times, probably too many, for properties which are not part of the response being decoded.

WSDL: https://fpy.noc.vmc.navblue.cloud/raidoapi/nocsoapapi.asmx?wsdl
Function: GetFlights
Response with some scalar values changed, 4GB of RAM couldn't handle it:

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <soap:Body>
      <GetFlightsResponse xmlns="http://raido.aviolinx.com/api/">
         <GetFlightsResult>
            <Errors />
            <Warnings />
            <StatusCode>200</StatusCode>
            <Result>
               <Flight>
                  <UniqueId>1</UniqueId>
                  <AirlineCode>AA</AirlineCode>
                  <ICAOAirlineCode>AAA</ICAOAirlineCode>
                  <FlightNumber>1</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T14:45:00</STD>
                  <STA>2025-01-11T19:40:00</STA>
                  <AircraftRegistration>LYAAA</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>S</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>60</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>0</ArrivalLocalTimeDiff>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>3270</ScheduleGCDKM>
                  <ScheduleGCDNM>1766</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>2</UniqueId>
                  <AirlineCode>BB</AirlineCode>
                  <ICAOAirlineCode>BBB</ICAOAirlineCode>
                  <FlightNumber>2</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T15:00:00</STD>
                  <STA>2025-01-11T19:10:00</STA>
                  <AircraftRegistration>LYBBB</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>S</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>60</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>0</ArrivalLocalTimeDiff>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>2644</ScheduleGCDKM>
                  <ScheduleGCDNM>1428</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>3</UniqueId>
                  <AirlineCode>CC</AirlineCode>
                  <ICAOAirlineCode>CCC</ICAOAirlineCode>
                  <FlightNumber>3</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T15:20:00</STD>
                  <STA>2025-01-11T22:00:00</STA>
                  <AircraftRegistration>TFPLA</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Canceled</Status>
                  <Type>A</Type>
                  <Canceled>true</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>-300</ArrivalLocalTimeDiff>
                  <IROPSParentId>0</IROPSParentId>
                  <ASMReasons>EQUI</ASMReasons>
                  <Complement>321LR</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>4495</ScheduleGCDKM>
                  <ScheduleGCDNM>2427</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>4</UniqueId>
                  <AirlineCode>DD</AirlineCode>
                  <ICAOAirlineCode>DDD</ICAOAirlineCode>
                  <FlightNumber>4</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T15:20:00</STD>
                  <STA>2025-01-11T22:00:00</STA>
                  <AircraftRegistration>LYDDD</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>S</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>-300</ArrivalLocalTimeDiff>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>4436</ScheduleGCDKM>
                  <ScheduleGCDNM>2395</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>5</UniqueId>
                  <AirlineCode>EE</AirlineCode>
                  <ICAOAirlineCode>EEE</ICAOAirlineCode>
                  <FlightNumber>5</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T15:50:00</STD>
                  <STA>2025-01-11T20:25:00</STA>
                  <AircraftRegistration>LYEEE</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>A</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>60</ArrivalLocalTimeDiff>
                  <IROPSParentId>0</IROPSParentId>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>3208</ScheduleGCDKM>
                  <ScheduleGCDNM>1732</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>6</UniqueId>
                  <AirlineCode>FF</AirlineCode>
                  <ICAOAirlineCode>FFF</ICAOAirlineCode>
                  <FlightNumber>6</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T15:50:00</STD>
                  <STA>2025-01-11T21:35:00</STA>
                  <AircraftRegistration>LYFFF</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>S</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>0</ArrivalLocalTimeDiff>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>4020</ScheduleGCDKM>
                  <ScheduleGCDNM>2170</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>7</UniqueId>
                  <AirlineCode>GG</AirlineCode>
                  <ICAOAirlineCode>GGG</ICAOAirlineCode>
                  <FlightNumber>7</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T15:50:00</STD>
                  <STA>2025-01-11T22:10:00</STA>
                  <AircraftRegistration />
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Canceled</Status>
                  <Type>A</Type>
                  <Canceled>true</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>-300</ArrivalLocalTimeDiff>
                  <ASMReasons>COMM</ASMReasons>
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>4230</ScheduleGCDKM>
                  <ScheduleGCDNM>2284</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>8</UniqueId>
                  <AirlineCode>HH</AirlineCode>
                  <ICAOAirlineCode>HHH</ICAOAirlineCode>
                  <FlightNumber>8</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T16:00:00</STD>
                  <STA>2025-01-11T22:00:00</STA>
                  <AircraftRegistration>LYHHH</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>S</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>-300</ArrivalLocalTimeDiff>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>3874</ScheduleGCDKM>
                  <ScheduleGCDNM>2092</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>9</UniqueId>
                  <AirlineCode>II</AirlineCode>
                  <ICAOAirlineCode>III</ICAOAirlineCode>
                  <FlightNumber>9</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T16:00:00</STD>
                  <STA>2025-01-11T22:05:00</STA>
                  <AircraftRegistration>LYIII</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>A</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>-300</ArrivalLocalTimeDiff>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>4100</ScheduleGCDKM>
                  <ScheduleGCDNM>2214</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
            </Result>
         </GetFlightsResult>
      </GetFlightsResponse>
   </soap:Body>
</soap:Envelope>

[zenas1210]

I've added some debug logging locally here to see property names being handled, and saw properties like ReservationComment in the output, which isn't even close to being in GetFlights response, so maybe that has something to do with it. Haven't experienced issues with other requests though. Would love to help fix this but I'm not yet familiar with the architecture and everything.

[veewee]

Thanks for sharing the example. That payload should definitely not take 4G of ram.

I'll try to take a look what's going on next week when I'm back behind my computer.

[veewee]

Just a small heads-up:
The problem here seems recursion.

PHP Fatal error: Uncaught Error: Xdebug has detected a possible infinite loop, and aborted your script with a stack depth of '512' frames

^ "Crew"
^ "ArrayOfRosterActivity"
^ "RosterActivity"
^ "ArrayOfCrewOnBoard"
^ "CrewOnBoard"
^ "Crew"
^ "ArrayOfRosterActivity"
^ "RosterActivity"
^ "ArrayOfCrewOnBoard"
^ "CrewOnBoard"
^ "Crew"
^ "ArrayOfRosterActivity"
^ "RosterActivity"
^ "ArrayOfCrewOnBoard"
^ "CrewOnBoard"
^ "Crew"
^ "ArrayOfRosterActivity"
^ "RosterActivity"
^ "ArrayOfCrewOnBoard"
^ "CrewOnBoard"
^ "Crew"
^ "ArrayOfRosterActivity"
^ "RosterActivity"
^ "ArrayOfCrewOnBoard"
^ "CrewOnBoard"
^ "Crew"
^ "ArrayOfRosterActivity"
^ "RosterActivity"
^ "ArrayOfCrewOnBoard"
^ "CrewOnBoard"

This recursion floods your memory until it crashes (unless you have xdebug enabled)

As you can see, the types are linking to each other from within a subtype.
Currently the ObjectAccess is trying to resolve to a type that can cover every potential optional object from within the response given the XSD in your WSDL. There is no failsafe system that can detect this recursive redundant types yet.

So that's what I'm gonna look into solving for this issue.

[veewee]

@zenas1210 You can test this PR out already:
#35

I still need to do some additional performance tests on big payloads - but this should be good to go.
The decoding now took about 4MB, which is doable I suppose :)

[zenas1210]

woah cool I'll try this out today

[zenas1210]

So far seems to work perfectly, tried out in a symfony app with a 5 megabyte response, didn't have to increase the memory limit beyond 256M. Thanks a lot!