[hermes] Fix memory leak

[ali-behjati]

@thmzlt identified a memory leak in our edge deployment. @Reisen and I worked on investigating it. We had an initial suspicion about libp2p because price service also apparently has a memory leak. Because of that in #876 we added go profiler to hermes. However, after more investigations, we realized that in price-service the node part has a leak and then it is probably not the libp2p and it's our code.

@Reisen suggested using heaptrack to profile the heap and I created a separate image with heaptrack and deployed it in the cluster. Using heaptrack_print (the cli tool) we can get the peak memory usages and after analyzing the logs over time I realized that the memory usages of this part of the code is increasing:

PEAK MEMORY CONSUMERS
86.58MB peak memory consumed over 111920 calls from                                                                                                                               
_cgo_7dd252f4a8ad_Cfunc__Cmalloc                                                                                                                                                  
  at /tmp/go-build/_cgo_export.c:53                                                                                                                                               
  in /usr/local/bin/hermes                                                                                                                                                        
86.58MB consumed over 111920 calls from:                                                                                                                                          
    runtime.asmcgocall                                                                                                                                                            
      at /usr/local/go/src/runtime/asm_amd64.s:848                                                                                                                                
      in /usr/local/bin/hermes

And then we checked our proxy method for communicating with go and found the issue. The issue was that apparently the go caller of our proxy method doesn't free the input that it passes to the method because the defer in a loop runs after the loop is over. I ran the fix for a couple of hours and the heap grow from this call stopped.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

2 Ignored Deployments

Name	Status	Preview	Comments	Updated (UTC)
example-oracle-amm	⬜️ Ignored (Inspect)			Jun 13, 2023 11:30am
xc-admin-frontend	⬜️ Ignored (Inspect)			Jun 13, 2023 11:30am