Request: Publish new xc-server container image

[fubar]

It looks like this change #766 is not included in the current container images: http://localhost:4200/api/latest_price_feeds?ids[]=0xca80ba6dc32e08d06f1aa886011eed1d77c77be9eb761cc10d72b7d0a2fd57a6&target_chain=evm returns a 400 with "target_chain" is not allowed (using docker-compose.testnet.yaml which specifies image public.ecr.aws/pyth-network/xc-server:v3.0.0).

Can you publish a new image with the latest changes? Thanks!

[guibescos]

Hey thanks so much for reporting.
A new version has been released, but we haven't updated the docker compose. Apologies for that.
Can you try using xc-server:v3.0.3?

[fubar]

Thanks @guibescos, v3.0.3 supports the target_chain parameter, but it has a few issues:

• Running the mainnet configuration (docker-compose.mainnet.yaml) causes price feed IDs with the 0x prefix to be rejected with Price Feed(s) with id(s) (...) not found. Removing the 0x prefix works.
• In the testnet configuration (docker-compose.testnet.yaml) it does not accept any price IDs. I tried testnet and mainnet IDs, with and without the 0x prefix, and all result in Price Feed(s) with id(s) (...) not found. In case it helps, running the container with log level set to debug shows a lot of "Ignoring an invalid VAA" messages in the log.

Let me know if you'd prefer a separate issue for these.

[guibescos]

Let me run it locally.

[guibescos]

For testnet try setting this environment variable (WORMHOLE_CLUSTER) from an upcoming PR https://github.com/pyth-network/pyth-crosschain/pull/814/files#diff-467a4b466afe28dd54ed5925bb753c03ca682cf41c2469d3bf73aacaae52a0f3R42 .
For the 0x issue can you explain how to reproduce it?

[fubar]

It works with the WORMHOLE_CLUSTER env var, thanks!

I wasn't able to reproduce the 0x prefix issue this time around, but I believe it was related to the spy not getting a peer connection on startup in some situations. My observations:

• When I start the containers from scratch (i.e. not a restart), they usually come up fine
• When both containers start up without issue, the price service container starts logging Sending (...) price update to no clients and Parsed a new Batch Price Attestation: (...) messages at high frequency shortly after launch. It takes another moment for it to return prices instead of Price Feed(s) with id(s) (...) not found. I believe I observed the issues with the 0x prefix during this stage. It now works with or without the 0x prefix.
• When I restart (stop and start) the containers without removing them from Docker, the spy container seems to not get a peer connection most times:
  • Most times no logs are generated by either container and all progress looks stalled
  • When I stop the containers in that situation, the spy container logs this before it exits:

    ERROR   wormhole-spy    zap/options.go:212      finished streaming call with code Unknown       {"grpc.start_time": "2023-05-17T00:23:08Z", "system": "grpc", "span.kind": "server", "grpc.service": "spy.v1.SpyRPCService", "grpc.method": "SubscribeSignedVAA", "peer.address": "172.31.0.3:49242", "x-forwarded-for": [], "error": "context canceled", "grpc.code": "Unknown", "grpc.time_ms": 56460.375}
    

  • Other times it takes up to a few minutes until the spy appears to get a connection and the price service container starts generating output
  • The system never comes up as quickly as when starting the containers from scratch
  • Switching between the docker-compose.testnet.yaml and docker-compose.mainnet.yaml configurations without first removing the containers from Docker does not seem to work at all and generates errors

Some of these may be known limitations, but it's valuable knowledge for us that the containers should be removed from Docker before (re)starting them.

[jayantk]

Thanks. we've seen some issues for the spy connecting to the p2p network in the past as well. Our next iteration of this service should address all of those, but in the meanwhile, i've submitted this PR #826 to document these problems.