Rate limiting for UDP and TCP packets

[zeusoo001]

Background

Libp2p does not rate-limit the processing of UDP and TCP packets, which may cause resource loss. There are physical limitations on the processing power(such as bandwidth, CPU, and memory) of nodes, and the competition for resources by different applications may cause the following problems:

• Risk of resource overload: Burst traffic or high concurrent requests may cause the node processing capacity to be saturated, affecting normal service response.
• Degradation of service quality (QoS): Critical traffic (such as transaction/block broadcast) may be delayed or lost because non-critical traffic occupies resources.
• Differences in protocol characteristics: UDP's connectionless and low-overhead characteristics make it easy to generate high-throughput data streams, but it lacks a built-in flow control mechanism. Although TCP has congestion control, it may still exceed the node's load threshold in extreme scenarios.

Rationale

Rate limiting for UDP and TCP packets can balance resource allocation, avoid resource loss caused by non-critical traffic occupying resources, improve program robustness, and ensure service reliability.

Expected goals of rate limiting:

• Resource utilization optimization: maximize throughput within the critical point of node load.
• Service resilience improvement: reduce the risk of system crash through traffic shaping.
• User experience consistency: avoid global service degradation due to local overload.

The normal rate of each message can be used as a benchmark to limit the message rate without affecting normal functions. The rate of each message is listed below:

Rate limiting for UDP message

• PingMessage: When a new neighbor is found, a PingMessage will be sent with a timeout of 15s and go for 3 attempts.

• FindNeighbours: When the neighbor discovery protocol is running, FindNeighbours will be sent to different nodes. The neighbor discovery protool runs every 7.2s.

Rate limiting for TCP message

• KeepAliveMessage-ping: When the channel does not send a message for 20s, a ping message will be sent to keep the channel alive.
• HelloMessage: After the TCP connection is established, HelloMessage will be sent for a handshake. It will only appear once in the entire connection cycle.
• StatusMessage: Node detection message, scheduled every 5 seconds.
• P2pDisconnectMessage: After receiving this message, the connection will be disconnected. It will only appear once in the entire connection cycle.

[jakamobiii]

The rate limit of data packets is indeed very important. I saw that you listed the frequency of some data packets. How to limit the rate specifically? Is there a more detailed plan?

[zeusoo001]

@jakamobiii

Based on the normal frequency of the message.

UDP message

• You can set the channel to process PingMessage and FindNeighbours at a rate of 1/s, and discard messages that exceed the frequency.

• PongMessage and Neighbours are response messages, which can be controlled by logic. Only when a request is initiated will the message be processed.

TCP message

• You can set the channel to process ping at a rate of 1/s, and discard messages that exceed the rate. Failure to reply to ping will cause timeout and disconnection. Only when the network jitter exceeds 19s will it be possible to process two pings in succession. At this time, it is close to timeout. After disconnection, you can choose a node with a more stable network.

• Since the pong message is a response message, it can be controlled by code logic. If no ping message is sent, pong will not be processed, or the connection will be disconnected directly because it violates the protocol.

• HelloMessage can be controlled by code logic. After the handshake is completed, if HelloMessage is received again, it is considered to be a violation of the protocol and the connection can be disconnected.

• After receiving P2pDisconnectMessage, the connection will be disconnected and no message will be responded to. There is no need to limit the rate.

• StatusMessage is special. It is a message sent by a special connection. After sending StatusMessage to each other, the connection will be closed, but there will be a large number of concurrent connections at the same time, so it is necessary to limit the rate based on IP . The rate can be set to 1/s.

[jakamobiii]

Will other TCP message have the same problem as StatusMessage?

[zeusoo001]

@jakamobiii Other TCP message are sent through normal p2p connections, which have a maximum connection limit, so there will not be a large number of normal p2p connections at the same time.

[xxo1shine]

• UDP's connectionless and low-overhead characteristics make it easy to generate high-throughput data streams, but it lacks a built-in flow control mechanism. Although TCP has congestion control

@zeusoo001 According to your issue, UDP's connectionless and low-overhead characteristics make it easy to generate high-throughput data streams. Is it necessary to limit the rate of UDP packets globally?

[zeusoo001]

@xxo1shine

Good question. It is indeed necessary to limit the message rate globally. My solution is as follows.

UDP inbound traffic is difficult to control, so the rate limit is mainly for outbound traffic. There are four types of UDP messages, including request messages (PongMessage and FindNeighbours) and response messages (PongMessage and Neighbours).

Response messages can generally be controlled by code logic to be processed or not.

Therefore, the traffic rate limit only takes effect on request messages. When the rate limit threshold is exceeded, the request message is discarded and no longer processed.

[xxo1shine]

@zeusoo001 UDP is connectionless, so it is difficult to limit the number of channels of a certain IP. Do we also need to consider limiting the rate based on IP?

[zeusoo001]

@xxo1shine It is indeed necessary to limit the rate based on IP. In addition, there is another point I want to optimize. Considering that the normal frequency of PingMessage and FindNeighbours is very low, it is enough to directly limit the overall rate of request messages. This can make the code more concise and achieve the desired effect.

[jakamobiii]

@zeusoo001 If the rate of UDP global traffic is limited, what should the threshold be set to?

[zeusoo001]

@jakamobiii This can be obtained based on actual test data and theoretical calculations. According to statistics of several online nodes, the maximum rate is 82k/s. The traffic basically comes from neighbors messages. The size of neighbors is 1.5k. If it is set to 1M, it can support more than 600 nodes to access concurrently. From the probability calculation, the neighbor discovery protocol is scheduled every 7.2s. Only when the number of k-bucket nodes reaches more than 4,000, can the threshold be reached in theory. The actual data in the k-bucket is only more than 200, so in theory, setting it to 1M is enough to meet the requirements. In addition, we can set this configuration to be configurable, so that the UDP global rate limit can be used more flexibly.

[xxo1shine]

@zeusoo001 If a large amount of traffic comes in, will the node enter a denial of service state? The UDP rate limit still needs further analysis.

[jakamobiii]

What if the UDP message is encrypted and signed?

[zeusoo001]

What if the UDP message is encrypted and signed?

@jakamobiii Encrypting and signing UDP messages will be more private and secure, but signature verification is currently time-consuming, which may introduce more problems.