Use env var to support blocking IP ranges #4806
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
justahero
force-pushed
the
feature/use-cidr-block-list
branch
4 times, most recently
from
524d71a to
cc88169
Compare
Turbo87
reviewed
May 11, 2022
Turbo87
added
C-internal 🔧
This is to reject requests from IP ranges that overload the API endpoints by fetching information from multiple clients at the same time. A recent incident highlighted this issue where a number of `crates?page` requests hit the ">0.5 failed requests threshold" alert. These requests timed out due to a high number of expensive requests made. These calls came in from about 100 different IPs in parallel. Before this change individual IPs or specific User Agents could be blocked. These existing approaches have some drawbacks, blocking single IPs would not have done much here as this is quite ineffective for this behaviour, while blocking a specific (common) User Agent may prevent valid usage of the API. Therefore a list of CIDR blocks should be used to check if IPs belong to certain ranges that send too many requests to expensive API endpoints. **Please note** this has to be done very carefully, it's generally a tool that may block whole regions of the internet.
justahero
force-pushed
the
feature/use-cidr-block-list
branch
from
8bee7b9 to
dfe0c87
Compare
|
@justahero just to make sure we're not waiting on each other: did you see the question above about IPv4-only? :) |
|
Interesting, indeed I did not see your comment. The reason for IPv4 based CIDR blocks was to get the block mechanism started while having a safeguard check on the given host prefix value. We could also remove this safeguard, easily add support for both IPv4 & IPv6 CIDR, then we need to be certain that the env var contains reasonable entries. I think that is fair, as this affects only the pagination endpoints. |
|
yeah, I think it would probably even simplify the code if we used generic CIDRs. unless there are any major downsides that I'm missing I think it'd be best to slightly tweak it to allow both :) |
justahero
force-pushed
the
feature/use-cidr-block-list
branch
2 times, most recently
from
46d97ea to
cbe1c4b
Compare
This refactors the `WEB_PAGE_OFFSET_CIDR_BLOCKLIST` block list to also support IPv6 based CIDR blocks.
justahero
force-pushed
the
feature/use-cidr-block-list
branch
from
cbe1c4b to
e0703b0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for the new
WEB_PAGE_OFFSET_CIDR_BLOCKLISTenvironment variable that holds a list of IPv4 based CIDR blocks. The use case of this env var is to block IPs coming from the same network or IP range. It should help to mitigate scenarios where a high number of distributed requests from multiple clients put a high load onto the application (e.g. bot network) which happen to come from the same IP range. The new block list is meant to complement the existing User Agent & IP block lists.Note the functionality is similar to the
WEB_PAGE_OFFSET_UA_BLOCKLISTenv var which blocks requests by matching user agents. It also only applies to the same (expensive) API endpoints that fetch data using pagination. The common experience using the web site should not be affected. The list is specified as a comma separated list of IPv4 based CIDR blocks (e.g."192.16.0.0/16").Currently the CIDR block list only supports IPv4 based entries, each block requires at least a host prefix of 16 bits.