TooManyRequestsException on GetBootstrapBrokers operation #6
Description
I'm using this repository to run a load test against AWS MSK.
The test cluster is as follows:
MSK Cluster: 9 x m5.12xlarge
Kafka version: 2.8.0
Authentication: IAM
Here is test spec into step function:
'''
{
"test_specification": {
"parameters": {
"cluster_throughput_mb_per_sec": [
1200,
1600,
2000
],
"consumer_groups": [
{
"num_groups": 1,
"size": 400
}
],
"num_producers": [
40
],
"client_props": [
{
"producer": "acks=all linger.ms=5 batch.size=262114 buffer.memory=2147483648 security.protocol=SASL_SSL",
"consumer": "security.protocol=SASL_SSL"
}
],
"num_partitions": [
400
],
"record_size_byte": [
5120
],
"replication_factor": [
3
],
"duration_sec": [
360
]
},
"skip_remaining_throughput": {
"less-than": [
"sent_div_requested_mb_per_sec",
0.995
]
}
}
}
'''
The AWS step function state machine is stuck in RunPerformanceTest. The CloudWatch dashboard also confirmed that no traffic is generated. There are 440 jobs being scheduled. 152 jobs were stuck in waiting for all the jobs to start (wait_for_all_tasks), and 288 jobs failed because of the following error:
2023-05-29T05:41:15.850Z
An error occurred (TooManyRequestsException) when calling the GetBootstrapBrokers operation (reached max retries: 4): Too Many Requests
Looks like the AWS API GetBootstrapBrokers has been rate limited.
Is this kind of test scale supported by this repository?