Repository navigation
UAA ratelimit is not handled and blocks integrationtests #1307
Description
Activity
I'm getting into the details of this, and I have a few questions:
- What's the current rate limit config for your UAA instance?
- When you say "HTTP 429 are returned", you mean that CF-java-client gets HTTP 429 from the UAA, correct?
- If that's the case, which requests exactly are being throttled?
I'm surprised that the tests make many requests to the UAA in parallel, most of the time CFJC should have a cached token that can be reused across many requests ; and with more recent additions to our tests, we have only 11 classes with
@CleanupCloudFoundryAfterClasswhich trigger the creation of a new client.I'm provisioning to test this out myself.
Initial testing suggests that the
userIdbean in theIntegrationTestConfigurationis a heavy user:19:03:51.732 POST /oauth/clients 19:03:52.460 POST /Users 19:03:53.116 GET /Groups?filter=displayName%20eq%20%22zones.read%22 19:03:53.287 GET /Groups?filter=displayName%20eq%20%22routing.routes.read%22 19:03:53.303 GET /Groups?filter=displayName%20eq%20%22clients.admin%22 19:03:53.306 GET /Groups?filter=displayName%20eq%20%22routing.routes.write%22 19:03:53.310 GET /Groups?filter=displayName%20eq%20%22scim.userids%22 19:03:53.313 GET /Groups?filter=displayName%20eq%20%22network.admin%22 19:03:53.315 GET /Groups?filter=displayName%20eq%20%22routing.router_groups.read%22 19:03:53.319 GET /Groups?filter=displayName%20eq%20%22scim.zones%22 19:03:53.322 GET /Groups?filter=displayName%20eq%20%22scim.create%22 19:03:53.324 GET /Groups?filter=displayName%20eq%20%22scim.read%22 19:03:53.327 GET /Groups?filter=displayName%20eq%20%22scim.invite%22 19:03:53.329 GET /Groups?filter=displayName%20eq%20%22routing.router_groups.write%22 19:03:53.332 GET /Groups?filter=displayName%20eq%20%22uaa.admin%22 19:03:53.334 GET /Groups?filter=displayName%20eq%20%22cloud_controller.admin%22 19:03:53.336 GET /Groups?filter=displayName%20eq%20%22zones.write%22 19:03:53.338 GET /Groups?filter=displayName%20eq%20%22idps.write%22 19:03:53.341 GET /Groups?filter=displayName%20eq%20%22clients.secret%22 19:03:53.351 GET /Groups?filter=displayName%20eq%20%22scim.write%22 19:03:53.649 POST /Groups/d351ac13-496c-4d63-a9d7-492c6d480f9f/members 19:03:53.797 POST /Groups/61e301c3-395f-4ef6-b1c7-83f6192b40a4/members 19:03:53.811 POST /Groups/fabf4bc4-53f8-464e-871d-e20951e1a5be/members 19:03:53.980 POST /Groups/907adadd-1cc2-490e-948e-9f64656a30da/members 19:03:54.007 POST /Groups/3dc60024-abb2-4a38-81a1-4e0bb81fe26e/members 19:03:54.010 POST /Groups/200407fe-054e-4bf6-8eb0-8624058f0c37/members 19:03:54.016 POST /Groups/12ff350f-83ae-47b0-82c1-989e632d1831/members 19:03:54.021 POST /Groups/9068507f-d84a-4bff-bd24-84faefc9472b/members 19:03:54.026 POST /Groups/bf7e75ad-e5cc-4d27-8c58-bb29fce264d6/members 19:03:54.032 POST /Groups/c27f86a1-e6a9-47c5-af5a-7504fd0b8f52/members 19:03:54.033 POST /Groups/4a817717-9450-4218-9d87-fe85ec005b00/members 19:03:54.048 POST /Groups/6b9ad6f1-c049-490e-ba2a-3b3ea6f3892b/members 19:03:54.051 POST /Groups/c0f57873-7489-46c9-8736-ee65a2fbf878/members 19:03:54.058 POST /Groups/8f489e93-7cb7-41d8-80cd-6300ef02c2de/members 19:03:54.071 POST /Groups/ad9d528c-7c0a-47cc-84b5-ceaa4ca9ce59/members 19:03:54.075 POST /Groups/9fb7762b-861a-46d9-9305-0fddba342fd0/members 19:03:54.076 POST /Groups/996bd947-6a19-40ba-a797-9a52ab6965af/members 19:03:54.081 POST /Groups/b481bac2-475d-48fd-ae60-95f567024033/membersI'm going to run the full test suite and see if there are other problematic tests.
I think the configuration used in my instance is this:
uaa_rate_limiter: credentialID: 'JWTjsonField:1:zid' limiterMappings: - name: AuthToken withCallerRemoteAddressID: 20r/s global: 60r/s pathSelectors: [ "equals:/oauth/token" ] - name: AuthAuthorize withCallerRemoteAddressID: 20r/s global: 60r/s pathSelectors: [ "equals:/oauth/authorize" ] - name: SCIM withCallerCredentialsID: 5r/s pathSelectors: - "startsWith:/Users" - "startsWith:/Groups"
So any call to /Users and /Groups is limited to 5 requests per second. Other limits do not create problems in my case.
My code checks all requests to "uaa" and does not interfere with requests to "api".
In the integration tests, I only evaluate one variable 'UAA_API_REQUEST_LIMIT' and use that value for anything requesting uaa. That's a bit of "overblocking", but I did not find a way to access the configuration in a programatically way.I will rebase and undo the changes that are related to bean @qualifier.
Thanks. For the record, here's the formatted version:
uaa_rate_limiter: credentialID: 'JWTjsonField:1:zid' limiterMappings: - name: AuthToken withCallerRemoteAddressID: 20r/s global: 60r/s pathSelectors: - "equals:/oauth/token" - name: AuthAuthorize withCallerRemoteAddressID: 20r/s global: 60r/s pathSelectors: - "equals:/oauth/authorize" - name: SCIM withCallerCredentialsID: 5r/s pathSelectors: - "startsWith:/Users" - "startsWith:/Groups"
If UAA is configured with ratelimiting, the integration tests do not work and productive code will fail in unpredictable ways when http 429 errors are returned.
Fixed in #1313 Adding a new API, available for customers but adding maintenance load for us.
Alternative fix in #1332 Improving the integration tests without huge new API. Code in integration-test can be used as template for customers facing the same problem.