Hitting unknown rate limit

[carewk]

Hi,
I'm building an app that is sending many requests to the API but we're hitting rate limits earlier than expected. I'm aware of the rate limits specified in the documentation.

For example, one of the requests being sent often is
GET /api/v2/routing/skills
and I'm expecting the rate limit for this to be 300 per minute, as defined by client.credentials.token.rate.per.minute, but I'm seeing requests being blocked (429) after only 100 requests.

The response message for these are as follows:
{"message":"Rate limit exceeded the maximum. Retry the request in [46] seconds","code":"too.many.requests.retry.after","status":429,"contextId":"<>","details":[],"errors":[]}

Notice that there is no "limits" object in this response. Continuing to send this request eventually starts returning the expected rate limit error message, after 300 requests:
{"message":"Rate limit exceeded the maximum. Retry the request in [42] seconds","code":"too.many.requests.retry.after","status":429,"contextId":"<>","details":[],"errors":[],"limit":{"key":"client.credentials.token.rate.per.minute","namespace":"platform.api","value":300}}

I'm unsure where this 100 limit is defined as it seems to be undocumented. I'm getting the same behaviour on retrieving routing queues so unsure if this is just the routing resource. This is not happening to my POST /api/v2/users/search requests, those requests go through up until the expected 180 rate limit for user resources:
{"message":"Rate limit exceeded the maximum. Retry the request in [43] seconds","code":"too.many.requests.retry.after","status":429,"contextId":<>","details":[],"errors":[],"limit":{"key":"user.rate.per.minute","namespace":"directory","value":180}}

Without the limits object in the response, I'm not sure how to investigate what's causing this to block early

UPDATE: Looks like this is only happening when I put a name filter in the query params, like
/api/v2/routing/skills?name=Training
Otherwise it continues until the expected 300 limit. If I add the query param, it falls over with the 429 error after 100 requests with no details or limit object

[Eos_Rios]

As I understand it - Limits have layers, like an onion.
You're only talking about the API specific limits, but there are also categorical and platform level limits as well and it sounds like you may have too many concurrent operations period going on, that's why they're failing with the vague messages, because they don't know about the other operations so can't tell you why you've hit a cap, just that the global throttle said you're in time out.

[natereprogle]

That's interesting, I don't see any documented rate limit on that endpoint either. In my org, hitting the /api/v2/organizations/limits/docs endpoint doesn't show any docs for /routing/skills, and hitting /api/v2/organizations/limits/namespaces/routing doesn't return it in that list either. It's just not there. I'm curious though, why so many calls to /routing/skills in the first place? That data shouldn't change that often, so you should be able to get away with local caching.

[carewk]

After some more investigation, it seems to only happen when the request has a name query parameter like
/api/v2/routing/skills?name=Training

@natereprogle The app handles large amounts of changes to user skills (and more resource changes), and I'm constantly needing to get user IDs, routing skill IDs etc, and make PUT requests to update the skills and their proficiencies. Additionally we have multiple replicas performing these requests. I've introduced some throttling to keep it below 300 requests per minute across all the requests, but I'm only raising this issue as I'm seeing this resource getting blocked at a much lower threshold.

Caching all these resources is something I'm going to start considering if nothing can be done about these limits

[natereprogle]

I think you're correct in your assumption that adding the ?name parameter causes that 100 request rate limit. I used Apache JMeter to test the rate limit and, with ?name in the http parameters, as soon as I hit request #100 I got rate limited.

However, when testing without the ?name parameter, I get rate limited at the expected 300 request mark.

I checked their limits page and just searched for the number 100 to see if there was any limit that may even be remotely related, and the closest one I found was in regards to flows, find.invocations.per.flow.max. But, even then, that's a .max limit and not a rate.per.<interval> limit, so according to their docs it's not even a rate limit by the naming convention.

I think Genesys needs to comment on this post to get more information regarding this mysterious unknown limit.

[John_Carnell]

Hey guys,

Just as a heads-up. I am working to track down the right dev team to take a look at this and provide direct feedback. Thanks for your patience.

John Carnell
Director, Developer Engagement

[carewk]

@John_Carnell Any update with this please? Don't want the thread to just be closed without any solution. Thank you.

[John_Carnell]

Hi Kevin,

Sorry, it has taken a little time to find the right dev team and I was out at a conference last week. Thanks for re-pinging on this. I. just reached out to the right group and they should respond shortly.

Thanks,
John Carnell
Director, Developer Engagement

[Eamonn_Clancy]

Hi Kevin, I've looked into this and you've hit a limit that's enforced by search (see here: https://developer.genesys.cloud/organization/search/search-limits). Calls to /api/v2/routing/skills?name=Training with the name filter are actually directed to search, which is why it's unclear how you ended up hitting that limit. Using the name filter calls search with the CONTAINS search type for the request which has 100 request limit.