Make featherless more reliable

[bkorycki]

Problem: we were hitting a lot of concurrency limit errors with featherless. Our account allows 10 concurrent "units", and with 32 workers we are constantly exceeding our limit throughout a benchmark run. We re-tried these errors 14 times, but this was inadequate and still resulted in many failed items in a benchmark.

Solution:

• Address the root concurrency problem with a semaphore. Dynamically set the upper-bound on the semaphore by getting the model's "cost" from featherless (e.g. requests to small models cost 1 unit, requests to medium models cost 2 units, etc...).
• Indefinitely retry concurrency errors as a backup in case we accidentally are running multiple featherless suts at once. This shouldn't happen because in practice we use concurrency keys in modellab. But this practice is entirely manual and subject to error.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅