Reliable LLM Call with Independent Retry Budgets
Updated
Problem statement
You are writing reliable_call, a helper that sends a conversation to an LLM API, checks each response with a validator, and retries when something goes wrong. In this exercise there is no real API. Instead, a script says what each call does. Calls are counted from 1: call 1 has the outcome outcomes[0], the response text responses[0] and the reason reasons[0], call 2 uses index 1, and so on. In the rules below, i is the list index for the current call (call i + 1).
Each message is a string role: content. The history starts as a copy of messages; messages itself is never changed. Every API call is sent the current history. The outcomes are:
"CLIENT_ERROR": the API call fails with a client error and returns no response. If the client retries used so far equalmaxClientRetries, stop withclient_error. Otherwise use one client retry, addCLIENT_ERROR_MESSAGE, which is"user: The API call failed. Please try again.", to the history, and call again."VALID": the call returnsresponses[i]and the validator accepts it. Stop withsuccess."INVALID": the call returnsresponses[i]and the validator returns false. The reason isvalidation returned False."VALIDATION_ERROR": the call returnsresponses[i]and the validator raises a validation error whose reason isreasons[i].
After "INVALID" or "VALIDATION_ERROR", the response is a failed response. If the validation retries used so far equal maxValidationRetries, stop with validation_error. Otherwise use one validation retry, add "assistant: " + response to the history, then add "user: That response was invalid: " + reason + ". Please try again with valid JSON.", and call again.
The two budgets are independent. A client error uses only a client retry, a failed response uses only a validation retry, and neither count is ever reset. A budget of 0 allows no retry for that kind of failure.
Return a list of strings, in this order:
- The status:
"success","client_error"or"validation_error". For an error, this is itsfailure_reason. - For a success, the accepted response. For an error, its message:
"Client error limit reached"or"Validation error limit reached". attempts: the total number of API calls made, as a decimal string.last_response: the response of the most recent call that returned one, or"None"if every call failed with a client error.- The number of failed responses as a decimal string, followed by the failed responses in call order (
all_responses). - The history that was sent with the last API call, one message per string.
Function
reliableCall(messages: String[], outcomes: String[], responses: String[], reasons: String[], maxClientRetries: int, maxValidationRetries: int) → String[]Examples
Example 1
messages = ["system: Reply with JSON only.","user: What is the total of order 17?"]outcomes = ["CLIENT_ERROR","INVALID","VALID"]responses = ["","Sure! It is 42.","{\"total\": 42}"]reasons = ["","",""]maxClientRetries = 1maxValidationRetries = 1return = ["success","{\"total\": 42}","3","{\"total\": 42}","1","Sure! It is 42.","system: Reply with JSON only.","user: What is the total of order 17?","user: The API call failed. Please try again.","assistant: Sure! It is 42.","user: That response was invalid: validation returned False. Please try again with valid JSON."]- Call 1 fails with a client error. One client retry is allowed, so the client error message is added and the call is retried.
- Call 2 returns
Sure! It is 42., and the validator returns false. One validation retry is allowed, so the response is added as an assistant message, followed by the correction with the reasonvalidation returned False. - Call 3 returns
{"total": 42}, which passes. The result is a success after3calls, with one failed response. The last five strings are the messages sent with call 3.
Example 2
messages = ["system: Reply with JSON only.","user: What is the total of order 17?"]outcomes = ["VALIDATION_ERROR","VALID"]responses = ["{total: 42}","{\"total\": 42}"]reasons = ["missing quotes around total",""]maxClientRetries = 2maxValidationRetries = 0return = ["validation_error","Validation error limit reached","1","{total: 42}","1","{total: 42}","system: Reply with JSON only.","user: What is the total of order 17?"]- Call 1 returns
{total: 42}, and the validator raises a validation error. - The validation budget is
0, so there is no retry: the call stops withvalidation_errorafter1call. The second scripted outcome is never used. - The only call was sent the two original messages.
Example 3
messages = ["system: Reply with JSON only.","user: What is the total of order 17?"]outcomes = ["VALIDATION_ERROR","CLIENT_ERROR","CLIENT_ERROR","CLIENT_ERROR"]responses = ["[]","","",""]reasons = ["expected an object","","",""]maxClientRetries = 2maxValidationRetries = 1return = ["client_error","Client error limit reached","4","[]","1","[]","system: Reply with JSON only.","user: What is the total of order 17?","assistant: []","user: That response was invalid: expected an object. Please try again with valid JSON.","user: The API call failed. Please try again.","user: The API call failed. Please try again."]- Call 1 returns
[]and fails validation, which uses the one validation retry. - Calls 2 and 3 fail with client errors and use both client retries. The validation failure did not use any of them, because the budgets are separate.
- Call 4 fails with a client error and no client retry is left, so the result is
client_errorafter4calls. The last response received was[], from call 1.
Constraints
1 <= messages.length <= 10- Each message has the form
role: content, whereroleissystem,userorassistantandcontenthas 1 to 40 characters. 1 <= outcomes.length <= 50responses.length == reasons.length == outcomes.lengthoutcomes[i]is"CLIENT_ERROR","VALID","INVALID"or"VALIDATION_ERROR".responses[i]is""whenoutcomes[i]is"CLIENT_ERROR". Otherwise it has 1 to 40 characters and is not"None".reasons[i]is""unlessoutcomes[i]is"VALIDATION_ERROR". Then it has 1 to 40 characters and does not end with a period.- Message contents, responses and reasons use only letters, digits, spaces and the symbols
{ } [ ] " : , . ! ? ' - _, with no leading or trailing space and no two spaces in a row. 0 <= maxClientRetries, maxValidationRetries <= 20- The script is long enough: the process always stops at or before the last outcome. Outcomes after the stopping call are never used.
More Stripe Online Assessment questions
- Asynchronous Event ProcessingOA · Seen Nov 2026
- Maximum Game Score with Fixed Forward JumpsOA · Seen Nov 2026
- Request Routing SystemOA · Seen Nov 2026
- Financial Account LedgerOA · PHONE SCREEN · Seen Nov 2026
- Chat Billing CalculationOA · Seen Nov 2026
- Deployment Window SchedulerOA · Seen Nov 2026
- Subscription and Usage-Based Billing CalculatorOA · Seen Nov 2026
- Split Tender Part 4 - RefundsOA · PHONE SCREEN · Seen Nov 2026
- Incident MonitorOA · Seen Nov 2026
- Split Tender Part 3 - SurchargesOA · Seen Nov 2026
Similar questions at other companies
- Parcel Event TrackingAirbnb · OAAirbnb interview questions
- Communications HandlerAkuna Capital · OAAkuna Capital interview questions
- Text Editor OperationsMicrosoft · OAMicrosoft interview questions
- Versioned Recipe SystemThe Trade Desk · OA
- In-Memory Filesystem CommandsPerplexity AI · PHONE SCREEN · ONSITE INTERVIEWPerplexity AI interview questions
- Stateful Log Transform and SearchRippling · PHONE SCREEN · ONSITE INTERVIEWRippling interview questions
- In-Memory Vault File SystemHarvey · ONSITE INTERVIEWHarvey interview questions
- File Storage ServiceCresta · PHONE SCREENCresta interview questions
- Query JSON Records with WHERE and ORDER BYGitHub · ONSITE INTERVIEW
- Return Every Matching DecryptionDuolingo · PHONE SCREENDuolingo interview questions