Excel Training Data and Model Response Audit
Problem statement
Audit training records for an Excel assistant, then evaluate the recorded responses of a pretrained model on the valid records. Return the record counts and a decision under the threshold policy below.
Records and normalization
- The arrays
prompts,tables,expectedResponses, andmodelResponseshave the same lengthN. Indexidescribes one record. tables[i]is that record's spreadsheet, represented as rows of integer cells. A table may be empty or ragged; these are conditions the audit must detect.expectedResponses[i]is the reference answer, andmodelResponses[i]is an already recorded model prediction. Do not run a model.- Trim only leading and trailing space (
' '), tab ('\t'), carriage return ('\r'), and newline ('\n'). Preserve case and every interior character. - Count every input record separately, including duplicates.
Valid records and correct predictions
A training record is valid if and only if all three conditions hold:
- Its trimmed prompt is nonempty.
- Its trimmed reference answer is nonempty.
- Its table has at least one row, every row has the same number of cells, and that common number is positive.
For valid records, treat the supplied reference answers as correct. The audit checks the table's structure rather than recomputing the reference answer from its cells. A model prediction is correct exactly when its trimmed text equals the trimmed reference, with case and interior whitespace preserved. An empty model prediction is incorrect; it does not make the training record invalid. Exclude invalid records from model evaluation.
Threshold decision
Return [validRecordCount, invalidRecordCount, correctPredictionCount, fineTuneDecision] in that order:
invalidRecordCount = N - validRecordCount.- If there are no valid records, set
fineTuneDecision = -1for insufficient evaluation data, regardless of the threshold. - Otherwise set
fineTuneDecision = 1if100 * correctPredictionCount < minimumAccuracyPercent * validRecordCount, meaning accuracy is below the required threshold. - Set
fineTuneDecision = 0when the threshold is met. Equality meets the threshold.
This decision is the exercise's fixed accuracy policy for a fine-tuning recommendation.
Function
auditExcelTrainingData(prompts: String[], tables: int[][][], expectedResponses: String[], modelResponses: String[], minimumAccuracyPercent: int) → int[]Examples
Example 1
prompts = ["sum"," ","first"]tables = [[[1,2]],[[3]],[[5],[6]]]expectedResponses = ["3","3","5"]modelResponses = ["3","3","6"]minimumAccuracyPercent = 60return = [2,1,1,1]The blank prompt makes the middle record invalid. Among the two valid records, only the first prediction matches, so accuracy is 1/2, below 60%.
Example 2
prompts = ["p","q"]tables = [[],[[1],[2,3]]]expectedResponses = ["a","b"]modelResponses = ["a","b"]minimumAccuracyPercent = 0return = [0,2,0,-1]The first table has no rows and the second is ragged. There are no valid records to evaluate, even though the threshold is 0%.
Example 3
prompts = ["\t p\n","q"]tables = [[[0]],[[5]]]expectedResponses = [" Yes\r\n","x"]modelResponses = ["Yes"," x\t"]minimumAccuracyPercent = 100return = [2,0,2,0]After trimming only the specified outer characters, both predictions exactly match their references. Accuracy equals 100%, so the threshold is met.
Constraints
0 <= N <= 100; all four record arrays have lengthN.- Each table has
0..10rows, each row has0..10cells, and the dataset contains at most1000numeric cells. Ragged rows are permitted as input. - Every numeric cell is an integer in
[-100, 100]. - Each string has length
0..100. Across all three string arrays there are at most3000characters. - Strings contain only printable ASCII characters with codes
32..126, plus tab, carriage return, and newline. 0 <= minimumAccuracyPercent <= 100.