N Gram Next Token Prediction
Problem statement
Fit an n-gram model from the supplied training sentences, then predict one next token for each query context.
Split each sentence and context on whitespace without crossing sentence boundaries. Use the final n - 1 context tokens. Choose the observed successor with the highest count, breaking ties by lexicographically smaller token. Return <unknown> when a complete context was not observed. For n = 1, use global token counts and ignore each context.
Function
predictNextTokens(trainingSentences: String[], n: int, contexts: String[]) → String[]Examples
Example 1
trainingSentences = ["the cat sat","the cat slept","the dog sat"]n = 2contexts = ["the","cat","dog"]return = ["cat","sat","sat"]For “the”, cat appears twice; the other contexts each have one most frequent successor.
Example 2
trainingSentences = ["red blue red","blue red green"]n = 1contexts = ["","anything"]return = ["red","red"]A unigram model ignores context and predicts the most frequent token.
Example 3
trainingSentences = ["a b c"]n = 3contexts = ["a b","b c","a"]return = ["c","<unknown>","<unknown>"]Only the complete observed two-token context has a successor.
Constraints
1 ≤ trainingSentences.length, contexts.length ≤ 10^4.1 ≤ n ≤ 5.- The total number of whitespace-separated tokens is at most
2 * 10^5. - Tokens are non-empty case-sensitive strings.