FastPrepN Gram Next Token Prediction

N Gram Next Token Prediction

Runway logoRunway● MediumFULLTIMEPHONE SCREEN
Learn

Problem statement

Fit an n-gram model from the supplied training sentences, then predict one next token for each query context.

Split each sentence and context on whitespace without crossing sentence boundaries. Use the final n - 1 context tokens. Choose the observed successor with the highest count, breaking ties by lexicographically smaller token. Return <unknown> when a complete context was not observed. For n = 1, use global token counts and ignore each context.

Function

predictNextTokens(trainingSentences: String[], n: int, contexts: String[]) → String[]

Examples

Example 1

trainingSentences = ["the cat sat","the cat slept","the dog sat"]n = 2contexts = ["the","cat","dog"]return = ["cat","sat","sat"]

For “the”, cat appears twice; the other contexts each have one most frequent successor.

Example 2

trainingSentences = ["red blue red","blue red green"]n = 1contexts = ["","anything"]return = ["red","red"]

A unigram model ignores context and predicts the most frequent token.

Example 3

trainingSentences = ["a b c"]n = 3contexts = ["a b","b c","a"]return = ["c","<unknown>","<unknown>"]

Only the complete observed two-token context has a successor.

Constraints

  • 1 ≤ trainingSentences.length, contexts.length ≤ 10^4.
  • 1 ≤ n ≤ 5.
  • The total number of whitespace-separated tokens is at most 2 * 10^5.
  • Tokens are non-empty case-sensitive strings.
See Runway hiring insights
public String[] predictNextTokens(String[] trainingSentences, int n, String[] contexts) {
    // Write your code here.
}
trainingSentences["the cat sat","the cat slept","the dog sat"]
n2
contexts["the","cat","dog"]
expected["cat", "sat", "sat"]
Checking account…