FastPrepCrawl Reachable URLs with Concurrent Workers

Crawl Reachable URLs with Concurrent Workers

Scale AI logoScale AI● MediumNEW GRADPHONE SCREEN
Learn

Problem statement

For this practice exercise, urls[i] is a unique, opaque page name, and neighbors[i] contains the indices of its outgoing links. The finite map is immutable; read links from these rows without performing network requests. A page with no map entry is represented by an empty row.

Starting at index start, follow every outgoing link, including links between different hosts. Return every reachable URL, including the start, exactly once in lexicographically increasing order. Cycles, self-links and repeated links are allowed.

Concurrency follow-up

The interview report required converting the crawler to multithreading and explaining where locks belong. For this practice version, use a bounded pool of at most four workers. Make the visited check and claim atomic, synchronize shared work bookkeeping, and finish only when no queued or active expansion remains. Explain why an empty queue alone is insufficient and why each URL is expanded at most once.

Function

crawlReachableUrls(urls: String[], neighbors: int[][], start: int) → String[]

Examples

Example 1

urls = ["a.com","b.com","c.com","z.com"]neighbors = [[1,2],[0],[3],[]]start = 0return = ["a.com","b.com","c.com","z.com"]

The source graph contains the a.com↔b.com cycle and a link to z.com. For this practice example, a.com is the start and the missing z.com map entry has no outgoing links.

Example 2

urls = ["a.com","b.com","c.com","z.com"]neighbors = [[1,2],[0],[3],[]]start = 3return = ["z.com"]

Starting at the terminal page reaches only that page.

Example 3

urls = ["c.com","a.com","b.com","x.com"]neighbors = [[1,2],[2],[0],[]]start = 1return = ["a.com","b.com","c.com"]

The cycle reaches three pages; x.com is disconnected. Output order is lexicographic, independent of worker scheduling.

Constraints

  • 1 <= urls.length <= 5000 and neighbors.length == urls.length.
  • There are at most 50000 outgoing references in total; every neighbor index and start is in [0, urls.length).
  • URL names are unique, case-sensitive, non-whitespace printable ASCII strings of length 1 to 128. Treat them as opaque names without normalization.
  • Lookups are finite, immutable and non-failing. The worker limit is a fixed practice resource choice; no extra input parameter is supplied.

More Scale AI problems

See Scale AI hiring insights
public String[] crawlReachableUrls(String[] urls, int[][] neighbors, int start) {
    // write your code here
}
urls["a.com","b.com","c.com","z.com"]
neighbors[[1,2],[0],[3],[]]
start0
expected["a.com", "b.com", "c.com", "z.com"]
Checking account…