LC: 1236. Web Crawler - spiralgo/algorithms GitHub Wiki
The Essence:
The web links can be assumed to be nodes of a graph. What needs to be done is getting the valid neighbors of a webpage and traversing to them if they are not already visited.
Details:
For the traversal, BFS or DFS can be used. Checking if links share the hostname can be easily done using built-in string splitting functions of programming languages.