Web Crawling Algorithm

A Web Crawling Algorithm is a Information Retrieval Algorithm that can solve a Web Crawling Task.

• Context:
  • It can be applied by a Web Crawling System.
  • It can range from being a Breadth-First Web Crawling Algorithm (using a FIFO queue) to being a Best-First Web Crawling Algorithm (Cho et al., 1998).
• See: Focused Web Crawling Algorithm.

---

---

References

1998

• (Cho et al., 1998) ⇒ Junghoo Choo, Hector Garcia-Molina, and Lawrence Page. “Efficient crawling through URL ordering." Computer Networks and ISDN Systems 30, no. 1 (1998): 161-172.

1994

• (Pinkerson, 1994) ⇒ B. Pinkerton. 1994. “Finding what people want: Experiences with the Web Crawler.” In: Proceedings of the 2nd International World Wide Web Conference

---