| 1. | Then , we analyzed solutions of the major techniques of the distributed web crawler , such as how the nodes of the crawler cooperate with each other , how the task is distributed , how to keep the important web fresh . we have proposed some practicable arithmetic to solve the problems mentioned above . besides , we implemented a robust , distensible , customized , distributed web crawler , and anatomized it
然后本文分析了分布式网络爬虫的关节技术的解决方法,比如分布式的各个结点如何协同工作,任务如何分配,如何保持重要网页的时新性等等,进而提出了一些实用的算法,解决了这些分布式网络爬虫的关键技术,实现了一个具备健壮性,可扩展性,可配置性的分布式网络爬虫系统,并就该分布式网络爬虫系统进行了仔细的剖析。 |
