Universal Crawler for web or local file system crawlering. The module enables you to extend the functionality of any location-related searches in really easy way - by overriding the appropriate default functions in the subclass. For example you can change default web filters or add some statistics. In "thread-version" branch you can also find multi-threaded version. In "icm" branch you cand find extended module for collecting statistics from logs of Apache Hadoop jobs' tasks. In master you can find Demo1 and Demo2 - extensions of crawler. Let me know if you will use my code, have some suggestions or just find this helpfull :)
-
Notifications
You must be signed in to change notification settings - Fork 1
Universal Crawler for web or local file system crawlering. The module enables you to extend the functionality of any location-related searches in really easy way - by overriding the appropriate default functions in the subclass. For example you can change default web filters or add some statistics. In "thread-version" branch you can also find mu…
wosiu/Crawler
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
About
Universal Crawler for web or local file system crawlering. The module enables you to extend the functionality of any location-related searches in really easy way - by overriding the appropriate default functions in the subclass. For example you can change default web filters or add some statistics. In "thread-version" branch you can also find mu…
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published