p5-WWW-Robot-0.026p0
configurable web traversal engine
Back to search · Project homepage
Description
This module implements a configurable web traversal engine, for a robot
or other web agent. Given an initial web page (URL), the Robot will get
the contents of that page, and extract all links on the page, adding
them to a list of URLs to visit.
Features of the Robot module include:
* Follows the Robot Exclusion Protocol.
* Supports the META element proposed extensions to the Protocol.
* Implements many of the Guidelines for Robot Writers.
* Configurable.
* Builds on standard Perl 5 modules for WWW, HTTP, HTML, etc.
Package information
- Ports path
- www/p5-WWW-Robot
- Package architecture
- *
- Maintainer
- The OpenBSD ports mailing-list <ports@openbsd.org>
- Categories
- www, perl5
- Available flavors
- None listed
These are ports metadata. Binary availability depends on the release, architecture and mirror. Build and test dependencies are not an installation checklist.