{"schema_version":1,"metadata":{"branch":"current","release":"7.9","architecture":"amd64","generated_at":"2026-09-09T04:04:41.004595+00:00","source_url":"https://cdn.openbsd.org/pub/OpenBSD/snapshots/packages/amd64/sqlports-7.55.tgz","source_sha256":"fcdbba1b9df747882aa9bab4af20ebec8dac10655069c90f1cd4145c06e1235e","package_count":12031,"source_kind":"sqlports"},"package":{"name":"crawl-0.4p5","path":"net/crawl","url":"/packages/current/net/crawl/","comment":"small and efficient HTTP crawler","homepage":"http://www.monkey.org/~provos/crawl/","maintainer":"The OpenBSD ports mailing-list <ports@openbsd.org>","description":"The crawl utility starts a depth-first traversal of the web at the\nspecified URLs.  It stores all JPEG images that match the configured\nconstraints. Crawl is fairly fast and allows for graceful termination. \nAfter terminating crawl, it is possible to restart it at exactly the\nsame spot where it was terminated. Crawl keeps a persistent database\nthat allows multiple crawls without revisiting sites.\n\nThe main reason for writing crawl was the lack of simple open source\nweb crawlers. Crawl is only a few thousand lines of code and fairly\neasy to debug and customize. \n\nFeatures\n\n+ Saves encountered JPEG images \n+ Image selection based on regular expressions and size constraints \n+ Resume previous crawl after graceful termination \n+ Persistent database of visited URLs \n+ Very small and efficient code \n+ Supports robots.txt \n","package_architecture":"amd64","stem":"crawl","readme":null,"dependencies":[{"path":"databases/db/v3","type":"library","package_spec":"","url":"/packages/current/databases/db/v3/"}],"reverse_dependencies":{"count":0,"url":null},"categories":["net"],"flavors":[],"only_for_architectures":[],"not_for_architectures":[]}}