proof-of-work proxy to protect web resources from scrapers
Project homepage: https://anubis.techaro.lol/
Description
Anubis acts as middleware between a reverse proxy and backend web server.
It assesses whether a connection is likely to be from a scraper bot and,
if this seems that there's a chance of this, it issues a SHA-256 proof-
of-work challenge before allowing the connection to proceed.
As of 1.14.x, Anubis decides to present a challenge using this logic:
User-Agent contains "Mozilla"
Request path is not in /.well-known, /robots.txt, or /favicon.ico
Request path is not obviously an RSS feed (ends with .rss, .xml, or .atom)
This should ensure that git clients, RSS readers, and other low-harm
clients can get through without issue, but high-risk clients such as
browsers and AI scraper bots impersonating browsers will get blocked.
When a challenge is passed, a signed JSON Web Token (JWT) is provided
as a cookie, allowing future requests to pass without triggering the
challenge.
Using Anubis will likely result in your website not being indexed by
some search engines. This is considered a feature, not a bug.
Package information
- Ports path
- www/anubis
- Package architecture
- amd64
- Maintainer
- The OpenBSD ports mailing-list <ports@openbsd.org>
- Categories
- www, lang/go
- Available flavors
- None listed
- Only for architectures
- aarch64, amd64, arm, i386, riscv64
Ports list build and test dependencies as well as runtime dependencies. pkg_add selects those needed for installation.
Direct dependencies
Build
OpenBSD README
This README contains unresolved ports variables; replace them before using its commands.
+-------------------------------------------------------------------------------
| Running anubis on OpenBSD
+-------------------------------------------------------------------------------
While Anubis is intended to be run in a Docker container, it can also be
manually configured and is able to run on OpenBSD. It must be "sandwiched"
between a front-end web server (for example, nginx in reverse proxy mode)
and a back-end.
Example configuration with nginx
================================
Anubis
------
Bind and target ports should be specified in ${SYSCONFDIR}/anubis.env.
Here's a contrived example:
export BIND=127.0.0.1:8923
export DIFFICULTY=3
export METRICS_BIND=:9090
export SERVE_ROBOTS_TXT=0
export TARGET=https://www.openbsd.org:443
Anubis listens for incoming HTTP connections on 127.0.0.1:8923, uses a
medium difficulty proof-of-work, and forwards connections to a backend
running at https://www.openbsd.org:443. It provides Prometheus-style
metrics at http://<anubis host>:9090/metrics (not bound to a specific
address here, but that is allowed).
Typically TARGET would be pointed at a backend server which is either
hosted locally, or hosted remotely but firewalled off so that bot traffic
can't reach it directly.
Front-end nginx
---------------
server {
listen 443 ssl;
listen [::]:443 ssl;
http2 on;
server_name somehost.example.org;
ssl_certificate /etc/ssl/server.crt;
ssl_certificate_key /etc/ssl/private/server.key;
ssl_protocols TLSv1.3 TLSv1.2;
ssl_ciphers 'TLSv1.3:TLSv1.2+AEAD+ECDHE';
location / {
proxy_pass http://127.0.0.1:8923;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host www.openbsd.org;
proxy_http_version 1.1;
}
}