ASN-based bot/hosting IP detection is considered one of the most effective ways to prevent bot traffic from your site.
ASN based bot detection
ASN (Autonomous System Number) is a unique identifier for an organization that owns a range of IP addresses. ISPs, large organizations, hosting/data centers, educational institutes, government organizations, etc., tend to be AS organizations.
Your average bot is likely hosted on a server of a hosting provider or data center. These bots use the IP addresses of the hosting provider’s ASN. By identifying the IP address of a bot and, progressively, its ASN, you can ban or block access to all non-human traffic from that hosting provider to your website or services hosted.
Average users tend to use ISP IP addresses, which are categorized under the ISP AS organization, while bots, crawlers, and scrapers use hosting IP addresses, which are categorized under the Hosting AS organization.
We have tons of resources about ASNs and the internet ecosystem in general. I highly recommend you to check out our blogs or our community posts (you are here) to learn more about it.
Now, What do I mean by easily or free? Well…
Identifying bots and hosting IP address easily
If you want an easy solution, you can choose our premium API or database services. We already provide bot detection and data center/hosting IP detection. The detection of AS types and anonymous IPs is backed by sophisticated data and behavior-based models.
Premium API services
- ASN information (AS Type: Hosting, ISP, Business, Education) | Basic API Tier
- Anonymous IP detection (Hosting IP address) | Standard API Tier
Documentation: API Responses - IPinfo.io
Premium database services
Documentation: Database Types - IPinfo.io
Or, Identifying bots and hosting IP address for free
Use a combination of your logs, our free API, and free IP to Country ASN database. For this approach, you have to build your own DIY bot detection service using the free IP to ASN database. Even though you may not have full coverage of anonymous IPs, and you might block a few genuine traffic IPs here and there, this process is an effective method of blocking data center/bot/hosting IP addresses.
To get started, download the free IP to ASN or IP to Country ASN database. Then, identify certain known ASNs that provide hosting services. You can do that by:
- Looking up suspicious IP addresses from your log and getting their ASN information through our API.
- Identifying popular data centers.
- Doing log enrichment with geolocation + asn information through API / Website / CLI.
Once you have the ASNs, you can get the list of IP ranges of those ASNs. Then feel free to just add them in your firewall config file.
Here is how I am doing it in our IP to Country ASN free database available on Snowflake:
I am choosing the following ASNs, which belong to Hetzner:
On Snowflake, I use the country_asn table and the range2cidr UDF.
SELECT
flat_data.value as HETZNER_IP_RANGES
FROM (
SELECT public.range2cidr(start_ip, end_ip) as ip_range
FROM public.country_asn
WHERE ASN='AS24940'
OR ASN='AS213230'
OR ASN='AS212317'
) hetzner_ips,
TABLE(FLATTEN(hetzner_ips.ip_range)) flat_data
This query returns a list of 3,673 IP ranges. You can check out the documentation of our Snowflake listings or community posts on Snowflake to learn about the query. By following this method, you can add more ASNs to the query to get a bigger IP range list. You should keep an eye on your log to identify ASNs of malicious IP ranges. Additionally, you should whitelist some of the ASN IP ranges if you want search engine crawlers to access your site or permit VPN usage.
If you have any questions, feel free to ask here in the community or drop a comment.
Related:
- Preventing bot activity with IPinfo's IP to Privacy Data
- Using IPinfo's IP to Country ASN database in Kaspersky Suricata Rules
- Getting IP data from anonymous IP addresses
IPinfo services referenced:
API
Database
Free / Open access services
Signup for a free account today, and get access to the following services: