Twitter query - Getting information of thousands of IP addresses

,

Original post.


Prerequisite

As I am generating random IP addresses, there is going to be some amount of Bogon IP addresses. I am using the CLI to generate the random IP addresses.

ipinfo randip -n 1000 -4 > ips.txt

The IPs are stored in ips.txt

CLI Operation

Summarize operation

The summarize command provides high-level information like this.

Code:

ipinfo summarize ips.txt

If you want, you can even output the data in JSON format.

ipinfo summarize -j ips.txt
output (click to expand)
{
  "total": 1000,
  "unique": 1000,
  "countries": {
    "CN": 87,
    "GB": 30,
    "JP": 50,
    "KR": 37,
    "US": 360
  },
  "cities": {
    "Ashburn, Virginia, US": 16,
    "Beijing, Beijing, CN": 16,
    "Columbus, Ohio, US": 49,
    "Seattle, Washington, US": 20,
    "Seoul, Seoul, KR": 18
  },
  "regions": {
    "California, US": 42,
    "England, GB": 28,
    "Ohio, US": 55,
    "Virginia, US": 27,
    "Washington, US": 42
  },
  "asns": {
    "AS4134 CHINANET-BACKBONE": 37,
    "AS4766 Korea Telecom": 21,
    "AS7018 AT\u0026T Services, Inc.": 20,
    "AS749 DoD Network Information Center": 48,
    "AS7922 Comcast Cable Communications, LLC": 19
  },
  "companies": {
    "Amazon Technologies Inc.": 17,
    "DoD Network Information Center": 48,
    "Korea Telecom": 19,
    "Microsoft Corporation": 15,
    "Verizon Business": 22
  },
  "ipTypes": {
    "business": 220,
    "education": 48,
    "hosting": 82,
    "isp": 501
  },
  "routes": {
    "AS31399 53.0.0.0/8": 4,
    "AS749 22.0.0.0/8": 6,
    "AS749 26.0.0.0/8": 6,
    "AS7922 50.128.0.0/9": 4,
    "AS7922 73.0.0.0/8": 6
  },
  "carriers": {
    "Claro": 3,
    "TIM": 4,
    "Türk Telekom": 2,
    "Verizon": 3,
    "Vivo": 5
  },
  "mobile": 42,
  "domains": {
    "comcast.net": 10,
    "myvzw.com": 8,
    "telecomitalia.it": 10,
    "total": 304,
    "verizon.net": 8
  },
  "privacy": {
    "vpn": 3,
    "proxy": 0,
    "hosting": 82,
    "relay": 1,
    "tor": 0
  },
  "privacyServices": {},
  "anycast": 1,
  "bogon": 136
}

Bulk operation

If you want granular information for each IP address, use the bulk command.

If your IP address list is too big, consider chunking the list into manageable sizes first, then iterate over the chunks and run the bulk command on individual lists. I will write the bash script and guide for it in the future, but here is the Stackoverflow post you should use.

I ran 1k IP addresses in a single go and had no problem. I believe the maximum amount of enrichment the bulk command can handle at one time is 10k, but I think it is better to split the IP list into manageable chunks.

Another point is that, output the data in CSV format. Then you can use something like miller or CSVkit’s CSVsql to run queries against. Or you can use Pandas, SQLite or duckdb to run proper analytics. Or you can output the data in JSON and use JQ.

Code:

ipinfo bulk -c ips.txt > ips_enriched.csv

Output

On the free tier you have the geolocation information and ASN information through the org field. We have a free IP to ASN database that you can use to get the AS domain / official website information as well.

You can get additional information like IP privacy information, company information, carrier etc. on the paid tier.