A question we receive from time to time is whether an IP database can be converted into individual key-value pairs and served from Redis. The idea is appealing. Redis is fast, it is in memory, and key-value lookups are simple to reason about. This post explores when that approach fits, and when it does not, using IPinfo Lite as the example.
The short version is this. The right tool depends on the shape of your data. For databases made of contiguous ranges, MMDB is a strong fit. For databases made of individual, non-contiguous IPs, a key-value store could a strong fit. Lite is the first kind. Converting it to individual keys works against the structure that makes it efficient.
How MMDB stores an IP range
MMDB is a read-only binary file format built for IP lookups. Its index is a binary radix tree, also called a trie, over the bits of the IP address. A lookup converts the IP to binary and walks the tree one bit at a time. The walk ends at a single data record.
Because the tree is organized by IP prefix, a whole CIDR range maps to one record.
This is why IPinfo Lite is compact. Lite holds about 3.8 million ranges and is mostly all inclusive, except for some bogon ranges. As an MMDB file it is around 30 MB. One range, such as 1.0.0.0/24, is a single record that covers all 256 addresses inside it.
The idea: explode ranges into Redis keys
The proposal is to expand every range into its individual IP addresses, store each IP as a key, and put the country data in the value. Each lookup then becomes an exact key match in Redis.
The problem is what this does to the data. Every IP inside a range maps to the same country and continent. Exploding a range does not add any new information. It stores the same value many times over.
A single /24 becomes 256 keys that all carry the same value. Scaled to the whole database, the effect is large.
| Storage form | Entries | Approximate size |
|---|---|---|
| MMDB, ranges | 3.8 million | ~30 MB on disk |
| Redis, exploded, string keys | ~3.7 billion | ~400 GB RAM |
| Redis, exploded, integer keys | ~3.7 billion | ~370 GB RAM |
| Redis, exploded, hash bucketing | ~3.7 billion | ~50 to 80 GB RAM |
| Redis, ranges in a sorted set | 3.8 million | a few hundred MB RAM |
The figures are estimates and vary with the Redis version and allocator. The direction is the point. The database grows from 30 MB to hundreds of GB.
There is also a hard limit. The numbers above are for IPv4 only. IPv6 cannot be exploded at all. IPv6 has 2 to the power of 128 possible addresses. A single /64 holds about 18 quintillion of them. There is not enough storage in existence to expand IPv6 into individual keys. So an exploded key-value store cannot represent the full database on its own. IPv6 has to stay as ranges.
Why the memory cost is so high
The cost is driven by per-entry overhead, not by the key itself. Each key-value pair in Redis carries internal structures: a hash table entry, a string header, a value object, and allocator rounding. That overhead is roughly 80 to 110 bytes per entry before the key and value are counted. Multiply that by 3.7 billion entries and the total reaches the hundreds of GB.
Would a Redis cache layer help?
Yes. A cache layer helps a great deal, and it is the right use of Redis here. The reason it helps is the key to the whole question.
A cache holds only the IPs that are actually being queried. That set is small. A real workload usually touches thousands to a few million distinct IPs, not the whole address space. Entries arrive on the first lookup, are reused on repeat lookups, and expire when they go cold.
If a cache works, why does full key-value storage not work?
The difference is what gets stored.
A cache works precisely because it does not store the cold majority. It keeps the small working set and lets everything else stay out of memory. Full materialization is the opposite. It stores all 3.7 billion IPv4 addresses whether they are ever queried or not. Most of those entries are never touched. You pay for the entire space to serve a tiny fraction of it.
So the cache gives you the per-IP speed you want for repeated lookups, at a fraction of the memory. The full key-value store gives you the same speed at thousands of times the cost.
What about converting IPs to integers as keys?
Using an integer key instead of a text key helps a little, but not enough to change the conclusion. An IPv4 integer is 4 bytes, shorter than the text form of up to 15 bytes. That saves perhaps 10 bytes on a roughly 100 byte entry, which is about 10 percent. The result is still in the hundreds of GB, because the per-entry overhead is the real driver.
Integer keys also do nothing for IPv6, because the problem there is the number of keys, not the size of each key.
Integer conversion is still useful, just in a different place. Converting range boundaries to integers is exactly what enables the sorted-set range approach below.
A better fit: keep the ranges in Redis
If you want to serve Lite from Redis, store the ranges, not the individual IPs. Redis sorted sets support range lookups. Encode each range start as a fixed-width big-endian byte string, then use ZRANGEBYLEX to find the range that contains a queried IP.
python
import redis
r = redis.Redis()
# Encode an IP as a fixed-width, big-endian byte string.
# IPv4 uses 4 bytes, IPv6 uses 16 bytes. Use a version tag to keep them separate.
def encode_ipv4(ip_int):
return b"4:" + ip_int.to_bytes(4, "big")
# Load: member = encoded range start + payload, score = 0 (ordering is lexical).
# r.zadd("lite:v4", {encode_ipv4(start_int) + b"|" + b"FR,France,EU,Europe": 0})
# Lookup: find the greatest range start that is less than or equal to the query IP.
def lookup_v4(ip_int):
key = encode_ipv4(ip_int)
hit = r.zrevrangebylex("lite:v4", b"[" + key, b"-", start=0, num=1)
return hit[0] if hit else None
This keeps the dataset at 3.8 million entries instead of billions. It fits in a few hundred MB of RAM, covers both IPv4 and IPv6, and gives O(log N) lookups.
One caveat is worth noting. Redis sorted set scores are 64-bit floating point numbers. They have a 53-bit mantissa and cannot hold a 128-bit IPv6 value precisely. That is why the byte-encoded lexical approach with ZRANGEBYLEX is the safer path. It works the same way for IPv4 and IPv6.
A practical architecture
The cleanest design combines both ideas. Keep a compact source of truth, and put a small cache in front of it for repeated lookups.
The source of truth is either the MMDB file at 30 MB or the ranges loaded into a Redis sorted set. The cache holds the hot IPs as individual keys with a TTL. On a hit, the cache returns the answer immediately. On a miss, the source of truth answers and the result is written to the cache. This gives the in-memory speed for hot IPs, keeps memory use low, and covers both IP versions.
Summary
| Question | Answer |
|---|---|
| Is MMDB a good fit for Lite? | Yes. Lite is contiguous ranges, which MMDB is built for. |
| Does exploding ranges add information? | No. Every IP in a range shares the same value. |
| What is the memory cost of exploding IPv4? | Hundreds of GB in Redis, versus 30 MB as MMDB. |
| Can IPv6 be exploded? | No. The address space is too large to materialize. |
| Does a cache layer help? | Yes. It stores only the hot working set. |
| Do integer keys solve it? | No. They save about 10 percent and do not help IPv6. |
| What is the recommended Redis design? | Store ranges in a sorted set, with a small cache for repeats. |
MMDB and a key-value store are both excellent tools. They are built for different data shapes. For contiguous ranges like Lite, ranges are the efficient unit. For non-contiguous individual IPs, such as a residential proxy dataset, key-value stores like LMDB shine.