We spent months rewriting the 13-year-old file format behind every IPinfo database. Some files came out more than 80% smaller, including the one built to run with no network connection at all.
The MMDB file format has been around almost as long as IPinfo has. A legacy data provider introduced it back in 2013 as the standard container for shipping IP data into firewalls, SIEMs, ad platforms, and everything in between. In the thirteen years since, the format itself barely changed.
It didn’t need to. Once something becomes the default that every tool already expects, there’s little commercial pressure to improve it. The company behind it built a large, durable business on licensing and brand recognition, not on continuing to invest in the format’s engineering.
IPinfo’s data has kept growing the whole time: more attributes per IP, better privacy signals, deeper carrier and hosting detail, wider coverage. Richer data means bigger files, and bigger files were starting to work against us: slower downloads, more memory to hold a database in RAM, less room on constrained hardware. So earlier this year we looked at a part of the pipeline almost nobody in the industry touches anymore: how the database file itself gets written, separate from what’s inside it.
The result is a new way of writing every MMDB file we publish. The data, the schema, and the answer to every lookup are unchanged: a query against the new file returns exactly what it returned against the old one. What changed is how much disk, bandwidth, and memory it takes to store and load that answer.
Every database got smaller. Some dramatically.
We rebuilt all of our MMDB databases the same way, then compared the old file to the new one for the same day’s data. The gain isn’t uniform: it depends on how much repetition is naturally present in a given database. A database where most IP ranges share one of a handful of values (a country, a yes/no flag, a short list of carrier names) has a lot of repeated structure to fold away. A database where nearly every record is closer to unique (an exact organization name per IP block, say) has less to give.
| Database | Before | After | Change |
|---|---|---|---|
| Privacy Detection | 618.4 MB | 123.7 MB | −80.0% |
| Privacy Detection Extended | 736.9 MB | 237.5 MB | −67.8% |
| Country | 23.0 MB | 9.4 MB | −58.9% |
| IP Geolocation | 648.9 MB | 300.5 MB | −53.7% |
| IPinfo Core | 1.42 GB | 692.8 MB | −51.2% |
| IPinfo Lite | 35.1 MB | 24.0 MB | −31.5% |
| Residential Proxy Detection | 13.39 GB | 10.57 GB | −21.0% |
| ASN | 14.1 MB | 12.0 MB | −15.1% |
| Company | 392.7 MB | 375.2 MB | −4.5% |
Across just these nine databases, a single day’s build now ships about 4.9 GB less data than it did under the old writer, with nothing removed from what any of them return.
IPinfo Lite: The one built to run offline
IPinfo Lite is IPinfo’s free, static database: country, ASN, and organization data for every IP range, shipped as a single downloadable file instead of a live API. Because it’s a file rather than a network lookup, no query it answers ever has to leave the machine running it: there’s no request to log, no round-trip to depend on, and no service that can go down. That property matters most exactly where it’s hardest to get: on IoT devices, inside air-gapped networks, and in any privacy-sensitive application that simply can’t send an IP address somewhere else to look it up.
Lite’s file dropped from about 35 MB to about 24 MB in the same rewrite, a 31% reduction, and it’s held in that range since, growing only with the normal, daily addition of new data.
A smaller file is what makes the offline use case realistic on the hardware that actually needs it. A 35 MB database was already workable on something like a Raspberry Pi; it was not workable on a microcontroller with a few megabytes of flash, or as a bundled asset inside a mobile app that already has its own size budget to protect. At 24 MB, and shrinking further as this work continues, IPinfo Lite fits into a lot more of those places than it used to.
This was our first pass at a file format almost no one else in the industry had touched in over a decade. We’re continuing the work. Expect these numbers to move again.