Using checksums to find data downloads updates

Our checksums API endpoint provides a hacky yet functional way to find if we have updated data for you.

One of the most common questions we get from our users is how to tell if the data downloads are updated and ready for them to download. In our FAQ, we have mentioned that:

Data is updated:

  • Daily - Every day, at 5 pm UTC
  • Weekly - First Monday of the week, at 5 pm UTC
  • Monthly - First Monday of the month, at 5 pm UTC

As you know, our dataset is backed by probe network scans, complex models and algorithms. These are computationally heavy operations, and there is a nature of unpredictability when it comes to data delivery time. More often than not our data is sometimes updated and ready for download before the schedule mentioned above. If you want the freshest data possible as soon as they are ready to download, you can take advantage of our checksums API and your existing data downloads.

Checksums API

Checkout out checksums documentation post ↓

Our data downloads come with a checksums API endpoint. To summarize, to get the checksums value of your intended download, you can call the /checksums API endpoint like so:<data_download_id>/checksums?token=$token

This will return a payload like this:

  "checksums": {
    "md5": "ec1e0b806d09a4d367514fca3d7bae2e",
    "sha1": "488b50ea5c4cb97378bfcaa78b38305def65c65f",
    "sha256": "389294a764a8989d41d74608f1515133f3b910585ad6b75cfe53a81c82f7838f"

You can compare the checksums hash value with the hash values generated by your database to verify the data integrity.

Using checksums to verify data download updates

The Checksums API endpoint provides the most convenient way to get information about our data downloads. Moreover, with our rate limit policies, you can not download our database more than three times a day.

To check if a new data download is available, you can compare the hash values of your current database and check the values from the checksums API endpoint. If the values match, you are on the latest DB. If the values don’t match, an updated data download is available.