MFdb is a portable P2P database backed by the IPFS Mutable File System

American Sports Oligarch uses IPFS to share sports data for the user interface.

Sports data has a relatively small number of records (a few hundred thousand) that are usually filled with small integers and floats so the raw size of the datasets is small enough to browse locally. But it’s large enough that loading it all into the browser memory to search is impractical.

The real world version of these datasets are prone to DDOS attacks because they’re hosted by the sports league and they’re really just trying to power their website.

So the goal is

• Build indexed dataset on a small machine. Store in local MFS.
• Update it daily.
• Pin on a commercial IPFS host.
• Publish new CID somewhere.
• Users see new version and pin the CID to their local MFS. Only new data should need to be downloaded.
• Data is already indexed in b-trees so users search and filter it in their own browser without needing to load the full dataset.
• Users share the data P2P primarily. The pin is a backup.

Need to be able to do basic sorting, filtering, and paging on this data in the UI without needing a cloud server to handle it. Performance in the prototypes I’m building is tolerable for these datasets but I’m definitely looking to improve it. The interface is still new and in flux but generally speaking it’s logically like a SQL database. There’s a schema and indexes are built when you insert and update data. There are transactions and you commit changes to write them to disk.

The IPFS Mutable File System is used as a transport format (sorta). The main benefit is that we can transport it directly in a format that can be searched immediately. Just load the b-tree for the appropriate column and only load the records that you need. SQL engines are very good at these use cases.

Or you can host the data on the cloud and search it from javascript directly.

There’s examples in the tests but I have a better proof of concept coming soon with the sports dataset.

Link: https://gitlab.com/american-space-software/mfdb