23 Flat File Database Open-source, JavaScript, Rust, Python, Java, PHP, and Go
What is a Flat File Database?
Flat-file databases, well, they're a kind of database that keep data in a plain text file, right? Every line of that text file holds a record, with fields split by delimiters, like commas or tabs.
Some of them don't have the fancy relational structure of other database types, but they're pretty simple and straightforward to use.
Where can you use these flat-file databases?
Well, there's a lot of places. They're a good fit for smaller applications, where you just need to read or write records with a simple structure. Like, they're great for storing stuff like configuration data, user accounts, or any other data that's pretty simple in structure.
Well, there's a bunch of reasons. They're really portable and don't need a lot of setup, which makes them perfect for prototyping and small projects. They don't need a separate database server, which can make deploying them a lot easier. Plus, because the data is just plain text, you can easily look at it and edit it with a text editor.
One cool thing you can do with flat-file databases is caching. By keeping frequently used data in a cache, the database doesn't have to read from the file system every time that data is needed. This can speed things up a lot, especially for workloads where there's a lot of reading going on.
Flat-file DB limitations
But, you gotta remember, flat-file databases have their limits. They don't have the fancy querying stuff that SQL databases do and they can't handle a ton of data. They also can't support a bunch of writers at the same time. So, while they're a great choice for some things, they're not gonna be the best fit for every application.
1. LowDB
LowDB is a simple flat-file database library that resembles MongoDB queries. It stores and retrieves information directly from a JSON formatted flat-file.
I've utilized it in various web and desktop projects based on Electron for some clients. It has proved to be dependable and speedy. However, I wouldn't recommend it for applications handling large amounts of data.
It supports Typescript and Lodash library.
Features
- Compact and Efficient: This ensures the database does not take up unnecessary resources.
- Minimalist Design: The simple design makes it user-friendly and easy to navigate.
- TypeScript Support: This allows developers who use TypeScript to easily work with the database.
- Plain JavaScript: The database can be used with simple JavaScript, making it accessible to a wide range of developers.
- Safe Atomic Writes: This feature ensures that all data operations are completed safely without any data loss or corruption.
- Customizable: Users have the option to change the storage, file format (such as JSON, YAML, etc.), or even add encryption through adapters. This flexibility allows the database to be tailored to specific project needs.
- Extended Functionality: The database can be extended with lodash, ramda, and more, giving it super powers and making it incredibly versatile.
- Testing Efficiency: For more efficient testing, it automatically switches to a high-speed in-memory mode.
Install
npm install lowdbUsage
import { JSONFilePreset } from 'lowdb/node'
// Read or create db.json
const defaultData = { posts: [] }
const db = await JSONFilePreset('db.json', defaultData)
// Update db.json
await db.update(({ posts }) => posts.push('hello world'))
// Alternatively you can call db.write() explicitely later
// to write to db.json
db.data.posts.push('hello world')
await db.write()
typicode2. NeDB (JavaScript)
NeDB is a powerful and lightweight JavaScript database, drawing inspiration from MongoDB. It seamlessly supports a MongoDB-like API. With its versatile use in Node.js and the browser, it stands as the ideal choice for apps requiring a database solution without the need for a full MongoDB instance.
It is an ideal solution for small web apps, desktop applications, and educational purposes, it's not just a good choice, it's the best choice.
louischatriot3. Teensy (JavaScript)
Teensy is a compact, highly efficient flat file database, adept at storing data in memory and periodically writing it to a file. Its performance metrics showcase its proficiency in swiftly seeking and placing items, maintaining an impressive speed even as the number of items increases.
Install
npm install --save teensyUse
const Teensy = require('teensy');
// create database
let DB = new Teensy('teensy.db', 1000).store();
// put data
DB.put({_id: 'Ringo', _rev: 2, color: 'Orange', fangs: true});4. TSDB (JavaScript)
TSDB is a fast and reliable database port of the original sdb database written in PHP. It is my first TypeScript package, and I welcome any comments or feedback on my website or social media.
Install
npm install ts-sdbUsage
import { TSDB } from 'tsdb';
// Create a new instance of TSDB
const db = new TSDB();
// Load the database
db.load('mydatabase');
// Perform operations on the database
db.create('key1', 'value1');
db.create('key2', 'value2');
db.update('key1', { value: 'newvalue' });
db.delete('key2');
// Save changes to the database
db.save();5. JsonStreamDb (Node.js)
This is a simple flat-file database that uses JSON files to store their data.
Usage
var JsonStreamDb = require('jsonstreamdb');
var db = new JsonStreamDb('test.jsonstreamdb');
db.pipe(process.stdout, {history: true});
db.create('users', '47c0479c-2083-4797-8d3c-419de31d45a7', {userName: 'geon'});
db.update('users', '47c0479c-2083-4797-8d3c-419de31d45a7', {favoriteAnimal: 'kittens'});
db.delete('users', '47c0479c-2083-4797-8d3c-419de31d45a7');6. Flat (PHP)
This is another open-source is a flat-file database with a mongo-like API, written in PHP. It is ideal for smart projects and examples that require CRUD operations.
Use it with caution, as it did not receive updates for years.
mattbit7. Daybreak (Ruby)
Daybreak confidently stands as a rapid, user-friendly key value store for Ruby. It boasts user-defined persistence and in-memory data storage, outstripping pstore and dbm in speed.
It expertly stores data in an append-only file and supports marshalled Ruby objects with finesse. Inclusive of Enumerable for functional methods, Daybreak effortlessly emulates a simple Ruby hash interface.
Install
$ gem install daybreak
Example
require 'daybreak'
db = Daybreak::DB.new "example.db"
# set the value of a key
db['foo'] = 2
# set the value of a key and flush the change to disk
db.set! 'bar', 2
# You can also use atomic batch updates
db.update :alpha => 1, :beta => 2
db.update! :alpha => 1, :beta => 2
# all keys are cast to strings via #to_s
db[1] = 2
db.keys.include? 1 # => false
db.keys.include? '1' # => true
# ensure changes are sent to disk
db.flush
# open up another db client
db2 = Daybreak::DB.new "example2.db"
db2['foo'] = 3
# Ruby objects work too
db2['baz'] = {:one => 1}
db2.flush
# Reread the changed file in the first db
db.load
p db['foo'] #=> 3
p db['baz'] #=> {:one => 1}
# Enumerable works too!
1000.times {|i| db[i] = i }
p db.reduce(0) {|m, k, v| m + k.last } # => 499500
# Compaction is always a good idea. It will cut down on the size of the Database
db.compact
p db['foo'] #=> 1
db2.load
p db2['foo'] #=> 1
# DBs can accessed from multiple processes at the same
# time. You can use #lock to make an operation atomic.
db.lock do
db['counter'] += 1
end
# If you want to synchronize only between threads, prefer synchronize over lock!
db.synchronize do
db['counter'] += 1
end
# DBs can have default values
db3 = Daybreak::DB.new "example3.db", :default => 'hello!'
db3['bingo'] #=> hello!
# If you don't like Marshal as serializer, you can write your own
# serializer. Inherit Daybreak::Serializer::Default
db4 = Daybreak::DB.new "example4.db", :serializer => MyJsonSerializer
# close the databases
db.close
db2.close
db3.close
db4.close
8. Rustbreak (Rust)
Rustbreak is a robust, standalone file database, drawing inspiration from Daybreak and finely tuned for maximum speed and simplicity. It positions itself as the prime choice for swift application development in Rust, providing unparalleled persistence and the ability to write diverse data.
Though it's necessary for the entire database to fit into memory and not all backends extend support for atomic saves, these are minor constraints against the backdrop of its immense advantages.
Importantly, Rustbreak is thread-safe and confidently facilitates Serde compatible storage.
TheNeikos9. Yellow-Table (PHP)
Yellow Table is a simple flat-file database management tool. To use it, download the ZIP file and copy it into your system/extensions folder.
Tables can be displayed using a [table] shortcut, with optional arguments for name, caption, and style. Files are stored in the media/tables folder, but additional folders can be created for organization.
GiovanniSalmeri10. QuartzDB (PHP)
QuartzDB is a superb flat-file database expertly optimized for time-series data. Perfect for storing GPS logs, chat logs, and sensor logs from IoT devices, it's the go-to choice for those who appreciate the ease of backing up text files and have confidence in their long-term durability over binary database files.
aaronpk11. Lazer (PHP)
Lazer is a commanding PHP library that cleverly leverages JSON files as a database, equipped with capabilities akin to ORM's. It assertively requires PHP 7.1+ and Composer.
It stands as a top-tier choice for developers: swift, meticulously documented, offering relations, multiple select and supporting CRUD operations.
Lazer-Database12. SleekDB (PHP)
SleekDB is a NoSQL-like database system implemented in PHP that stores data in plain JSON files without the need for any third-party dependencies. Unlike relational databases such as SQLite, MySQL, PostgreSQL, and MariaDB, SleekDB is more akin to MongoDB in terms of functionality.
It is specifically designed to manage a few gigabytes of data, making it ideal for low to medium operation loads. It is not intended for handling high-load IO operations.
SleekDB is particularly effective as a database engine for websites with low to medium traffic, simplifying database management. Another key feature of SleekDB is its ability to cache all query data by default, making dynamic sites as fast as static ones.
While it is a flat file database, it diverges from the norm by storing data in multiple JSON files rather than a single one. This approach enhances concurrency support compared to fully flat file database systems. The final query output is cached and later reused from a single file, eliminating the need to traverse all available files.
Features
- Lightweight and Fast: SleekDB stores data in plain-text using JSON format, eliminating binary conversions. It also features a default query cache layer.
- Schema-Free: No schema is required, enabling the insertion of any data types.
- Query on Nested Properties: Schema-free data allows filtering and conditioning on nested properties of JSON documents.
- Dependency-Free: Only requires PHP to run and supports PHP 7+. No need for third-party plugins or software.
- Caching: SleekDB employs a default caching layer, serving data from cache by default and regenerating cache automatically.
- Rich Conditions and Filters: Allows for multiple conditional comparisons, text searches, and sorting on multiple and nested properties.
- On-Demand Data Processing: No background process or network protocol is required. All data for a query is fetched at runtime within the same PHP process.
- Universal Compatibility: Runs perfectly on shared servers or VPS.
- User-Friendly: SleekDB has an elegant and easy-to-understand API for data handling.
- Easy Data Management: Data stored in files makes backup, import, and export tasks easier.
- Actively Maintained: Created by @rakibtg and actively maintained by Timucin, SleekDB is continually improved in terms of code quality and new features.
13. FlatFiles (C#)
FlatFiles is an authoritative library that proficiently reads and writes CSV, fixed-length, and other flat file formats. This library prioritizes schema definition and proudly offers direct mapping between files and classes.
The schema, intelligently defined by data columns in your file, expertly guides the reader or writer in extracting or writing out values. FlatFiles commands comprehensive control over parsing/formatting behavior, handling unusual edge cases with ease. For those working with ADO.NET, it also confidently supports DataTables and IDataReader.
jehugaleahsa14. FlatDB (Java)
FlatDB is an incredibly intuitive library for flat-file databases. It empowers you to create tables using Java classes, proactively updates tables when new fields are introduced, and ensures a seamless connection to your database file.
Please note, it necessitates Java 8 or higher.
CyR1en15. Flatbase (PHP)
Flatbase is a PHP-based flat file database that is lightweight, easy to install with minimal configuration, has a simple intuitive API, and is suitable for small data sets, low-load applications, and testing or prototyping.
Installation
composer require flatbase/flatbaseExample
<?php
$storage = new Flatbase\Storage\Filesystem('/path/to/storage/dir');
$flatbase = new Flatbase\Flatbase($storage);
$flatbase->insert()->in('users')
->set(['name' => 'Adam', 'height' => "6'4"])
->execute();
$flatbase->read()->in('users')
->where('name', '=', 'Adam')
->first();
// (array) ['name' => 'Adam', 'height' => "6'4"]
16. Filebase (PHP)
Despite the name, this is a different database than the last one.
Filebase is an uncompromisingly robust flat file database storage solution. There's no need for MySQL or an overpriced SQL server. All database entries are securely stored in files, accessible for direct modification or through our sleek API.
It supports version control, facilitating effortless sharing and syncing of data among teams. It's designed to work seamlessly with PHP 5.6 and PHP 7+.
Features
While Filebase is simple in design, it offers a range of advanced features:
- Storage of Key/Value and Array-based Data
- Data Querying
- Customizable filters
- Query Caching
- Database Backup capabilities
- Data Formatting (encode/decode)
- Data Validation upon saving
- CRUD (Create, Read, Update, Delete) APIs
- File locking when saving
- User-friendly Method Naming
tmarois17. DBJson (Python)
DBJson is a robust flat file database system that confidently safeguards its data within the filesystem, leveraging a blend of folders and JSON files. It's essential to underscore that DBJson is NOT DESIGNED FOR PRODUCTION USE.
Yet, it stands as the optimal solution for situations where the establishment of a formal schema and the execution of additional tasks common to Object-Relational Mapping (ORM) frameworks, like SQLAlchemy, appear excessive or superfluous.
Install
pip install dbjsonExample
from dbjson import DB
# Use the syntax below if you plan to include raw code in your project.
# from dbjson.db import DB
# Initializing DB class from dbjson.main
db = DB()
# Test Data
data = {
"id": 1,
"first_name": "Vivyan",
"last_name": "Treherne",
"email": "[email protected]",
"ip_address": "94.254.247.240"
}
collection = "users"
# Adding Record
data = db.createRecord(collection, data)
print(data)
# Response -> {'__id__': 'f00ae4e3ca8c3e318a68acc460e5f401', '__data__': {'id': 1, 'first_name': 'Vivyan', 'last_name': 'Treherne', 'email': '[email protected]', 'ip_address': '94.254.247.240'}}
# Updating Record
record_key = "f00ae4e3ca8c3e318a68acc460e5f401"
to_update = [
{"email": "[email protected]"},
{"ip_address": "google.com"}
]
data = db.updateRecord(collection, "f00ae4e3ca8c3e318a68acc460e5f401", to_update)
print(data)
# Response -> {'id': 1, 'first_name': 'Vivyan', 'last_name': 'Treherne', 'email': '[email protected]', 'ip_address': 'google.com'}
# Deleting Record
db.removeRecord(collection, record_key)19. Flat-file-db (JavaScript)
Flat-file-db is an efficient flat file database for Node.js. It supports JSON and retains all data in memory. The database uses an append-only method for data persistence, which guarantees small file sizes and robust consistency. To utilize it, install it through npm and provide a database file to the flat-file-db constructor.
Although it is still operational and preferred by many projects, it has not received updates for years.
Install
npm install flat-file-dbUsage
var flatfile = require('flat-file-db');
var db = flatfile('/tmp/my.db');
db.on('open', function() {
db.put('hello', {world:1}); // store some data
console.log(db.get('hello')) // prints {world:1}
db.put('hey', {world:2}, function() {
// 'hey' is now fully persisted
});
});20. TinyDB (Python)
TinyDB is a lightweight, document-oriented database that is written in pure Python, requiring no external servers or dependencies. With a simple and clean API, it is designed for ease of use. It allows storage of any document represented as a dictionary, similar to MongoDB.
For projects necessitating advanced features like multi-process or thread access, table indexing, an HTTP server, managing table relationships, or ACID guarantees, TinyDB might not be your optimal choice.
If your project calls for high-speed database performance, TinyDB might not be the ideal fit. In these instances, it's worth exploring other databases such as SQLite, Buzhug, CodernityDB, or MongoDB.
Features
- Lightweight and document-oriented: TinyDB is efficient, storing any document as a dictionary, much like MongoDB.
- Pure Python: TinyDB is written purely in Python and requires no external servers or dependencies.
- User-friendly: TinyDB comes with a simple and clean API, making it easy and fun to use.
- Compatibility: It is compatible with Python 3.5+ and PyPy.
- Extensibility: TinyDB allows for customization by writing new storages or modifying the behaviour of storages with Middlewares.
- Test Coverage: TinyDB boasts a 100% test coverage.
Example
>>> from tinydb import TinyDB, Query
>>> db = TinyDB('/path/to/db.json')
>>> db.insert({'int': 1, 'char': 'a'})
>>> db.insert({'int': 1, 'char': 'b'})Query language
>>> User = Query()
>>> # Search for a field value
>>> db.search(User.name == 'John')
[{'name': 'John', 'age': 22}, {'name': 'John', 'age': 37}]
>>> # Combine two queries with logical and
>>> db.search((User.name == 'John') & (User.age <= 30))
[{'name': 'John', 'age': 22}]
>>> # Combine two queries with logical or
>>> db.search((User.name == 'John') | (User.name == 'Bob'))
[{'name': 'John', 'age': 22}, {'name': 'John', 'age': 37}, {'name': 'Bob', 'age': 42}]
>>> # Apply transformation to field with `map`
>>> db.search((User.age.map(lambda x: x + x) == 44))
>>> [{'name': 'John', 'age': 22}]
>>> # More possible comparisons: != < > <= >=
>>> # More possible checks: where(...).matches(regex), where(...).test(your_test_func)
21. FlatDB (PHP)
FlatDB is an exceptionally efficient flat file database, masterfully designed to persist data using nothing more than PHP and flat files. It's the ideal solution when no other database is at your disposal.
Features
- Lightweight and Secure
- Easy to get started, just include flatdb.php to your app
- No external dependencies
- Built-in caching system for better performance
- Powerful API with chaining methods
- CRUD (create, read, update, delete) operations
- Supports custom indexes
- Supports WHERE(), LIMIT(), OFFSET(), ORDER(), FIND(), COUNT() and other methods
- Supports multiple databases and tables
onyxhat22. flatfile (Go)
The package confidently provides a powerful utility to read fixed-width records from a flat-file database or text file, effortlessly wiping away the need for boilerplate code for data reading and mapping to struct fields.
The record layout mapping can be swiftly defined using struct tags, and, brilliantly, the package also packs an 'Unmarshal' method to seamlessly convert a record into a struct.
ahmedalhulaibi23. PHP JLDB (JSON Lite DB)
Simple and powerfull tool that allows to use json file like a database. It provides collection of methods that you can use like a database query builder.
mohamed-samir907
