LiteFS - SQLite DB를 클러스터에 복제하는 FUSE기반 파일 시스템 (github.com/superfly)

LiteFS - SQLite DB를 클러스터에 복제하는 FUSE기반 파일 시스템을 소개합니다.

깃허브에서는 LiteFS 을 아래와 같이 소개하고 있네요..
LiteFS is a FUSE-based file system for replicating SQLite databases across a cluster of machines. 

한마디로 LiteFS 을 정의하자면, SQLite DB를 클러스터에 복제하는 FUSE기반 파일 시스템이라고 머릿속에 넣어두시면 될것 같네요...

사용방법은 아래와 같고요...

Usage

Install dependencies

LiteFS currently only runs on Linux and it requires the FUSE 3 library which is available on newer Linux system. To install, run the following:

apt install fuse3 libfuse-dev

You will also need to run Consul for leader election. You can run in development mode if you're testing locally:

consul agent -dev

Configure & run LiteFS

You'll also need a config file for your LiteFS instance. You can find an example config in the repo or you can simply set a few of the fields like this:

mount-dir: "/path/to/mnt"

http:
  addr: ":20202"

consul:
  url: "http://localhost:8500"
  advertise-url: "http://localhost:20202"

Then create the mount directory:

mkdir /path/to/mnt

And run LiteFS by passing in the path to your config:

litefs -config /path/to/litefs.yml

Testing your setup

You can now run SQLite against your LiteFS directory:

sqlite3 /path/to/mnt/my.db

Executing commands against the database should work the same as a regular file system.

Attaching a second node

You can run another instance of LiteFS with a separate config & mount directory:

mount-dir: "/path/to/another_mnt"

http:
  addr: ":30303"

consul:
  url: "http://localhost:8500"
  advertise-url: "http://localhost:30303"

When you start the second instance, it should say that it is connected to the primary.

Executing SQL write commands on the primary node such as CREATE TABLE or INSERT, it should instantly show the data propagated to the replica node. If you try to execute these write commands on the replica, you'll receive an error as the replicas are read-only.

If you SIGINT (CTRL-C) the first instance, it will destroy the lease and the second node should take over as primary. You can now issue write commands to this new primary node. Connecting your first instance again should begin replicating data right away.

Caveats

If litefs does not exit cleanly then you may need to manually run umount to unmount the file system before re-mounting it:

umount -f /path/to/mnt

litefs will not unmount cleanly if there is a SQLite connection open so be sure to close your application or sqlite3 sessions before unmounting.

마지막으로, LiteFS의 특징을 간단하게 정리해보자면 아래와 같습니다.

• 패스쓰루 파일 시스템처럼 동작해서 SQLite DB의 쓰기 동작을 인터셉트해서 처리하여, 여러대의 머신으로 구성된 클러스터에 복제
• FUSE 3 + Leader Election(Consul 세션) + HTTP Server (노드간 변경 수신용 API 서버)
• 현재는 Linux에서만 동작
• LTX(Liteserver Transaction File) 포맷으로 SQLite의 트랜잭션 데이터를 감지해내고, 트랜잭션 단위로 저장 및 복제
  → WAL방식과는 다르게 replication에 적합하게 최적화된 포맷

 

그외에 자세한 내용은 아래 github 내용을 참고하시길 추천드립니다.

오늘의 블로그는 여기까지고요.
항상 믿고 봐주셔서 감사합니다.