From e4a825976e56bde128629e2240dc41ddc919838e Mon Sep 17 00:00:00 2001 From: Fran Date: Fri, 27 Oct 2023 19:13:38 +0200 Subject: [PATCH] updated readme --- README.md | 125 +++++---- out.html | 264 ++++++++++++++++++ out.md | 264 ++++++++++++++++++ .../tomfran/lsm/tree/LSMTreeAddBenchmark.java | 2 +- .../tomfran/lsm/tree/LSMTreeGetBenchmark.java | 3 +- .../com/tomfran/lsm/utils/BenchmarkUtils.java | 3 +- src/main/java/com/tomfran/lsm/Main.java | 41 +-- .../java/com/tomfran/lsm/tree/LSMTree.java | 2 +- 8 files changed, 624 insertions(+), 80 deletions(-) create mode 100644 out.html create mode 100644 out.md diff --git a/README.md b/README.md index 2b481b8..f56701d 100644 --- a/README.md +++ b/README.md @@ -12,31 +12,33 @@ An implementation of the Log-Structured Merge Tree (LSM tree) data structure in 1. [SSTable](#sstable-1) 2. [Skip-List](#skip-list-1) 3. [Tree](#tree-1) -5. [Implementation status](#Implementation-status) +5. [Possible future improvements](#possible-improvements) +6. [References](#references) -## Console +### Console To interact with a toy tree you can use `./gradlew run -q` to spawn a console. ![console.png](misc%2Fconsole.png) ---- - # Architecture +Architecture overview, from SSTables, which are the disk-resident portion of the database, Skip Lists, used +as memory buffers, and finally to the combination of the twos to create insertion, lookup and deletion primitives. + ## SSTable Sorted String Table (SSTable) is a collection of files modelling key-value pairs in sorted order by key. It is used as a persistent storage for the LSM tree. -### Components +**Components** - _Data_: key-value pairs in sorted order by key, stored in a file; - _Sparse index_: sparse index containing key and offset of the corresponding key-value pair in the data; - _Bloom filter_: a [probabilistic data structure](https://en.wikipedia.org/wiki/Bloom_filter) used to test whether a key is in the SSTable. -### Key lookup +**Key lookup** The basic idea is to use the sparse index to find the key-value pair in the data file. The steps are: @@ -49,7 +51,7 @@ The steps are: The search is as lazy as possible, meaning that we read the minimum amount of data from disk, for instance, if the next key length is smaller than the one we are looking for, we can skip the whole key-value pair. -### Persistence +**Persistence** A table is persisted to disk when it is created. A base filename is defined, and three files are present: @@ -57,12 +59,12 @@ A table is persisted to disk when it is created. A base filename is defined, and - `.index`: index file; - `.bloom`: bloom filter file. -**Data format** +Data format: - `n`: number of key-value pairs; - ``: key-value pairs. -**Index format** +Index format: - `s`: number of entries in the whole table; - `n`: number of entries in the index; @@ -71,7 +73,7 @@ A table is persisted to disk when it is created. A base filename is defined, and - `s_1, s_2, ..., s_n`: remaining keys after a sparse index entry, used to exit from search; - ``: keys in the index. -**Filter format** +Filter format: - `m`: number of bits in the bloom filter; - `k`: number of hash functions; @@ -82,8 +84,6 @@ To save space, all integers are stored in [variable-length encoding](https://nlp.stanford.edu/IR-book/html/htmledition/variable-byte-codes-1.html), and offsets in the index are stored as [deltas](https://en.wikipedia.org/wiki/Delta_encoding). ---- - ## Skip-List A [skip-list](https://en.wikipedia.org/wiki/Skip_list) is a probabilistic data structure that allows fast search, @@ -92,7 +92,7 @@ insertion and deletion of elements in a sorted sequence. In the LSM tree, it is used as an in-memory data structure to store key-value pairs in sorted order by key. Once the skip-list reaches a certain size, it is flushed to disk as an SSTable. -### Operations details +**Operations details** The idea of a skip list is similar to a classic linked list. We have nodes with forward pointers, but also levels. We can think about a @@ -108,40 +108,57 @@ we are looking for. Then we move down to the next level and repeat the process u Insertions, deletions, and updates are done by first locating the element, then performing the operation on the node. All of them have an average time complexity of `O(log(n))`. ---- - ## Tree -... +Having defined SSTables and Skip Lists we can obtain the final structure as a combination of the two. +The main idea is to use the latter as an in-memory buffer, while the former efficiently stores flushed +buffers. + +**Insertion** -### Components +Each insert goes directly to a Memtable, which is a Skip List under the hood, so the response time is quite fast. +There exists a threshold, over which the mutable structure is made immutable by appending it to the _immmutable +memtables LIFO list_ and replaced with a new mutable list. -... +The immutable memtable list is asynchronously consumed by a background thread, which takes the next available +list and create a disk-resident SSTable with its content. -### Insertion +**Lookup** -... +While looking for a key, we proceed as follows: -### Lookup +1. Look into the in-memory buffer, if the key is recently written it is likely here, if not present continue; +2. Look into the immutable memtables list, iterating from the most recent to the oldest, if not present continue; +3. Look into disk tables, iterating from the most recent one to the oldest, if not present return null. -... +**Deletions** -### Write-ahead logging +To delete a key, we do not need to delete all its replicas, from the on-disk tables, we just need a special +value called _tombstone_. Hence a deletion is the same as an insertion, but with a value set to null. While looking for +a key, if we encounter a null value we simply return null as a result. -... +**SSTable Compaction** ---- +The most expensive operation while looking for a key is certainly the disk search, and this is why bloom filters are +crucial for negative +lookup on SSTables. But no bloom filter can save us if too many tables are available to search, hence we need +_compaction_. + +When flushing a Memtable, we create an SSTable of level one. When the first level reaches a certain threshold, +all its tables are merged into a level-two table, and so on. This permits us to save storage and query fewer +tables in lookups. + +Note that this style of compaction is not standard, there are various sophisticated techniques, but for the sake of +this project this simple level-like compaction works wonders. # Benchmarks I am using [JMH](https://openjdk.java.net/projects/code-tools/jmh/) to run benchmarks, the results are obtained on AMD Ryzen™ 5 4600H with 16GB of RAM and 512GB SSD. -> Take those with a grain of salt, development is still in progress. - To run them use `./gradlew jmh`. -## SSTable +**SSTable** - Negative access: the key is not present in the table, hence the Bloom filter will likely stop the search; - Random access: the key is present in the table, the order of the keys is random. @@ -154,7 +171,7 @@ c.t.l.sstable.SSTableBenchmark.randomAccess thrpt 5 7989.945 ± 40 ``` -## Bloom filter +**Bloom filter** - Add: add keys to a 1M keys Bloom filter with 0.01 false positive rate; - Contains: test whether the keys are present in the Bloom filter. @@ -166,7 +183,7 @@ c.t.l.bloom.BloomFilterBenchmark.contains thrpt 5 3567392.634 ± 220377 ``` -## Skip-List +**Skip-List** - Get: get keys from a 100k keys skip-list; - Add/Remove: add and remove keys from a 100k keys skip-list. @@ -179,7 +196,7 @@ c.t.l.memtable.SkipListBenchmark.get thrpt 5 487265.620 ± 8201 ``` -## Tree +**Tree** - Get: get elements from a tree with 1M keys; - Add: add 1M distinct elements to a tree with a memtable size of 2^18 @@ -191,28 +208,24 @@ c.t.l.tree.LSMTreeGetBenchmark.get thrpt 5 9426.951 ± 241 ``` ---- - -## Implementation status - -- [x] SSTable - - [x] Init - - [x] Read - - [x] Compaction - - [x] Ints compression - - [x] Bloom filter - - [x] Indexes persistence - - [x] File initialization -- [x] Skip-List - - [x] Operations - - [x] Iterator -- [x] Tree - - [x] Operations - - [x] Background flush - - [x] Background compaction - - [ ] Write ahead log -- [x] Benchmarks - - [x] SSTable - - [x] Bloom filter - - [x] Skip-List - - [x] Tree +## Possible improvements + +There is certainly space for improvement on this project: + +1. Blocked bloom filters: its a variant of a classic array-like bloom filter which is more cache efficient; +2. Search fingers in the Skip list: the idea is to keep a pointer to the last search, and start from there with + subsequent queries; +3. Proper level compaction in the LSM tree; +4. Write ahead log for the insertions, without this, a crash makes all the in-memory writes disappear; +5. Proper recovery: handle crashes and reboots, using existing SSTables and the write-ahead log. + +I don't have the practical time to do all of this, perhaps the first two points will be handled in the future. + +## References + +- [Database Internals](https://www.databass.dev/) by Alex Petrov, specifically chapters about Log-Structured Storage and + File Formats; +- [A Skip List Cookbook](https://api.drum.lib.umd.edu/server/api/core/bitstreams/17176ef8-8330-4a6c-8b75-4cd18c570bec/content) + by William Pugh. + +_If you found this useful or interesting do not hesitate to ask clarifying questions or get in touch!_ diff --git a/out.html b/out.html new file mode 100644 index 0000000..9c6e7c0 --- /dev/null +++ b/out.html @@ -0,0 +1,264 @@ +

An implementation of the Log-Structured Merge Tree (LSM tree) data +structure in Java.

+

Table of Contents

+
    +
  1. Architecture +
      +
    1. SSTable
      2. +
    2. Skip-List
      3. +
    3. Tree
      4. +
    2. +
  2. Benchmarks +
      +
    1. SSTable
      2. +
    2. Skip-List
      3. +
    3. Tree
      4. +
    3. +
  3. Possible future +improvements
    4. +
  4. References
    5. +
+

To interact with a toy tree you can use ./gradlew run -q +to spawn a console.

+
+console.png + +
+
+

1 Architecture

+

Architecture overview, from SSTables, which are the disk-resident +portion of the database, Skip Lists, used as memory buffers, and finally +to the combination of the twos to create insertion, lookup and deletion +primitives.

+

1.1 SSTable

+

Sorted String Table (SSTable) is a collection of files modelling +key-value pairs in sorted order by key. It is used as a persistent +storage for the LSM tree.

+

1.1.1 Components

+
    +
  • Data: key-value pairs in sorted order by key, stored in a +file;
    • +
  • Sparse index: sparse index containing key and offset of the +corresponding key-value pair in the data;
    • +
  • Bloom filter: a probabilistic data +structure used to test whether a key is in the SSTable.
    • +
+

1.1.2 Key lookup

+

The basic idea is to use the sparse index to find the key-value pair +in the data file. The steps are:

+
    +
  1. Use the Bloom filter to test whether the key might be in the +table;
    2. +
  2. If the key might be present, use binary search on the index to find +the maximum lower bound of the key;
    3. +
  3. Scan the data from the position found in the previous step to find +the key-value pair. The search can stop when we are seeing a key greater +than the one we are looking for, or when we reach the end of the +table.
    4. +
+

The search is as lazy as possible, meaning that we read the minimum +amount of data from disk, for instance, if the next key length is +smaller than the one we are looking for, we can skip the whole key-value +pair.

+

1.1.3 Persistence

+

A table is persisted to disk when it is created. A base filename is +defined, and three files are present:

+
    +
  • <base_filename>.data: data file;
    • +
  • <base_filename>.index: index file;
    • +
  • <base_filename>.bloom: bloom filter file.
    • +
+

Data format

+
    +
  • n: number of key-value pairs;
    • +
  • <key_len_1, value_len_1, key_1, value_1, ... key_n, value_n>: +key-value pairs.
    • +
+

Index format

+
    +
  • s: number of entries in the whole table;
    • +
  • n: number of entries in the index;
    • +
  • o_1, o_2 - o_1, ..., o_n - o_n-1: offsets of the +key-value pairs in the data file, skipping the first one;
    • +
  • s_1, s_2, ..., s_n: remaining keys after a sparse index +entry, used to exit from search;
    • +
  • <key_len_1, key_1, ... key_len_n, key_n>: keys in +the index.
    • +
+

Filter format

+
    +
  • m: number of bits in the bloom filter;
    • +
  • k: number of hash functions;
    • +
  • n: size of underlying long array;
    • +
  • b_1, b_2, ..., b_n: bits of the bloom filter.
    • +
+

To save space, all integers are stored in variable-length +encoding, and offsets in the index are stored as deltas.

+
+

1.2 Skip-List

+

A skip-list is +a probabilistic data structure that allows fast search, insertion and +deletion of elements in a sorted sequence.

+

In the LSM tree, it is used as an in-memory data structure to store +key-value pairs in sorted order by key. Once the skip-list reaches a +certain size, it is flushed to disk as an SSTable.

+

1.2.1 Operations details

+

The idea of a skip list is similar to a classic linked list. We have +nodes with forward pointers, but also levels. We can think about a level +as a fast lane between nodes. By carefully constructing them at +insertion time, searches are faster, as they can use higher levels to +skip unwanted nodes.

+

Given n elements, a skip list has log(n) +levels, the first level containing all the elements. By increasing the +level, the number of elements is cut roughly by half.

+

To locate an element, we start from the top level and move forward +until we find an element greater than the one we are looking for. Then +we move down to the next level and repeat the process until we find the +element.

+

Insertions, deletions, and updates are done by first locating the +element, then performing the operation on the node. All of them have an +average time complexity of O(log(n)).

+
+

1.3 Tree

+

Having defined SSTables and Skip Lists we can obtain the final +structure as a combination of the two. The main idea is to use the +latter as an in-memory buffer, while the former efficiently stores +flushed buffers.

+

1.3.1 Insertion

+

Each insert goes directly to a Memtable, which is a Skip List under +the hood, so the response time is quite fast. There exists a threshold, +over which the mutable structure is made immutable by appending it to +the immmutable memtables LIFO list and replaced with a new +mutable list.

+

The immutable memtable list is asynchronously consumed by a +background thread, which takes the next available list and create a +disk-resident SSTable with its content.

+

1.3.2 Lookup

+

While looking for a key, we proceed as follows:

+
    +
  1. Look into the in-memory buffer, if the key is recently written it is +likely here, if not present continue;
    2. +
  2. Look into the immutable memtables list, iterating from the most +recent to the oldest, if not present continue;
    3. +
  3. Look into disk tables, iterating from the most recent one to the +oldest, if not present return null.
    4. +
+

1.3.3 Deletions

+

To delete a key, we do not need to delete all its replicas, from the +on-disk tables, we just need a special value called tombstone. +Hence a deletion is the same as an insertion, but with a value set to +null. While looking for a key, if we encounter a null value we simply +return null as a result.

+

1.3.4 SSTable Compaction

+

The most expensive operation while looking for a key is certainly the +disk search, and this is why bloom filters are crucial for negative +lookup on SSTables. But no bloom filter can save us if too many tables +are available to search, hence we need compaction.

+

When flushing a Memtable, we create an SSTable of level one. When the +first level reaches a certain threshold, all its tables are merged into +a level-two table, and so on. This permits us to save storage and query +fewer tables in lookups.

+

Note that this style of compaction is not standard, there are various +sophisticated techniques, but for the sake of this project this simple +level-like compaction works wonders.

+
+

2 Benchmarks

+

I am using JMH to run +benchmarks, the results are obtained on AMD Ryzen™ 5 4600H with 16GB of +RAM and 512GB SSD.

+

To run them use ./gradlew jmh.

+

2.1 SSTable

+
    +
  • Negative access: the key is not present in the table, hence the +Bloom filter will likely stop the search;
    • +
  • Random access: the key is present in the table, the order of the +keys is random.
    • +
+

+Benchmark                                       Mode  Cnt        Score        Error  Units
+c.t.l.sstable.SSTableBenchmark.negativeAccess  thrpt    5  3316202.976 ±  32851.546  ops/s
+c.t.l.sstable.SSTableBenchmark.randomAccess    thrpt    5     7989.945 ±     40.689  ops/s
+
+

2.2 Bloom filter

+
    +
  • Add: add keys to a 1M keys Bloom filter with 0.01 false positive +rate;
    • +
  • Contains: test whether the keys are present in the Bloom +filter.
    • +
+
Benchmark                                       Mode  Cnt        Score        Error  Units
+c.t.l.bloom.BloomFilterBenchmark.add           thrpt    5  3190753.307 ±  74744.764  ops/s
+c.t.l.bloom.BloomFilterBenchmark.contains      thrpt    5  3567392.634 ± 220377.613  ops/s
+
+

2.3 Skip-List

+
    +
  • Get: get keys from a 100k keys skip-list;
    • +
  • Add/Remove: add and remove keys from a 100k keys skip-list.
    • +
+

+Benchmark                                       Mode  Cnt        Score        Error  Units
+c.t.l.memtable.SkipListBenchmark.addRemove     thrpt    5   430239.471 ±   4825.990  ops/s
+c.t.l.memtable.SkipListBenchmark.get           thrpt    5   487265.620 ±   8201.227  ops/s
+
+

2.4 Tree

+
    +
  • Get: get elements from a tree with 1M keys;
    • +
  • Add: add 1M distinct elements to a tree with a memtable size of +2^18
    • +
+
Benchmark                                       Mode  Cnt        Score        Error  Units
+c.t.l.tree.LSMTreeAddBenchmark.add             thrpt    5   540788.751 ±  54491.134  ops/s
+c.t.l.tree.LSMTreeGetBenchmark.get             thrpt    5     9426.951 ±    241.190  ops/s
+
+
+

2.5 Possible improvements

+

There is certainly space for improvement on this project:

+
    +
  1. Blocked bloom filters: its a variant of a classic array-like bloom +filter which is more cache efficient;
    2. +
  2. Search fingers in the Skip list: the idea is to keep a pointer to +the last search, and start from there with subsequent queries;
    3. +
  3. Proper level compaction in the LSM tree;
    4. +
  4. Write ahead log for the insertions, without this, a crash makes all +the in-memory writes disappear;
    5. +
  5. Proper recovery: handle crashes and reboots, using existing SSTables +and the write-ahead log.
    6. +
+

I don’t have the practical time to do all of this, perhaps the first +two points will be handled in the future.

+
+

2.6 References

+ +
+

If you found this useful or interesting do not hesitate to ask +clarifying questions or get in touch!

diff --git a/out.md b/out.md new file mode 100644 index 0000000..9c6e7c0 --- /dev/null +++ b/out.md @@ -0,0 +1,264 @@ +

An implementation of the Log-Structured Merge Tree (LSM tree) data +structure in Java.

+

Table of Contents

+
    +
  1. Architecture +
      +
    1. SSTable
      2. +
    2. Skip-List
      3. +
    3. Tree
      4. +
    2. +
  2. Benchmarks +
      +
    1. SSTable
      2. +
    2. Skip-List
      3. +
    3. Tree
      4. +
    3. +
  3. Possible future +improvements
    4. +
  4. References
    5. +
+

To interact with a toy tree you can use ./gradlew run -q +to spawn a console.

+
+console.png + +
+
+

1 Architecture

+

Architecture overview, from SSTables, which are the disk-resident +portion of the database, Skip Lists, used as memory buffers, and finally +to the combination of the twos to create insertion, lookup and deletion +primitives.

+

1.1 SSTable

+

Sorted String Table (SSTable) is a collection of files modelling +key-value pairs in sorted order by key. It is used as a persistent +storage for the LSM tree.

+

1.1.1 Components

+
    +
  • Data: key-value pairs in sorted order by key, stored in a +file;
    • +
  • Sparse index: sparse index containing key and offset of the +corresponding key-value pair in the data;
    • +
  • Bloom filter: a probabilistic data +structure used to test whether a key is in the SSTable.
    • +
+

1.1.2 Key lookup

+

The basic idea is to use the sparse index to find the key-value pair +in the data file. The steps are:

+
    +
  1. Use the Bloom filter to test whether the key might be in the +table;
    2. +
  2. If the key might be present, use binary search on the index to find +the maximum lower bound of the key;
    3. +
  3. Scan the data from the position found in the previous step to find +the key-value pair. The search can stop when we are seeing a key greater +than the one we are looking for, or when we reach the end of the +table.
    4. +
+

The search is as lazy as possible, meaning that we read the minimum +amount of data from disk, for instance, if the next key length is +smaller than the one we are looking for, we can skip the whole key-value +pair.

+

1.1.3 Persistence

+

A table is persisted to disk when it is created. A base filename is +defined, and three files are present:

+
    +
  • <base_filename>.data: data file;
    • +
  • <base_filename>.index: index file;
    • +
  • <base_filename>.bloom: bloom filter file.
    • +
+

Data format

+
    +
  • n: number of key-value pairs;
    • +
  • <key_len_1, value_len_1, key_1, value_1, ... key_n, value_n>: +key-value pairs.
    • +
+

Index format

+
    +
  • s: number of entries in the whole table;
    • +
  • n: number of entries in the index;
    • +
  • o_1, o_2 - o_1, ..., o_n - o_n-1: offsets of the +key-value pairs in the data file, skipping the first one;
    • +
  • s_1, s_2, ..., s_n: remaining keys after a sparse index +entry, used to exit from search;
    • +
  • <key_len_1, key_1, ... key_len_n, key_n>: keys in +the index.
    • +
+

Filter format

+
    +
  • m: number of bits in the bloom filter;
    • +
  • k: number of hash functions;
    • +
  • n: size of underlying long array;
    • +
  • b_1, b_2, ..., b_n: bits of the bloom filter.
    • +
+

To save space, all integers are stored in variable-length +encoding, and offsets in the index are stored as deltas.

+
+

1.2 Skip-List

+

A skip-list is +a probabilistic data structure that allows fast search, insertion and +deletion of elements in a sorted sequence.

+

In the LSM tree, it is used as an in-memory data structure to store +key-value pairs in sorted order by key. Once the skip-list reaches a +certain size, it is flushed to disk as an SSTable.

+

1.2.1 Operations details

+

The idea of a skip list is similar to a classic linked list. We have +nodes with forward pointers, but also levels. We can think about a level +as a fast lane between nodes. By carefully constructing them at +insertion time, searches are faster, as they can use higher levels to +skip unwanted nodes.

+

Given n elements, a skip list has log(n) +levels, the first level containing all the elements. By increasing the +level, the number of elements is cut roughly by half.

+

To locate an element, we start from the top level and move forward +until we find an element greater than the one we are looking for. Then +we move down to the next level and repeat the process until we find the +element.

+

Insertions, deletions, and updates are done by first locating the +element, then performing the operation on the node. All of them have an +average time complexity of O(log(n)).

+
+

1.3 Tree

+

Having defined SSTables and Skip Lists we can obtain the final +structure as a combination of the two. The main idea is to use the +latter as an in-memory buffer, while the former efficiently stores +flushed buffers.

+

1.3.1 Insertion

+

Each insert goes directly to a Memtable, which is a Skip List under +the hood, so the response time is quite fast. There exists a threshold, +over which the mutable structure is made immutable by appending it to +the immmutable memtables LIFO list and replaced with a new +mutable list.

+

The immutable memtable list is asynchronously consumed by a +background thread, which takes the next available list and create a +disk-resident SSTable with its content.

+

1.3.2 Lookup

+

While looking for a key, we proceed as follows:

+
    +
  1. Look into the in-memory buffer, if the key is recently written it is +likely here, if not present continue;
    2. +
  2. Look into the immutable memtables list, iterating from the most +recent to the oldest, if not present continue;
    3. +
  3. Look into disk tables, iterating from the most recent one to the +oldest, if not present return null.
    4. +
+

1.3.3 Deletions

+

To delete a key, we do not need to delete all its replicas, from the +on-disk tables, we just need a special value called tombstone. +Hence a deletion is the same as an insertion, but with a value set to +null. While looking for a key, if we encounter a null value we simply +return null as a result.

+

1.3.4 SSTable Compaction

+

The most expensive operation while looking for a key is certainly the +disk search, and this is why bloom filters are crucial for negative +lookup on SSTables. But no bloom filter can save us if too many tables +are available to search, hence we need compaction.

+

When flushing a Memtable, we create an SSTable of level one. When the +first level reaches a certain threshold, all its tables are merged into +a level-two table, and so on. This permits us to save storage and query +fewer tables in lookups.

+

Note that this style of compaction is not standard, there are various +sophisticated techniques, but for the sake of this project this simple +level-like compaction works wonders.

+
+

2 Benchmarks

+

I am using JMH to run +benchmarks, the results are obtained on AMD Ryzen™ 5 4600H with 16GB of +RAM and 512GB SSD.

+

To run them use ./gradlew jmh.

+

2.1 SSTable

+
    +
  • Negative access: the key is not present in the table, hence the +Bloom filter will likely stop the search;
    • +
  • Random access: the key is present in the table, the order of the +keys is random.
    • +
+

+Benchmark                                       Mode  Cnt        Score        Error  Units
+c.t.l.sstable.SSTableBenchmark.negativeAccess  thrpt    5  3316202.976 ±  32851.546  ops/s
+c.t.l.sstable.SSTableBenchmark.randomAccess    thrpt    5     7989.945 ±     40.689  ops/s
+
+

2.2 Bloom filter

+
    +
  • Add: add keys to a 1M keys Bloom filter with 0.01 false positive +rate;
    • +
  • Contains: test whether the keys are present in the Bloom +filter.
    • +
+
Benchmark                                       Mode  Cnt        Score        Error  Units
+c.t.l.bloom.BloomFilterBenchmark.add           thrpt    5  3190753.307 ±  74744.764  ops/s
+c.t.l.bloom.BloomFilterBenchmark.contains      thrpt    5  3567392.634 ± 220377.613  ops/s
+
+

2.3 Skip-List

+
    +
  • Get: get keys from a 100k keys skip-list;
    • +
  • Add/Remove: add and remove keys from a 100k keys skip-list.
    • +
+

+Benchmark                                       Mode  Cnt        Score        Error  Units
+c.t.l.memtable.SkipListBenchmark.addRemove     thrpt    5   430239.471 ±   4825.990  ops/s
+c.t.l.memtable.SkipListBenchmark.get           thrpt    5   487265.620 ±   8201.227  ops/s
+
+

2.4 Tree

+
    +
  • Get: get elements from a tree with 1M keys;
    • +
  • Add: add 1M distinct elements to a tree with a memtable size of +2^18
    • +
+
Benchmark                                       Mode  Cnt        Score        Error  Units
+c.t.l.tree.LSMTreeAddBenchmark.add             thrpt    5   540788.751 ±  54491.134  ops/s
+c.t.l.tree.LSMTreeGetBenchmark.get             thrpt    5     9426.951 ±    241.190  ops/s
+
+
+

2.5 Possible improvements

+

There is certainly space for improvement on this project:

+
    +
  1. Blocked bloom filters: its a variant of a classic array-like bloom +filter which is more cache efficient;
    2. +
  2. Search fingers in the Skip list: the idea is to keep a pointer to +the last search, and start from there with subsequent queries;
    3. +
  3. Proper level compaction in the LSM tree;
    4. +
  4. Write ahead log for the insertions, without this, a crash makes all +the in-memory writes disappear;
    5. +
  5. Proper recovery: handle crashes and reboots, using existing SSTables +and the write-ahead log.
    6. +
+

I don’t have the practical time to do all of this, perhaps the first +two points will be handled in the future.

+
+

2.6 References

+ +
+

If you found this useful or interesting do not hesitate to ask +clarifying questions or get in touch!

diff --git a/src/jmh/java/com/tomfran/lsm/tree/LSMTreeAddBenchmark.java b/src/jmh/java/com/tomfran/lsm/tree/LSMTreeAddBenchmark.java index ba4b08e..3b94187 100644 --- a/src/jmh/java/com/tomfran/lsm/tree/LSMTreeAddBenchmark.java +++ b/src/jmh/java/com/tomfran/lsm/tree/LSMTreeAddBenchmark.java @@ -28,7 +28,7 @@ public void setup() throws IOException { } @TearDown - public void teardown() throws IOException, InterruptedException { + public void teardown() throws InterruptedException { tree.stop(); Thread.sleep(5000); BenchmarkUtils.deleteDir(DIR); diff --git a/src/jmh/java/com/tomfran/lsm/tree/LSMTreeGetBenchmark.java b/src/jmh/java/com/tomfran/lsm/tree/LSMTreeGetBenchmark.java index 53639e0..3443f10 100644 --- a/src/jmh/java/com/tomfran/lsm/tree/LSMTreeGetBenchmark.java +++ b/src/jmh/java/com/tomfran/lsm/tree/LSMTreeGetBenchmark.java @@ -35,8 +35,9 @@ public void setup() throws IOException { } @TearDown - public void teardown() throws IOException { + public void teardown() throws InterruptedException { tree.stop(); + Thread.sleep(5000); BenchmarkUtils.deleteDir(DIR); } diff --git a/src/jmh/java/com/tomfran/lsm/utils/BenchmarkUtils.java b/src/jmh/java/com/tomfran/lsm/utils/BenchmarkUtils.java index 4de9e23..7806492 100644 --- a/src/jmh/java/com/tomfran/lsm/utils/BenchmarkUtils.java +++ b/src/jmh/java/com/tomfran/lsm/utils/BenchmarkUtils.java @@ -4,7 +4,6 @@ import com.tomfran.lsm.types.ByteArrayPair; import java.io.File; -import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; import java.util.Random; @@ -13,7 +12,7 @@ public class BenchmarkUtils { - public static LSMTree initTree(Path dir, int memSize, int levelSize) throws IOException { + public static LSMTree initTree(Path dir, int memSize, int levelSize) { // setup directory if (Files.exists(dir)) deleteDir(dir); diff --git a/src/main/java/com/tomfran/lsm/Main.java b/src/main/java/com/tomfran/lsm/Main.java index b7a7048..a710c71 100644 --- a/src/main/java/com/tomfran/lsm/Main.java +++ b/src/main/java/com/tomfran/lsm/Main.java @@ -13,7 +13,7 @@ public class Main { static final String DIRECTORY = "LSM-data"; - public static void main(String[] args) { + public static void main(String[] args) throws InterruptedException { if (new File(DIRECTORY).exists()) deleteDir(); @@ -48,25 +48,28 @@ public static void main(String[] args) { System.out.print("> "); String command = scanner.nextLine(); - String[] parts = command.split(" "); - - String msg; - switch (parts[0]) { - case "s", "set" -> { - tree.add(new ByteArrayPair(parts[1].getBytes(), parts[2].getBytes())); - System.out.println("ok"); - } - case "d", "del" -> { - tree.delete(parts[1].getBytes()); - System.out.println("ok"); - } - case "g", "get" -> { - byte[] value = tree.get(parts[1].getBytes()); - System.out.println((value == null || value.length == 0) ? "not found" : new String(value)); + try { + String[] parts = command.split(" "); + + switch (parts[0]) { + case "s", "set" -> { + tree.add(new ByteArrayPair(parts[1].getBytes(), parts[2].getBytes())); + System.out.println("ok"); + } + case "d", "del" -> { + tree.delete(parts[1].getBytes()); + System.out.println("ok"); + } + case "g", "get" -> { + byte[] value = tree.get(parts[1].getBytes()); + System.out.println((value == null || value.length == 0) ? "not found" : new String(value)); + } + case "h", "help" -> System.out.println(help); + case "e", "exit" -> exit = true; + default -> System.out.println("Unknown command"); } - case "h", "help" -> System.out.println(help); - case "e", "exit" -> exit = true; - default -> System.out.println("Unknown command"); + } catch (Exception e) { + System.out.printf("### error while executing command: \"%s\"\n", command); } } tree.stop(); diff --git a/src/main/java/com/tomfran/lsm/tree/LSMTree.java b/src/main/java/com/tomfran/lsm/tree/LSMTree.java index 6382cfc..82e9ca1 100644 --- a/src/main/java/com/tomfran/lsm/tree/LSMTree.java +++ b/src/main/java/com/tomfran/lsm/tree/LSMTree.java @@ -132,7 +132,7 @@ public byte[] get(byte[] key) { /** * Stop the background threads. */ - public void stop() { + public void stop() throws InterruptedException { memtableFlusher.shutdownNow(); tableCompactor.shutdownNow(); }