Skip to content

What's new in URLFrontier 2.3
Compare
Choose a tag to compare
@jnioche jnioche released this 24 Oct 11:15
· 83 commits to master since this release
2.3
14dbdb2

What's Changed

  • Use multiple threads for putting URLs by @jnioche in #65
  • Multithread reading from queues by @jnioche in #66
  • Bugfix Exception when trying to delete a non existent crawl #70
  • Dependency updates #68
  • Bugfix ShardedRocksDBService does not return ack when identical URLs are sent in short succession #67
  • Batch write operations #64

Multithreading the writes gives an gain of up to 54% while querying the queues in parallel improves the performance by up to 8x.

Full Changelog: 2.2...2.3

Contributors

jnioche
Assets 5
Loading