v1.11.0

epi052/feroxbusterv1.11.0Dec 27, 2020by epi052

AI Summary

Implements a similarity filter using the SSDeep algorithm to exclude pages that match a provided example page.

Key Highlights

Added `--filter-similar-to` option
Uses SSDeep algorithm for hashing response bodies
Filters pages with 95% similarity

New Features

Similarity filtering using SSDeep algorithm

Full Release Notes

- added the ability to specify an example page for filtering pages that are similar to the given example
  - `--filter-similar-to` requests the page passed to it via CLI, after which it hashes 
the response body using the [SSDeep algorithm](https://ssdeep-project.github.io/ssdeep/index.html).  All subsequent 
pages are hashed and compared to the original request's hash. If the comparison of the two hashes meets a certain 
percentage of similarity (currently 95%), then that request will be filtered out.
  - SSDeep was selected as it does a good job of identifying near-duplicate pages once content-length reaches a certain 
size, while remaining performant.  Other algorithms were tested but resulted in huge performance hits (orders of 
magnitude slower on requests/second).