Struct ReusableDeduplicator 

pub struct ReusableDeduplicator<T> { /* private fields */ }

A helper for deduplicating items in an iterator while amortizing the storage cost of deduplication.

When deduplicating items in an iterator, a set of which items have been seen must be kept. This involves underlying storage that grows as more items are seen for the first time. When many iterators are being deduplicated, this storage cost can add up in terms of the number of underlying allocations, and resizing operations of the set.

ReusableDeduplicator allows for reuse of the underlying set between iterations.

Safety

In order to support reusing the underlying storage between iterations, without having to clone the items, we directly hash items and track the presence of their hash value. This is a subtle distinction compared to using the items themselves, as it means that we lose the typical fallback of comparing items for equality if there is a hash collision detected. Consistent with other code in this repository that deals with things like metric tags, we use a high-quality hash function and make a calculated bet that collisions are extremely unlikely.

Implementations

impl<T: Eq + Hash> ReusableDeduplicator<T>

pub fn new() -> Self

Create a new ReusableDeduplicator.

pub fn deduplicated<'item, I>( &mut self, iter: I, ) -> Deduplicated<'_, 'item, I, T>

where I: Iterator<Item = &'item T>,

Creates a wrapper iterator over the given iterator that deduplicates the items.

Trait Implementations

impl<T: Debug> Debug for ReusableDeduplicator<T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.