Skip to content

F
folly
Commit ede7e6eb authored by Marc Celani's avatar Marc Celani Committed by Facebook Github Bot
Browse files
Options

shrink_to_fit TDigest after merge 

Summary:
When used in a QuantileEstimator, some sliding window buckets can be sparse or all together empty. In this case, reserving memory in the constructor and in merge is wasteful. Don't reserve in the constructor (this is a complete waste) and call shrink_to_fit in merge.

It's worth noting that the benchmarks *are* shrinking the digest, because the digests are only 85 or so items big, so it is exercising this path.

Reviewed By: yfeldblum

Differential Revision: D7855242

fbshipit-source-id: 61bcbaaf17dd46f8b98d171af43516ceb8d84510
parent 15c7ad65
Hide whitespace changes
Inline Side-by-side
Showing with 27 additions and 27 deletions
folly/stats/TDigest.cpp
View file @ ede7e6eb
......@@ -148,6 +148,7 @@ TDigest TDigest::merge(Range<const double*> sortedValues) const {
}
result.sum_ += cur.add(sumsToMerge, weightsToMerge);
compressed.push_back(cur);
compressed.shrink_to_fit();
result.centroids_ = std::move(compressed);
return result;
}
......@@ -244,6 +245,7 @@ TDigest TDigest::merge(Range<const TDigest*> digests) {
}
result.sum_ += cur.add(sumsToMerge, weightsToMerge);
compressed.push_back(cur);
compressed.shrink_to_fit();
result.count_ = count;
result.min_ = min;
......
folly/stats/TDigest.h
View file @ ede7e6eb
......@@ -49,9 +49,7 @@ namespace folly {
class TDigest {
public:
explicit TDigest(size_t maxSize = 100)
: maxSize_(maxSize), sum_(0.0), count_(0.0), max_(NAN), min_(NAN) {
centroids_.reserve(maxSize);
}
: maxSize_(maxSize), sum_(0.0), count_(0.0), max_(NAN), min_(NAN) {}
/*
* Returns a new TDigest constructed with values merged from the current
......
folly/stats/test/TDigestBenchmark.cpp
View file @ ede7e6eb
......@@ -138,33 +138,33 @@ BENCHMARK_RELATIVE_NAMED_PARAM(estimateQuantile, 1000_p999, 1000, 0.999)
* ============================================================================
* folly/stats/test/TDigestBenchmark.cpp relative time/iter iters/s
* ============================================================================
* merge(100x1) 2.22us 451.08K
* merge(100x5) 59.59% 3.72us 268.81K
* merge(100x10) 43.70% 5.07us 197.14K
* merge(1000x1) 10.63% 20.86us 47.93K
* merge(1000x5) 6.60% 33.60us 29.76K
* merge(1000x10) 4.52% 49.10us 20.37K
* merge(100x1) 2.23us 449.36K
* merge(100x5) 59.15% 3.76us 265.78K
* merge(100x10) 41.72% 5.33us 187.46K
* merge(1000x1) 10.18% 21.86us 45.75K
* merge(1000x5) 6.34% 35.11us 28.48K
* merge(1000x10) 4.45% 50.01us 19.99K
* ----------------------------------------------------------------------------
* mergeDigests(100x10) 25.30us 39.53K
* mergeDigests(100x30) 21.70% 116.58us 8.58K
* mergeDigests(100x60) 9.22% 274.53us 3.64K
* mergeDigests(1000x60) 0.90% 2.81ms 356.13
* mergeDigests(100x10) 24.65us 40.57K
* mergeDigests(100x30) 21.03% 117.21us 8.53K
* mergeDigests(100x60) 8.92% 276.47us 3.62K
* mergeDigests(1000x60) 0.88% 2.80ms 357.15
* ----------------------------------------------------------------------------
* estimateQuantile(100x1_p001) 9.50ns 105.28M
* estimateQuantile(100_p01) 63.49% 14.96ns 66.85M
* estimateQuantile(100_p25) 14.89% 63.81ns 15.67M
* estimateQuantile(100_p50) 11.56% 82.15ns 12.17M
* estimateQuantile(100_p75) 15.44% 61.53ns 16.25M
* estimateQuantile(100_p99) 74.77% 12.70ns 78.72M
* estimateQuantile(100_p999) 117.16% 8.11ns 123.34M
* estimateQuantile(100x1_p001) 9.40ns 106.40M
* estimateQuantile(100_p01) 63.42% 14.82ns 67.49M
* estimateQuantile(100_p25) 14.81% 63.47ns 15.75M
* estimateQuantile(100_p50) 11.26% 83.47ns 11.98M
* estimateQuantile(100_p75) 15.22% 61.76ns 16.19M
* estimateQuantile(100_p99) 76.04% 12.36ns 80.91M
* estimateQuantile(100_p999) 115.85% 8.11ns 123.27M
* ----------------------------------------------------------------------------
* estimateQuantile(1000_p001) 27.10% 35.05ns 28.53M
* estimateQuantile(1000_p01) 8.58% 110.64ns 9.04M
* estimateQuantile(1000_p25) 1.94% 488.67ns 2.05M
* estimateQuantile(1000_p50) 1.39% 684.81ns 1.46M
* estimateQuantile(1000_p75) 1.96% 483.88ns 2.07M
* estimateQuantile(1000_p99) 8.99% 105.70ns 9.46M
* estimateQuantile(1000_p999) 36.87% 25.76ns 38.82M
* estimateQuantile(1000_p001) 27.57% 34.08ns 29.34M
* estimateQuantile(1000_p01) 8.53% 110.24ns 9.07M
* estimateQuantile(1000_p25) 1.92% 488.24ns 2.05M
* estimateQuantile(1000_p50) 1.37% 684.40ns 1.46M
* estimateQuantile(1000_p75) 1.94% 485.23ns 2.06M
* estimateQuantile(1000_p99) 8.87% 105.90ns 9.44M
* estimateQuantile(1000_p999) 36.64% 25.65ns 38.99M
* ============================================================================
*/
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment