Skip to content

F
folly
Commit 506b9a32 authored by Tom Jackson's avatar Tom Jackson Committed by Jordan DeLong
Browse files
Options

folly::gen, or Comprehensions->Folly 

Summary:
Moving Comprehensions library into Folly and refactoring its interface to be much more modular and composable. There are now two core abstractions:

# Generators: Standalone classes supporting ##apply()## and optionally ##foreach()##. These all inherit from ##GenImpl<T, Self>##.
# Operators: Standalone classes which, when composed with a generator, produce a new generator. These all inherit from ##OperatorImpl<Self>##.

These generators may be composed with ##operator|## overloads which only match ##const GenImpl<T, Self>&## on the left like ##gen | op##.  Additionally, generator may be consumed inline with ##gen | lambda## like ##gen | [](int x) { cout << x << endl; };##.

With this design, new operators may be added very simply without modifying the core library and templates are instantiated only exactly as needed.

Example:

```lang=cpp
auto sum = seq(1, 10) | filter(isPrime) | sum;
seq(1, 10) | [](int i) {
cout << i << endl;
};
```

Test Plan: Unit tests

Reviewed By: andrei.alexandrescu@fb.com

FB internal diff: D542215
parent 99d9bfdd
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 2518 additions and 0 deletions
folly/experimental/Gen-inl.h 0 → 100644
View file @ 506b9a32
This diff is collapsed.
folly/experimental/Gen.h 0 → 100644
View file @ 506b9a32
/*
* Copyright 2012 Facebook, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#pragma once
#include <functional>
#include <memory>
#include <type_traits>
#include <utility>
#include <algorithm>
#include "folly/Range.h"
#include "folly/Optional.h"
/**
* Generator-based Sequence Comprehensions in C++, akin to C#'s LINQ
* @author Tom Jackson <tjackson@fb.com>
*
* This library makes it possible to write declarative comprehensions for
* processing sequences of values efficiently in C++. The operators should be
* familiar to those with experience in functional programming, and the
* performance will be virtually identical to the equivalent, boilerplate C++
* implementations.
*
* Generator objects may be created from either an stl-like container (anything
* supporting begin() and end()), from sequences of values, or from another
* generator (see below). To create a generator that pulls values from a vector,
* for example, one could write:
*
* vector<string> names { "Jack", "Jill", "Sara", "Tom" };
* auto gen = from(names);
*
* Generators are composed by building new generators out of old ones through
* the use of operators. These are reminicent of shell pipelines, and afford
* similar composition. Lambda functions are used liberally to describe how to
* handle individual values:
*
* auto lengths = gen
* | mapped([](const fbstring& name) { return name.size(); });
*
* Generators are lazy; they don't actually perform any work until they need to.
* As an example, the 'lengths' generator (above) won't actually invoke the
* provided lambda until values are needed:
*
* auto lengthVector = lengths | asVector();
* auto totalLength = lengths | sum;
*
* 'auto' is useful in here because the actual types of the generators objects
* are usually complicated and implementation-sensitive.
*
* If a simpler type is desired (for returning, as an example), VirtualGen<T>
* may be used to wrap the generator in a polymorphic wrapper:
*
* VirtualGen<float> powersOfE() {
* return seq(1) | mapped(&expf);
* }
*
* To learn more about this library, including the use of infinite generators,
* see the examples in the comments, or the docs (coming soon).
*/
namespace folly { namespace gen {
template<class Value, class Self>
class GenImpl;
template<class Self>
class Operator;
class EmptySequence : std::exception {
public:
virtual const char* what() const noexcept {
return "This operation cannot be called on an empty sequence";
}
};
class Less {
public:
template<class First,
class Second>
auto operator()(const First& first, const Second& second) const ->
decltype(first < second) {
return first < second;
}
};
class Greater {
public:
template<class First,
class Second>
auto operator()(const First& first, const Second& second) const ->
decltype(first > second) {
return first > second;
}
};
template<int n>
class Get {
public:
template<class Value>
auto operator()(Value&& value) const ->
decltype(std::get<n>(std::forward<Value>(value))) {
return std::get<n>(std::forward<Value>(value));
}
};
class Move {
public:
template<class Value>
auto operator()(Value&& value) const ->
decltype(std::move(std::forward<Value>(value))) {
return std::move(std::forward<Value>(value));
}
};
class Identity {
public:
template<class Value>
auto operator()(Value&& value) const ->
decltype(std::forward<Value>(value)) {
return std::forward<Value>(value);
}
};
namespace detail {
template<class Self>
struct FBounded;
/*
* Type Traits
*/
template<class Container>
struct ValueTypeOfRange {
private:
static Container container_;
public:
typedef decltype(*std::begin(container_))
RefType;
typedef typename std::decay<decltype(*std::begin(container_))>::type
StorageType;
};
/*
* Sources
*/
template<class Container,
class Value = typename ValueTypeOfRange<Container>::RefType>
class ReferencedSource;
template<class Value,
class Container = std::vector<typename std::decay<Value>::type>>
class CopiedSource;
template<class Value, bool endless = false, bool endInclusive = false>
class Sequence;
template<class Value, class First, class Second>
class Chain;
template<class Value, class Source>
class Yield;
/*
* Operators
*/
template<class Predicate>
class Map;
template<class Predicate>
class Filter;
template<class Predicate>
class Until;
class Take;
class Skip;
template<class Selector, class Comparer = Less>
class Order;
/*
* Sinks
*/
template<class Seed,
class Fold>
class FoldLeft;
template<class Reducer>
class Reduce;
class Sum;
template<class Selector,
class Comparer>
class Min;
template<class Container>
class Collect;
template<template<class, class> class Collection = std::vector,
template<class> class Allocator = std::allocator>
class CollectTemplate;
template<class Collection>
class Append;
}
/**
* Polymorphic wrapper
**/
template<class Value>
class VirtualGen;
/*
* Source Factories
*/
template<class Container,
class From = detail::ReferencedSource<const Container>>
From fromConst(const Container& source) {
return From(&source);
}
template<class Container,
class From = detail::ReferencedSource<Container>>
From from(Container& source) {
return From(&source);
}
template<class Container,
class Value =
typename detail::ValueTypeOfRange<Container>::StorageType,
class CopyOf = detail::CopiedSource<Value>>
CopyOf fromCopy(Container&& source) {
return CopyOf(std::forward<Container>(source));
}
template<class Value,
class From = detail::CopiedSource<Value>>
From from(std::initializer_list<Value> source) {
return From(source);
}
template<class Container,
class From = detail::CopiedSource<typename Container::value_type,
Container>>
From from(Container&& source) {
return From(std::move(source));
}
template<class Value, class Gen = detail::Sequence<Value, false, false>>
Gen range(Value begin, Value end) {
return Gen(begin, end);
}
template<class Value,
class Gen = detail::Sequence<Value, false, true>>
Gen seq(Value first, Value last) {
return Gen(first, last);
}
template<class Value,
class Gen = detail::Sequence<Value, true>>
Gen seq(Value begin) {
return Gen(begin);
}
template<class Value,
class Source,
class Yield = detail::Yield<Value, Source>>
Yield generator(Source&& source) {
return Yield(std::forward<Source>(source));
}
#define GENERATOR(type, body) \
::folly::gen::generator<type>( \
[=](const std::function<void(type)>& yield) \
{ body })
/*
* Operator Factories
*/
template<class Predicate,
class Map = detail::Map<Predicate>>
Map mapped(const Predicate& pred = Predicate()) {
return Map(pred);
}
template<class Predicate,
class Map = detail::Map<Predicate>>
Map map(const Predicate& pred = Predicate()) {
return Map(pred);
}
template<class Predicate,
class Filter = detail::Filter<Predicate>>
Filter filter(const Predicate& pred = Predicate()) {
return Filter(pred);
}
template<class Predicate,
class Until = detail::Until<Predicate>>
Until until(const Predicate& pred = Predicate()) {
return Until(pred);
}
template<class Selector,
class Comparer = Less,
class Order = detail::Order<Selector, Comparer>>
Order orderBy(const Selector& selector,
const Comparer& comparer = Comparer()) {
return Order(selector, comparer);
}
template<class Selector,
class Order = detail::Order<Selector, Greater>>
Order orderByDescending(const Selector& selector) {
return Order(selector);
}
template<int n,
class Get = detail::Map<Get<n>>>
Get get() {
return Get();
}
/*
* Sink Factories
*/
template<class Seed,
class Fold,
class FoldLeft = detail::FoldLeft<Seed, Fold>>
FoldLeft foldl(const Seed& seed, const Fold& fold) {
return FoldLeft(seed, fold);
}
template<class Reducer,
class Reduce = detail::Reduce<Reducer>>
Reduce reduce(const Reducer& reducer) {
return Reduce(reducer);
}
template<class Selector,
class Min = detail::Min<Selector, Less>>
Min minBy(const Selector& selector = Selector()) {
return Min(selector);
}
template<class Selector,
class MaxBy = detail::Min<Selector, Greater>>
MaxBy maxBy(const Selector& selector = Selector()) {
return MaxBy(selector);
}
template<class Collection,
class Collect = detail::Collect<Collection>>
Collect as() {
return Collect();
}
template<template<class, class> class Container = std::vector,
template<class> class Allocator = std::allocator,
class Collect = detail::CollectTemplate<Container, Allocator>>
Collect as() {
return Collect();
}
template<class Collection,
class Append = detail::Append<Collection>>
Append appendTo(Collection& collection) {
return Append(&collection);
}
}} // folly::gen
#include "folly/experimental/Gen-inl.h"
folly/experimental/test/GenBenchmark.cpp 0 → 100644
View file @ 506b9a32
/*
* Copyright 2012 Facebook, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include "folly/experimental/Gen.h"
#include <glog/logging.h>
#include <atomic>
#include "folly/Benchmark.h"
using namespace folly;
using namespace folly::gen;
using std::ostream;
using std::pair;
using std::set;
using std::vector;
using std::tuple;
static std::atomic<int> testSize(1000);
static vector<int> testVector =
seq(1, testSize.load())
| mapped([](int) { return rand(); })
| as<vector>();
static vector<vector<int>> testVectorVector =
seq(1, 100)
| map([](int i) {
return seq(1, i) | as<vector>();
})
| as<vector>();
auto square = [](int x) { return x * x; };
auto add = [](int a, int b) { return a + b; };
auto multiply = [](int a, int b) { return a * b; };
BENCHMARK(Sum_Basic_NoGen, iters) {
int limit = testSize.load();
int s = 0;
while (iters--) {
for (int i = 0; i < limit; ++i) {
s += i;
}
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_RELATIVE(Sum_Basic_Gen, iters) {
int limit = testSize.load();
int s = 0;
while (iters--) {
s += range(0, limit) | sum;
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_DRAW_LINE()
BENCHMARK(Sum_Vector_NoGen, iters) {
int s = 0;
while (iters--) {
for (auto& i : testVector) {
s += i;
}
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_RELATIVE(Sum_Vector_Gen, iters) {
int s = 0;
while (iters--) {
s += from(testVector) | sum;
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_DRAW_LINE()
BENCHMARK(Count_Vector_NoGen, iters) {
int s = 0;
while (iters--) {
for (auto& i : testVector) {
if (i * 2 < rand()) {
++s;
}
}
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_RELATIVE(Count_Vector_Gen, iters) {
int s = 0;
while (iters--) {
s += from(testVector)
| filter([](int i) {
return i * 2 < rand();
})
| count;
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_DRAW_LINE()
BENCHMARK(Fib_Sum_NoGen, iters) {
int s = 0;
while (iters--) {
auto fib = [](int limit) -> vector<int> {
vector<int> ret;
int a = 0;
int b = 1;
for (int i = 0; i * 2 < limit; ++i) {
ret.push_back(a += b);
ret.push_back(b += a);
}
return ret;
};
for (auto& v : fib(testSize.load())) {
s += v;
v = s;
}
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_RELATIVE(Fib_Sum_Gen, iters) {
int s = 0;
while (iters--) {
auto fib = GENERATOR(int, {
int a = 0;
int b = 1;
for (;;) {
yield(a += b);
yield(b += a);
}
});
s += fib | take(testSize.load()) | sum;
}
folly::doNotOptimizeAway(s);
}
struct FibYielder {
template<class Yield>
void operator()(Yield&& yield) const {
int a = 0;
int b = 1;
for (;;) {
yield(a += b);
yield(b += a);
}
}
};
BENCHMARK_RELATIVE(Fib_Sum_Gen_Static, iters) {
int s = 0;
while (iters--) {
auto fib = generator<int>(FibYielder());
s += fib | take(30) | sum;
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_DRAW_LINE()
BENCHMARK(VirtualGen_0Virtual, iters) {
int s = 0;
while (iters--) {
auto numbers = seq(1, 10000);
auto squares = numbers | map(square);
auto quads = squares | map(square);
s += quads | sum;
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_RELATIVE(VirtualGen_1Virtual, iters) {
int s = 0;
while (iters--) {
VirtualGen<int> numbers = seq(1, 10000);
auto squares = numbers | map(square);
auto quads = squares | map(square);
s += quads | sum;
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_RELATIVE(VirtualGen_2Virtual, iters) {
int s = 0;
while (iters--) {
VirtualGen<int> numbers = seq(1, 10000);
VirtualGen<int> squares = numbers | map(square);
auto quads = squares | map(square);
s += quads | sum;
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_RELATIVE(VirtualGen_3Virtual, iters) {
int s = 0;
while (iters--) {
VirtualGen<int> numbers = seq(1, 10000);
VirtualGen<int> squares = numbers | map(square);
VirtualGen<int> quads = squares | map(square);
s += quads | sum;
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_DRAW_LINE()
BENCHMARK(Concat_NoGen, iters) {
int s = 0;
while (iters--) {
for (auto& v : testVectorVector) {
for (auto& i : v) {
s += i;
}
}
}
folly::doNotOptimizeAway(s);
}
BENCHMARK_RELATIVE(Concat_Gen, iters) {
int s = 0;
while (iters--) {
s += from(testVectorVector) | rconcat | sum;
}
folly::doNotOptimizeAway(s);
}
// Results from a dual core Xeon L5520 @ 2.27GHz:
//
// ============================================================================
// folly/experimental/test/GenBenchmark.cpp relative time/iter iters/s
// ============================================================================
// Sum_Basic_NoGen 301.60ns 3.32M
// Sum_Basic_Gen 103.20% 292.24ns 3.42M
// ----------------------------------------------------------------------------
// Sum_Vector_NoGen 200.33ns 4.99M
// Sum_Vector_Gen 99.44% 201.45ns 4.96M
// ----------------------------------------------------------------------------
// Count_Vector_NoGen 19.07fs 52.43T
// Count_Vector_Gen 166.67% 11.44fs 87.38T
// ----------------------------------------------------------------------------
// Fib_Sum_NoGen 4.15us 241.21K
// Fib_Sum_Gen 48.75% 8.50us 117.58K
// Fib_Sum_Gen_Static 113.24% 3.66us 273.16K
// ----------------------------------------------------------------------------
// VirtualGen_0Virtual 10.05us 99.48K
// VirtualGen_1Virtual 29.63% 33.93us 29.47K
// VirtualGen_2Virtual 20.47% 49.09us 20.37K
// VirtualGen_3Virtual 15.30% 65.68us 15.23K
// ----------------------------------------------------------------------------
// Concat_NoGen 2.34us 427.15K
// Concat_Gen 90.04% 2.60us 384.59K
// ============================================================================
int main(int argc, char *argv[]) {
google::ParseCommandLineFlags(&argc, &argv, true);
runBenchmarks();
return 0;
}
folly/experimental/test/GenTest.cpp 0 → 100644
View file @ 506b9a32
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment