Commit 9385c05a authored by Andrew Smith's avatar Andrew Smith Committed by Facebook GitHub Bot

Add Transform overload that accepts a Transformer object

Summary:
This diff reduces the amount of memory that transforms use. Currently, transform explicitly takes an executor and a transform lambda (and resumableTransform takes an additional initialization lambda). Often, these lambdas end up taking pointers to some shared state object. In the case of resumableTransform, both lambdas need to capture a pointer to the same shared state object. Furthermore, the executor passed to transform is often present on the shared state object, making it redundant. This problem will get even worse when we add rate limiters (which also can be placed on the shared state object).

To reduce memory waste, this diff adds a new overload for transform and resumableTransform that takes in a Transformer object. This transformer object must implement getExecutor, transformValue, and (for resumableTransform) initializeValue. This allows a transformer object to contain just one 8-byte pointer to a shared state object, rather than duplicating pointers for different lambda captures and executors (and eventually rate limiters). It also slightly simplifies code in certain cases, as the shared state can be accessed as a field from any function in the class (rather than captured and passed down as a parameter to all helper functions).

We will still leave the existing overloads for cases where it is simpler for the user to not create a new object (when the user doesn't care as much about the memory savings).

Reviewed By: aary

Differential Revision: D33033365

fbshipit-source-id: e57c16e76ad3795a7f93b39565f553e76623beb6
parent b56cffe6
This diff is collapsed.
......@@ -79,6 +79,25 @@ Receiver<OutputValueType> transform(
folly::Executor::KeepAlive<folly::SequencedExecutor> executor,
TransformValueFunc transformValue);
/**
* This overload accepts arguments in the form of a transformer object. The
* transformer object must have the following functions:
*
* folly::Executor::KeepAlive<folly::SequencedExecutor> getExecutor();
*
* folly::coro::AsyncGenerator<OutputValueType&&> transformValue(
* folly::Try<InputValueType> inputValue);
*/
template <
typename ReceiverType,
typename TransformerType,
typename InputValueType = typename ReceiverType::ValueType,
typename OutputValueType =
typename decltype(std::declval<TransformerType>().transformValue(
std::declval<folly::Try<InputValueType>>()))::value_type>
Receiver<OutputValueType> transform(
ReceiverType inputReceiver, TransformerType transformer);
/**
* This function is similar to the above transform function. However, instead of
* taking a single input receiver, it takes an initialization function that
......@@ -141,6 +160,29 @@ Receiver<OutputValueType> resumableTransform(
InitializeTransformFunc initializeTransform,
TransformValueFunc transformValue);
/**
* This overload accepts arguments in the form of a transformer object. The
* transformer object must have the following functions:
*
* folly::Executor::KeepAlive<folly::SequencedExecutor> getExecutor();
*
* std::pair<std::vector<OutputValueType>, Receiver<InputValueType>>
* initializeTransform();
*
* folly::coro::AsyncGenerator<OutputValueType&&> transformValue(
* folly::Try<InputValueType> inputValue);
*/
template <
typename TransformerType,
typename ReceiverType =
typename decltype(std::declval<TransformerType>()
.initializeTransform())::StorageType::second_type,
typename InputValueType = typename ReceiverType::ValueType,
typename OutputValueType =
typename decltype(std::declval<TransformerType>().transformValue(
std::declval<folly::Try<InputValueType>>()))::value_type>
Receiver<OutputValueType> resumableTransform(TransformerType transformer);
/**
* An OnClosedException passed to a transform callback indicates that the input
* channel was closed. An OnClosedException can also be thrown by a transform
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment