Summary: Extending ApplyTuple to support this common functional construct, with good forwarding semantics.

Reviewed By: yfeldblum

Differential Revision: D4787560

fbshipit-source-id: 2c740e448e0cb916abe948b79709d5ecd8ba54bb

Summary:
[Folly] Cut extraneous typename in `StringKeyedCommon.h`.

In this case, `typename` is optional, and, to avoid confusion, should be removed.

"In this case" means in a template definition, with a qualified non-dependent name.

Reviewed By: ericniebler

Differential Revision: D4792087

fbshipit-source-id: dbb9d2c747432a1f9bc8823e72a7668e78a2fbaa

Summary:
This diff adds two tests to futures/test/ViaTest.cpp:
viaDummyExecutorFutureSetValueFirst and
viaDummyExecutorFutureSetCallbackFirst. The latter resulted in a
dead-lock before the fix contained in this diff.

It is important that the callback function is destroyed after
it is executed, since it may capture objects (like a Promise)
that should be destroyed (so that e.g. a corresponding Future
throws BrokenPromise). When the callback is executed via an
executor, it is possible that the executor doesn't get around to
executing the task. We shouldn't rely on the task being executed
to do necessary clean-up. That clean-up should (also) happen
when the task (with its captured data) is destroyed (in the
spirit of RIAA).

Reviewed By: djwatson

Differential Revision: D4779215

fbshipit-source-id: d029cf8b8f7b55e1b03357749c5fb62d95986ca7

Summary:
Add codec that does automatic format detection for decompression.
The codec returned by `getCodec(CodecType::AUTOMATIC)` can decompress codecs `LZ4_FRAME`, `ZSTD`, `ZLIB`, `GZIP`, and `LZMA2`.
Additionally, the function `getAutomaticCodec()` allows users to pass their own custom codecs, but the 5 default codecs must also be supported.
We force the default codecs on users because having a common subset of formats is in general useful, and because we want to be able to add codecs to this set.

When compiled in debug mode, on construction it checks that the codecs are compatible.
This helps make sure that users don't accidentally add ambiguous codecs, and if we add a new default codec in the future, tests will fail on ambiguity.

Reviewed By: yfeldblum

Differential Revision: D4760065

fbshipit-source-id: 9e65844aba9f1ae3d5c7b86643931782c06ab3eb

Summary:
If the compiler doesn't see thru the call to tryRead which always initializes the val variable, it will emit the following error.
I was getting error when compiling my android project:
error: 'val' may be used uninitialized in this function [-Werror=maybe-uninitialized]
This should not have a negative perf impact as smart compilers which inline call to tryRead can elliminate the initialization.

Reviewed By: djwatson

Differential Revision: D4790864

fbshipit-source-id: f353cfe54ca4d056b6ddfc075d00580c9f2d391e

Summary:
`ZlibCodec::doUncompress()` always allocates at least 4 MiB when the uncompressed size is unkown.
We can get a 10-20% speed boost for small inputs (as well as significant memory savings) if we dynamically choose the buffer size.

Benchmarks before the change:
  ============================================================================
  experimental/terrelln/CompressionBenchmark.cpp  relative  time/iter  iters/s
  ============================================================================
  uncompress(ZLIB_100)                                         2.96us  338.19K
  uncompress(ZLIB_1000)                                        7.22us  138.44K
  uncompress(ZLIB_10000)                                      36.86us   27.13K
  uncompress(ZLIB_100000)                                    299.70us    3.34K
  uncompress(ZLIB_1000000)                                     2.74ms   365.32
  uncompress(ZLIB_10000000)                                   26.91ms    37.16
  uncompress(ZLIB_100000000)                                 273.92ms     3.65
  ============================================================================
After:
  ============================================================================
  experimental/terrelln/CompressionBenchmark.cpp  relative  time/iter  iters/s
  ============================================================================
  uncompress(ZLIB_100)                                         2.34us  426.68K
  uncompress(ZLIB_1000)                                        6.24us  160.38K
  uncompress(ZLIB_10000)                                      35.52us   28.16K
  uncompress(ZLIB_100000)                                    296.30us    3.37K
  uncompress(ZLIB_1000000)                                     2.74ms   365.25
  uncompress(ZLIB_10000000)                                   27.16ms    36.82
  uncompress(ZLIB_100000000)                                 273.72ms     3.65
  ============================================================================

Reviewed By: yfeldblum

Differential Revision: D4783871

fbshipit-source-id: 0fcce191ea520fd2117551db679c86f5ff734893

Summary: It's needed downstream in HHVM.

Reviewed By: yfeldblum

Differential Revision: D4785204

fbshipit-source-id: fe81ed8e23497fffeb44b2a366c457b788bcaada

Reviewed By: yfeldblum

Differential Revision: D4782707

fbshipit-source-id: 535b42b4f2558cadc78037268d6de81a8bb49840

Summary: In some situations it is useful to be able to sort the JSON keys according to some different scheme for readability, so allow the sorting function to be changed by the serializer_opts

Reviewed By: yfeldblum

Differential Revision: D4782077

fbshipit-source-id: 032fa60a38804452bd1c22c67ba897521cb2cd1d

Summary:
D4715918 broke open source builds on Ubuntu 14.04, since it has lz4-r114, but the lz4 frame API was introduced in [r123](https://github.com/lz4/lz4/blob/r123/lz4frame.h).
Put the `LZ4FrameCodec` behind a lz4 version check.
Fixes https://github.com/facebook/fbthrift/issues/209.

Reviewed By: yfeldblum

Differential Revision: D4780830

fbshipit-source-id: 19492a7e6bdd128e610c36b5778274e19eff9548

Summary: Passing a temporary default value to `get_ref_default()` returns a dangling reference when it is used.

Reviewed By: lbrandy, yfeldblum

Differential Revision: D4768769

fbshipit-source-id: 4c58a17dc7662ad553cf88a8544dae20016d2f6f

Summary:
Fix a correctness bug in "SWMRList.h".

Thanks to Maged for the tip! The old code omitted setting a removed node's
"next" pointer to `nullptr`, which meant that if the writer removed node
A and then node B = A->next while the reader was looking at B, then the reader
might happily keep chasing pointers B->next, B->next->next,... without ever
noticing that it was now on a "dead branch" of the linked list.

(And then there's a bit of a trick: you really do have to remove the node
first and *then* set its "next" pointer to null, because if you do the null
assignment first, you'll truncate the list, which means that some readers
will see a truncated list and return the wrong answer. I've added comments
to try to explain this to future-me.)

Style nit: Avoid doing multiple atomic operations on the same line of code.

----

Modernize the parameter-passing conventions in SWMRList.h.

Pass `T`s by const reference, except in `add` where we're likely to want to
make a copy of the `T` anyway. In that case it would be more "STL-correct"
to supply two different overloads `add(T&&)` and `add(const T&)`, but this
is just an example so it makes sense to keep things simple.

----

Fix an undefined behavior in SWMRList example.

Searching an empty SWMRList<int> always invokes undefined behavior by
comparing an uninitialized `T elem` against `val` on the final line of
the `contains` function. Besides, this patch allows SWMRList to work
with `T`s that aren't default-constructible or even copy-constructible.

----

Closes https://github.com/facebook/folly/pull/566

Reviewed By: djwatson

Differential Revision: D4772359

Pulled By: yfeldblum

fbshipit-source-id: 8f96573530800675cb56006aa91e7a5c5c1fb85d

Summary:
Two changes:
- Add a lock holder interface to enable users to move exclusive access and implicitly release the held lock.
- Change an assert to a static_assert to check at compile time if a provided function fits in the Function structure without dynamic allocation.

Reviewed By: djwatson

Differential Revision: D4770229

fbshipit-source-id: 89408164c08d7660231a6ca4e37637dd688356cd

Summary:
Add a `tryRead()`, and endian variants, which try to read into an arithmetic type, and if there isn't enough data they return false.
One use case is to quickly check if an IOBuf starts with a certain prefix, benchmarks show that using `tryReadLE()` is 6x faster than using `pullAtMost()` and `memcmp()`.

Reviewed By: yfeldblum

Differential Revision: D4767855

fbshipit-source-id: feb8c61092772933d4b8496b27d464559ff8b827

Summary: `get_ref_default()` won't accept temporary default values after D4768769.

Reviewed By: yfeldblum

Differential Revision: D4769043

fbshipit-source-id: 46b039e13b80721922f69036c6cc02f5642a26f9

Summary:
The `LZMA2` codec doesn't allow decompression without passing the uncompressed size.
Every other codec that supports streaming decompression without the uncompressed size by streaming into an IOBuf chain.
Benchmarks show that reducing the `defaultBufferLength` to 256 KiB doesn't slow down decompression of large files, but it speeds up decompression of small files (< 1000 bytes).

Reviewed By: yfeldblum

Differential Revision: D4751571

fbshipit-source-id: 39dbe6754a1ecdc2b7ba3107e9face926d4c98ca

Summary:
It is ~10% faster to call `ZSTD_decompress()` than use the
streaming API. The streaming API does some extra `memcpy`s that we
can avoid. We are working on improving the speed of the streaming
API in the case where all the data can be processed in one shot,
but that won't be available in the stable ZSTD api for a few versions.

Reviewed By: yfeldblum

Differential Revision: D4731058

fbshipit-source-id: 39026c499c0f5002466097b5afe7e30f850e0ae8

Summary:
The LZ4 Frame codec encodes data using the LZ4 frame format.
One advantage of the LZ4 frame format is that it has 4 magic bytes in the header, so users can transparently determine compression type.
It also allows the user to interop with the lz4 command line tool.

Reviewed By: yfeldblum

Differential Revision: D4715918

fbshipit-source-id: 689833fef526b1cfe98685179e7b494380d49cba

Summary:
Extract the environment-as-STL-map logic out of `test::EnvVarSaver` and into a
separate class `experimental::EnvironmentState` so that other code which needs
to manipulate the environment can do so more easily.

Add routines to set the process environment from the state of an
`EnvironmentState`'s map, and to extract the environment in the forms required
by `Subprocess` and the UNIX routines `execve` etc.

Reviewed By: yfeldblum

Differential Revision: D4713307

fbshipit-source-id: 6b1380dd29b9ba41c97b886814dd3eee91fc1c0f

Summary:
Looks like some tests for the `io` directory were left behind when they were moved out of experimental. Re-enabled them in the new location, except for the compression test because it takes a lot longer. It's still built and can be executed manually.

Also added some preprocessor guards around compression codecs that might not be compiled in (and therefore fail during test SetUp).

Depends on #547 because the path to gtest appears in the new Makefile.am's `CPPFLAGS`.
Closes https://github.com/facebook/folly/pull/550

Reviewed By: yfeldblum

Differential Revision: D4631244

Pulled By: Orvid

fbshipit-source-id: 496b2c272e4f7293822f159776a02f7dd0c9e04d

Reviewed By: yfeldblum, ot

Differential Revision: D4742262

fbshipit-source-id: 7137bb8e1cb39c74e8ba726e76e9a8a6f3e1984c

Randomize the Unix socket name in AsyncSocketTest.SendMessageAncillaryData test to avoid collisions.

Summary:
Our test framework reports frequent failures of AsyncSocketTest.SendMessageAncillaryData. According to the logs the socket fails to bind:

    folly/io/async/test/AsyncSocketTest2.cpp:3098: Failure
    Expected: (bind(lfd, (struct sockaddr*)&addr, sizeof(addr))) != (-1), actual: -1 vs -1
    Bind failed: 98

This diff adds the socket name randomization to avoid name collisions between tests running concurrently on the same test box.

Reviewed By: yfeldblum

Differential Revision: D4758942

fbshipit-source-id: 6066dbc18222a4521c40b2ff218cb7dab8bd789d

Summary:
Change every instance of EXPECT_EQ(false, ... to the simpler/shorter EXPECT_FALSE(...
Do the same for ASSERT_EQ, for consistency, and for true/TRUE.

Reviewed By: Orvid

Differential Revision: D4751397

fbshipit-source-id: 045293dbf8be1167f0cb1b85a2f9cd59a6a30f73

Summary:
Flat combining template that takes the following template parameters:
 T         Concurrent data structure using FC interface
 Mutex Mutex type (default std::mutex)
 Atom  Atomic template (default std::atomic)
 Req    Optional request structure to hold custom info (default dummy type bool)

Flat combining (FC) was introduced in the SPAA 2010 paper Flat Combining and the Synchronization-Parallelism Tradeoff, by Danny Hendler, Itai Incze, Nir Shavit, and Moran Tzafrir.
http://mcg.cs.tau.ac.il/projects/projects/flat-combining

Reviewed By: djwatson

Differential Revision: D4602402

fbshipit-source-id: 38327f752a3e92bb01e5496c321d8c87c818087a

Summary: The "l" in "Haskell" is not a singleton. Also, friends don't let friends rely on operator precedence.

Reviewed By: yfeldblum

Differential Revision: D4756653

fbshipit-source-id: 7b5360bfc1b26fac06062e4adbfae3a1e477c2dc

Summary: D4719112 changed the signature of the constructor but didn't update this callsite...

Reviewed By: djwatson

Differential Revision: D4754745

fbshipit-source-id: ae538a3ffb64cdc211f23220ea5e0d7bb7eff9b3

Summary: On Android, we should also use a relaxed singleton

Reviewed By: andriigrynenko

Differential Revision: D4754227

fbshipit-source-id: 157844e618d216b768bbb07f36c380ba2f4c9ab8

Summary: Sometimes tools need to output a little more help context before or after help.

Reviewed By: yfeldblum

Differential Revision: D4719112

fbshipit-source-id: e6f59312d722e0669053644d862c81758855b538

Summary: The gflags portability header should always be used in Folly. There were a couple of places gflags was being used directly. This fixes that.

Reviewed By: yfeldblum

Differential Revision: D4748674

fbshipit-source-id: ba3c0564a39f870f1f5f9371e32d06eb84591ba7

Summary:
sorted_vector_{set,map} can silently drop insert if the hinted
insertion position is too small.  This diff fixes it.

Reviewed By: yfeldblum

Differential Revision: D4747319

fbshipit-source-id: 31e399d07a0b77b700edf034dc723cb997dc8e16

Summary:
`LZ4Codec::doCompress()` and `doUncompress()` need to coalesce a `const IOBuf*`, so they `clone()` and then `coalesce()`.
It is more efficient to do this in one step, as shown by the benchmarks.

Reviewed By: yfeldblum, simpkins

Differential Revision: D4727802

fbshipit-source-id: f8d5f6187f2ee804301232b3c75b848330169fc5

Summary: Submitting mostly to see what breaks.

Reviewed By: yfeldblum

Differential Revision: D4249032

fbshipit-source-id: d4e3fdfd57750c8dcabdb4c01ab3b528c4818624

Summary: VS 2017 is here and the bugs these were hitting have been fixed, so re-enable them.

Reviewed By: yfeldblum

Differential Revision: D4742829

fbshipit-source-id: 898e7cfea9de9ffa813c562c12e0e18e0577c8c3

In AsyncSocketTest.SendMessageFlags test use folly::test::TemporaryFile object instead of directly creating a file with O_TMPFILE flag

Summary: O_TMPFILE flag is not available on older Linux systems (supported starting Linux 3.11). Got complains about it on GitHub (https://fburl.com/9d848m7i) Will use folly::test::TemporaryFile instead.

Reviewed By: yfeldblum

Differential Revision: D4729733

fbshipit-source-id: 903563825c5b939e35c61725da559e33d21212c4

Summary: This code looks really weird and is otherwise confusing.

Reviewed By: andriigrynenko

Differential Revision: D4730812

fbshipit-source-id: d9f4e73145e81f085ce959a76cdb568d3ae4eb52

Summary: Re-landing D4618623 after it had to be reverted due to it's use of `FixedString`, which had broken GCC 4.8. `FixedString` should now be compiling correctly on GCC 4.8 so this should be fine now.

Reviewed By: yfeldblum

Differential Revision: D4727022

fbshipit-source-id: f85c6952fd82e6cd6042f8055ef54d9422f8b0f9

Summary:
folly/python/futures.h provides some helper functions for bridging folly::future to asyncio.Future.

folly/python/NotificationQueueExecutor.h is a Driveable executor that has a fileno() method that can be monitored using (select, epoll) to determine if the drive method should be called.

folly/python/executor.pyx is an implementation of a "driver" for the NotificationQueueExecutor from the python asyncio side. It tracks also keeps track of asyncio eventloops to Executor mappings.

the getExecutor() from folly/python/futures.h uses that mapping to return the correct executor for this python thread.

Reviewed By: andriigrynenko, yfeldblum

Differential Revision: D4687029

fbshipit-source-id: e79314606ffa18cb6933fe6b749991bfea646cde

Summary:
Fix warnings exposed by gcc's -Wshadow-compatible-local.
Rename lambda parameter "ts" to "timeSeries".

This avoids the following errors:

  folly/test/TimeseriesTest.cpp:802:27: error: declaration of 'ts' shadows a previous local [-Werror=shadow-compatible-local]
  folly/test/TimeseriesTest.cpp:809:31: error: shadowed declaration is here [-Werror=shadow-compatible-local]

Reviewed By: yfeldblum, liuxuli

Differential Revision: D4735180

fbshipit-source-id: 59b3fecf4031b3c6615e5b764fbc4ea04754eb50

Summary: Use perfect forwarding in `EventBaseLocal::getOrCreate` to make it work with noncopyable but moveable arguments.

Reviewed By: yfeldblum

Differential Revision: D4730566

fbshipit-source-id: fa02348b7a9217fef980ec5e743b5990b9d19e9a

Summary: This is more in line with "traditional" output from cli tools on *nix.

Reviewed By: yfeldblum

Differential Revision: D4719055

fbshipit-source-id: 2389a4a2a49f18d4b3d0257f68036f73f8fbc49e