Reviewed By: lbrandy

Differential Revision: D3964328

fbshipit-source-id: 0b1dc2d4b2a1565019770668445a624625b8257e

Summary:
This diff adds BatchDispatcher, which can be used to batch values while performing IO.
This would be useful in writing single id code in node, and would be use to batch ids while doing IO at storage adapter.

Differential Revision: D3900404

fbshipit-source-id: f53aa352344ff55674c7544302b6a1b4726214b6

Summary:
Almost every method of `fbstring` needs to perform category
dispatching. The category constants are `size_t`, which become 8-byte
immediate values in the dispatching code, so even a simple `if
(category() == Category::isSmall)` is quite large. When inlined
hundreds of thousands of time, it adds up.

This diff redefines the category type to be 1 byte (without changing
the ABI). It also optimizes `size()` and `c_str()` and makes them
branch-free, which probably is not going to have any perf impact but
it saves a few bytes.

Generated code for some small functions:

- `reset()`
Before:
```
48 ba 00 00 00 00 00    movabs $0x1700000000000000,%rdx
00 00 17
48 89 f8                mov    %rdi,%rax
c6 07 00                movb   $0x0,(%rdi)
48 89 57 10             mov    %rdx,0x10(%rdi)
```

20 bytes

After:
```
48 89 f8                mov    %rdi,%rax
c6 47 17 17             movb   $0x17,0x17(%rdi)
c6 07 00                movb   $0x0,(%rdi)
```

10 bytes

- `c_str()`
Before:
```
48 b8 00 00 00 00 00    movabs $0xc000000000000000,%rax
00 00 c0
48 85 47 10             test   %rax,0x10(%rdi)
74 08                   je     401fd8
48 8b 07                mov    (%rdi),%rax
c3                      retq
0f 1f 40 00             nopl   0x0(%rax)
48 89 f8                mov    %rdi,%rax
```

26 bytes (without the `retq`)

After:
```
f6 47 17 c0             testb  $0xc0,0x17(%rdi)
48 89 f8                mov    %rdi,%rax
48 0f 45 07             cmovne (%rdi),%rax
```

11 bytes

- `size()`
Before:
```
48 b8 00 00 00 00 00    movabs $0xc000000000000000,%rax
00 00 c0
48 85 47 10             test   %rax,0x10(%rdi)
74 08                   je     401fa8
48 8b 47 08             mov    0x8(%rdi),%rax
c3                      retq
0f 1f 00                nopl   (%rax)
48 0f be 57 17          movsbq 0x17(%rdi),%rdx
b8 17 00 00 00          mov    $0x17,%eax
48 29 d0                sub    %rdx,%rax
```

36 bytes (without the `retq`)

After:
```
0f b6 57 17             movzbl 0x17(%rdi),%edx
b8 17 00 00 00          mov    $0x17,%eax
48 29 d0                sub    %rdx,%rax
48 0f 48 47 08          cmovs  0x8(%rdi),%rax
```

17 bytes

Reviewed By: philippv

Differential Revision: D3957276

fbshipit-source-id: ef40d82bbbb0456b1044421cd02133c268abe39b

Summary:
D3278901 never worked, GCC (at least 4.9) failed to propagate
`FOLLY_TARGET_ATTRIBUTE("arch=haswell")` down, `blsr` and `select64`
weren't inlined in EF coding on Haswell, showing up in CPU profile.

This diff switches back to inline assembly, removing second dispatching
mechanism (IFUNC; in additional to caller-side `Default` / `Nehalem` / `Haswell`
dispatching) for `compression::instructions`, and disables functionality on
compilers not supporting AT&T syntax (MSVC++) for now.

Added `FOLLY_ALWAYS_INLINE` to make sure failure to inline results in
compilation failure.

Reviewed By: ot

Differential Revision: D3959438

fbshipit-source-id: e40573fbfbf38991caa2cd70293aeaeeec3afad7

Summary:
Any attempt to copy or move an object returned by folly::partial
yields a compiler error because it is invoking the constructor
for wrapping callable objects, which triggers a type mismatch.

This diff fixes that by explicitly naming the default implementations
of copy and move constructors.

This diff also adds additional tests that fail to compile without this fix.

And then this diff also moves the Partial class into folly::detail,
because it is not meant to be named in user code, but only returned
by the folly::partial function.

Reviewed By: mhx

Differential Revision: D3923809

fbshipit-source-id: a8883951afd2a1999acbfffc51296393b058f860

Summary:
A ton of these functions are tiny and should be inlined. `throw std::out_of_range("bla")` generates quite a lot of code.

See godbolt from {D3940968}: https://godbolt.org/g/9K36Km

  advanceNoInline(S):
          movq    %rdi, %rax
          sarq    $32, %rax
          cmpl    %eax, %edi
          jg      .L20
          leal    1(%rdi), %eax
          ret
  .L20:
          subq    $8, %rsp
          call    S::outOfRange() [clone .isra.0]

vs previous implementation

  advance(S):
          movq    %rdi, %rdx
          sarq    $32, %rdx
          cmpl    %edx, %edi
          jg      .L14
          leal    1(%rdi), %eax
          ret
  .L14:
          pushq   %rbp
          pushq   %rbx
          movl    $16, %edi
          subq    $8, %rsp
          call    __cxa_allocate_exception
          movl    $.LC0, %esi
          movq    %rax, %rdi
          movq    %rax, %rbx
          call    std::out_of_range::out_of_range(char const*)
          movl    std::out_of_range::~out_of_range(), %edx
          movl    typeinfo for std::out_of_range, %esi
          movq    %rbx, %rdi
          call    __cxa_throw
          movq    %rax, %rbp
          movq    %rbx, %rdi
          call    __cxa_free_exception
          movq    %rbp, %rdi
          call    _Unwind_Resume

This pattern is in the standard library as well:
```
      reference
      at(size_type __n)
      {
	if (__n >= _Nm)
	  std::__throw_out_of_range(__N("array::at"));
	return _M_instance[__n];
      }
```

https://github.com/gcc-mirror/gcc/blob/edd716b6b1caa1a5cb320a8cd7f626f30198e098/libstdc%2B%2B-v3/include/tr1/array#L138

Reviewed By: yfeldblum

Differential Revision: D3947075

fbshipit-source-id: 6e174c725791762a533a534c8482ea9576460b86

Summary:
libdwarf headers moved from `/usr/include` to `/usr/include/libdwarf` in [dwarfutils 20160613-2](http://metadata.ftp-master.debian.org/changelogs/main/d/dwarfutils/unstable_changelog).

Adds a check for the library as well, and fixes a tiny typo in the `libunwind` check.

No longer checks for `libdwarf.h`; only `dwarf.h` is used.
Closes https://github.com/facebook/folly/pull/480

Reviewed By: yfeldblum

Differential Revision: D3878961

Pulled By: Orvid

fbshipit-source-id: 7970d33511bf6d351cec2c8077cb07b85f2f8d33

Summary:
No need to define our own.

Generated code is similar: https://godbolt.org/g/5xWrNx (one extra move instruction).

```
.L20:
        subq    $8, %rsp
        call    S::outOfRange() [clone .isra.0]
```

```
.LC1:
        .string "out of range"
.L26:
        subq    $8, %rsp
        movl    $.LC1, %edi
        call    std::__throw_out_of_range(char const*)
```

Reviewed By: Orvid

Differential Revision: D3945578

fbshipit-source-id: c65e9dea55e8f01f51766b2695af68d2bc92c266

Summary:
Acquire a SharedMutex at ThreadExit to ensure that after unlinking the ThreadEntry from
the list, future accessAllThreads() won't miss a destroying thread.

This is quite a dangerous fix as it changes some lock ordering semantics. ThreadLocal
elements are now destroyed while holding a lock, so if the destruction function
acquires a different lock, ordering must be consistent with other
uses of accessAllThreads().

I've made accessAllThreads() an opt-in feature via a template parameter and changed
all existing uses. I've also fixed a few lock ordering issues that arose due to this
change.

Reviewed By: andriigrynenko

Differential Revision: D3931072

fbshipit-source-id: 4d464408713184080079698df453b95873bb1a6c

Summary:
When using connectx to do TFO, apple has a bug
where the second write after a TFO write will cause
the socket to throw an ENOTCONN error instead of an
EAGAIN. Linux handles this case fine and returns an
EAGAIN, however apple returns ENOTCONN.

We solve this by treating ENOTCONN as an EAGAIN temporarily.

Reviewed By: yfeldblum

Differential Revision: D3942681

fbshipit-source-id: ab4f0b5fd6cdcfe9c584ea00849705a2d739d65f

Summary:
Adds TFO functions for apple devices
Also allows android as well by removing that
restriction. Newer versions of android support
TFO, so this allows us to experiment with them

Reviewed By: yfeldblum

Differential Revision: D3942664

fbshipit-source-id: faf439783b018cf7c987a2e3ade5ea6c0c02bf48

Summary:
Here's some godbolt: https://godbolt.org/g/9K36Km

  advanceNoInline(S):
          movq    %rdi, %rax
          sarq    $32, %rax
          cmpl    %eax, %edi
          jg      .L20
          leal    1(%rdi), %eax
          ret
  .L20:
          subq    $8, %rsp
          call    S::outOfRange() [clone .isra.0]

vs previous implementation

  advance(S):
          movq    %rdi, %rdx
          sarq    $32, %rdx
          cmpl    %edx, %edi
          jg      .L14
          leal    1(%rdi), %eax
          ret
  .L14:
          pushq   %rbp
          pushq   %rbx
          movl    $16, %edi
          subq    $8, %rsp
          call    __cxa_allocate_exception
          movl    $.LC0, %esi
          movq    %rax, %rdi
          movq    %rax, %rbx
          call    std::out_of_range::out_of_range(char const*)
          movl    std::out_of_range::~out_of_range(), %edx
          movl    typeinfo for std::out_of_range, %esi
          movq    %rbx, %rdi
          call    __cxa_throw
          movq    %rax, %rbp
          movq    %rbx, %rdi
          call    __cxa_free_exception
          movq    %rbp, %rdi
          call    _Unwind_Resume

Reviewed By: ot

Differential Revision: D3940968

fbshipit-source-id: b47a41e7cdd863fcef099ff3c21860b2979ee6e8

Summary:
`Range` has a somewhat inconsistent API: most methods assume
argument validity as a precondition (mirroring their STL
counterparts), others check the arguments and throw for invalid ones.

Since `Range` is intended as a zero-cost abstraction on top of
pointers/iterators, unchecked methods should be preferred. At this
point however we cannot change the semantics of `advance()` and other
methods. This diff adds new unchecked versions of these methods.

Reviewed By: luciang

Differential Revision: D3938480

fbshipit-source-id: 6952683ee0716aa1584e79584158fbf3e083b52e

Summary:
The original code `AC_DEFINE([FOLLY_SUPPLY_MISSING_INT128_TRAITS...` resulted in defining the constant FOLLY_FOLLY_SUPPLY_MISSING_INT128_TRAITS in folly-config.h. This patch fix the name of the constant so that the name match with what appear in Traits.h.
Closes https://github.com/facebook/folly/pull/485

Reviewed By: yfeldblum

Differential Revision: D3931349

Pulled By: Orvid

fbshipit-source-id: bd7e7d3580d02134d36706bd5836822135232d26

Summary: Fix includes in memory_resource.h

Reviewed By: yfeldblum

Differential Revision: D3935848

fbshipit-source-id: 713ce85ad17072779dfa1c667459e367c5e1d4b7

Summary:
`SharedMutexImpl::lockSharedImpl` has a potentially uninitialized access:

  Assume state = 0
  canAlreadyDefer = (state & kMayDefer) != 0 ==> false
  aboveDeferThreshold = (state & kHasS) >= (kNumSharedToStartDeferring - 1) * kIncrHasS ==> false

  if (canAlreadyDefer || (aboveDeferThreshold && !drainInProgress)) ==> false

  line:1452: gotSlot(slot)->compare_exchange_strong(...) uses slot uninitialized

Reviewed By: Orvid

Differential Revision: D3933638

fbshipit-source-id: 0fbce5c00b8b1f34e50c302cb88def97853c5afe

Summary:
This commit introduces a simple function reference type, similar to
std::reference_wrapper, but the template parameter is the function
signature type rather than the type of the referenced object.

A folly::FunctionRef is cheap to construct as it contains only a
pointer to the referenced callable and a pointer to a function which
invokes the callable.

The user of FunctionRef must be aware of the reference semantics:
storing a copy of a FunctionRef is potentially dangerous and should
be avoided unless the referenced object definitely outlives the
FunctionRef object. Thus any function that accepts a FunctionRef
parameter should only use it to invoke the referenced function and
not store a copy of it. Knowing that FunctionRef itself has reference
semantics, it is generally okay to use it to reference lambdas that
capture by reference.

Reviewed By: ericniebler

Differential Revision: D3277364

fbshipit-source-id: 0a7676919cd240da5b6e1f94cadba4289e0aca28

Summary:
This is a start to wrapping various SSL objects going forward so different
binaries can choose different version of OpenSSL (i.e., BoringSSL, OpenSSL
1.1.0, OpenSSL 1.0.2, etc.). There's no change to the caller - everyone just
uses 'SSLSession', but the implementation details vary

Reviewed By: siyengar

Differential Revision: D3707791

fbshipit-source-id: f895334a768cb7d43b41af40c9bc06be5307cc7f

Summary:
OpenSSL SSL_METHOD->get_cipher_by_char is not present in either OpenSSL
1.1.0 or BoringSSL. In addition, knekritz reports that time's being spent in
binary searching for cipher names.

Since the ciphercodes and names are (fairly) static, we store these in a static
hash map.

Reviewed By: siyengar

Differential Revision: D3275185

fbshipit-source-id: 08b36f3e73239b415b74c6ecc30ed65832d9ebd0

Summary: Makes it easy to split a Future into multiple Futures

Reviewed By: rongrong

Differential Revision: D3885897

fbshipit-source-id: 6ac9fb22444dd828fbdebb44b06bf3d93d0f7583

Summary: See title.

Reviewed By: nbronson

Differential Revision: D3914188

fbshipit-source-id: dd9ccd0c48911632d229ae675cc40d835ea14724

Summary:
In debug mode, CpuId() ends up triggering external linkage from inside this __ifunc__.
However, in PIC, the relocation of external symbols are not ready yet, causing a segfault.

This only reproes in dynamic linking (PIC / so) and dbg mode.

Reviewed By: luciang

Differential Revision: D3895239

fbshipit-source-id: 2b7856c10abb5cfe24736d5bfac28e7e9d0e8272

Summary: Switch from mkstemp to the slightly more portable `folly::test::TemporaryFile`

Reviewed By: jsedgwick

Differential Revision: D3890411

fbshipit-source-id: e98d1e3a5adae92af1bb36f6213b194f633fab0f

Reviewed By: nbronson

Differential Revision: D3905865

fbshipit-source-id: 2743af4ae1b34adb0f8e611e672f9b6068430ec9

Summary:
Sometime we have such unit-test failure

```
folly/futures/test/TimekeeperTest.cpp:134: Failure
Value of: flag
  Actual: false
Expected: true
```

This diff should make it less flaky, though I am not sure what's the best way to fix this flaky unit-test.

Reviewed By: yfeldblum

Differential Revision: D3889630

fbshipit-source-id: e7486d75fbcb3081d06724d213d4a2cf8942955d

Summary:
Add logs for TFO to AsyncSocket
and wangle

Reviewed By: knekritz

Differential Revision: D3906104

fbshipit-source-id: 9a79b6f91273f8017a5e0d72fe5bdc8eff645ebc

Summary: Fix the build due to a missing gflags header

Reviewed By: yfeldblum

Differential Revision: D3905525

fbshipit-source-id: cc09189196a04741162c2d1561545d3fec417e5e

Summary:
libc++ 3.9 and above define the __throw* functions inside stdexcept, so
defining them ourselves leads to compilation errors when compiling
against libc++ 3.9. Add appropriate preprocessor guards to avoid this.

Reviewed By: meyering

Differential Revision: D3898284

fbshipit-source-id: 435a28c2b3a83ee4d8f5af0df0343c524469011e

Summary:
When we attach and detach from the
event base threads during connections
in the case of SSL it could result in
a crash while using TFO.

When TFO was enabled connectTimeout_
was set, which was not detached.

In the case when we would
fall back from TFO in case TFO did not
succeed, we would try and schedule a
connect timeout again. However because
we are now scheduled in a new evb thread,
this would cause the scheduleTimeout to
assert that it was not in the correct evb
thread.

This fixes the issue by attaching and detaching
connect timeouts as well.

Reviewed By: ngoyal

Differential Revision: D3892490

fbshipit-source-id: 278c0b8029022144cd59366ceb0ce83f0a60a307

Summary:
This diff changes the definition of `folly::Func` from `std::function<void()>`
over to `folly::Function<void()>`. This mostly affects the interface of
`folly::Executor` and derived and related classes. By using
`folly::Function<void()>`, we allow to have lambdas capturing move-only types
passed to executors. This continues an effort to get rid of the potentially
dangerous `folly::MoveWrapper` by allowing to capture move-only types in lambdas
when using `folly::Future`, `folly::EventBase` and now `folly::Executor`.

Reviewed By: yfeldblum

Differential Revision: D3706449

fbshipit-source-id: 11c2cee32cb9f0298c39b7b1695a477777eeb3aa

Summary: Make draft of hazard pointers prototype public

Reviewed By: djwatson

Differential Revision: D3870280

fbshipit-source-id: e029efa336585055f67687059e10ae11766f8d7f

Summary:
`EventBase` has its own methods to wait till it's ready so we can avoid using
barriers and remove boost dependency.

Reviewed By: mzlee

Differential Revision: D3894408

fbshipit-source-id: f050a982c98c4f672cf295845115686c95fc7919

Summary:
Updating fibers code to use https://github.com/llvm-mirror/compiler-rt/commit/b0477747dfa8a9706f2c902e877e616aca51e06f

Patch by andriigrynenko!

Reviewed By: andrewcox

Differential Revision: D3500482

fbshipit-source-id: f51f4bb4ebf0d95a898eb1d4098459aa691acd61

Summary:
Existing logic is broken (unable to correctly handle chained `IOBuf`
in case of both `compress` and `uncompress`) and has unnecessarly strict
`needsUncompressedLength() == true` requirement.

This diff switches `ZSTDCodec` to use streaming to handle chained `IOBuf`,
drops `needsUncompressedLength() == true`.

Reviewed By: luciang

Differential Revision: D3827579

fbshipit-source-id: 0ef6a9ea664ef585d0e181bff6ca17166b28efc2

Summary:
When trying to find an empty deferred reader slot, getting the current CPU can
take quite a few cycles, e.g. >1% CPU on SMC (https://fburl.com/434646643).

Let's track the last slot used by this thread and try that slot first before reading
CPU id and doing the search.

u-benchmark results seem to be improving generally (though a bit noisy and not
sure how much to trust). Results w/ this diff on left side:
P56648675

Reviewed By: nbronson

Differential Revision: D3857793

fbshipit-source-id: 8b1c005362c82e748a663100f889b0b99dc257fe

Summary: Fixing ODR violations...

Reviewed By: simpkins, yfeldblum

Differential Revision: D3875667

fbshipit-source-id: 0d8ec0b48e14fffb7e3e60c0e68e2576b2f58d1e

Summary:
Since `shellify` interprets the command as a format string we need to escape
`{` and `}`.

Reviewed By: ldemailly, simpkins

Differential Revision: D3874605

fbshipit-source-id: f47db387c3a44a3ba1c0c1d4726b7212fcb5ef3e

Summary:
Update the stats APIs to correcly distinguish between TimePoint and Duration
types.

This does leave addValue() and update() APIs in place that accept Duration
values, for backwards compatibility.  These should eventually be removed once
all code has been converted to call the new APIs.

Reviewed By: yfeldblum

Differential Revision: D3808805

fbshipit-source-id: 36d6574ba4a09db7eb9f1a35e47addd3e07f8461

Summary: Let the caller do a move, don't force one.

Reviewed By: yfeldblum

Differential Revision: D3873129

fbshipit-source-id: 40c6bf564bcbf794830c99ea1248a9c1bb30e9b0

Summary:
I've found all code paths that contributed to the number of threads in new ScribeClient. Here's a list of flags that reduce the number of thread:
  --default_tls_thread_count=1 # was 32
  --sr2_event_base_pool_size=1
  --has_thrift_dispatcher_reporter=0 # creates a thrift server with a few extra threads
  --observer_manager_pool_size=1 # default is 4

Optimizations in this diff:
- Don't initialize OBCClient object if it is not used
- Added FLAG_observer_manager_pool_size to control size of the pool in ObserverManager

Currently OBC counters are switched off globally in ScribeCliean, it means that new code path that creates SR instances is disabled, but eventually we plan to switch it on. Clients that don't use SR and want to keep their RSS/threads profile low, should pass flags listed above. I'll announce this in ScribeUsers group as well.

Differential Revision: D3870704

fbshipit-source-id: 0efad6b3dc43c072ab11cac7e9461c09532ea11c