Summary: Pull Request resolved: https://github.com/facebook/folly/pull/944

Reviewed By: Orvid

Differential Revision: D10231693

Pulled By: yfeldblum

fbshipit-source-id: 9c0c5d7c4c1e4d9d9860c71e608e47d661a41fdd

Summary:
icc accepts the syntax, but doesn't emit the definition into the
object file.

Reviewed By: ricklavoie, yfeldblum

Differential Revision: D10228495

fbshipit-source-id: 73c1676deb4963a9c30886414cc1722d44a79214

Summary:
Remove falls to Future::then that take a nullary lambda. This is deprecated and will be deleted.

This change replaces them with thenValue([](auto&&){...}) or similar.

This does not replace all calls to then with thenValue - other forms will be replaced when those forms of then are due for deletion.

Reviewed By: yfeldblum

Differential Revision: D10219070

fbshipit-source-id: 13b6f64ee2d7c8741537fe1131bdf3b251361413

Summary:
Similar to `--dynamic_cputhreadpoolexecutor`, this is a temporary
kill switch until all outstanding issues are addressed that would allow us
to use dynamic thread pools unconditionally.

Reviewed By: djwatson

Differential Revision: D10231230

fbshipit-source-id: b0abd77ff2a6a7ce31727e14eca799a6f3b5535a

Summary:
There were some read-modify-write operations that were behaving as read and
then modify.  This is incorrect when you use DeterministicAtomic without a
backing schedule.

This is okay if the test is operating in the presence of a
DeterministicSchedule schedule, but is probably worth fixing anyway.  If using
DeterministicAtomic without a backing schedule is considered a bug, we can add
an assertion in beforeSharedAccess() and afterSharedAccess() so users know not
to try that

Reviewed By: djwatson, nbronson

Differential Revision: D10190682

fbshipit-source-id: 6a3173d904a40bbc5c96b934a76aff90a48c8608

Summary:
icc gets confused on the parameter unpacking otherwise.

https://godbolt.org/z/rSRZYn

Reviewed By: yfeldblum

Differential Revision: D10147211

fbshipit-source-id: 85de581ffa4c487523a5e9f173636f74863a82ac

Summary:
When they come before the explicit instantiations of Rob, icc fails to
compile certain combintions of calls.

Reviewed By: alexeyt

Differential Revision: D10144943

fbshipit-source-id: ce63a4d289f64d1d644b116ab99b45b9da013162

Summary:
`folly::SharedMutex` allows `unlock_shared()` to be called in a different thread from the one that locked it
There are some places in folly that use this property.

Example: In `ObserverManager`, the calling thread

1. acquires the read lock
2. then it schedules some async work and gives ownership of the lock guard to the lambda
3. the thread that performs the async work will unlock the mutex

However, this causes false positives for TSAN. It looks like this breaks TSAN's assumptions that a read-write mutex is always read locked and unlocked in the same thread.
In the example above, TSAN thinks the calling thread in step (1) still owns the mutex, even though semantically the mutex is now owned by the thread in step (3).

This results in false positives for running unit tests with an annotated version of `folly::SharedMutex`.
TSAN reports a lock inversion between the `folly::Singleton vaule_.state_` mutex, and the read lock referenced in `ObserverManager`.

To suppress this error, this change annotates ObserverManager to let TSAN know that the the thread in step 1 has dropped
the lock, and the thread in step 3 now owns the lock.

Reviewed By: andriigrynenko

Differential Revision: D9797937

fbshipit-source-id: c0e5277bb9cb2f404df8abd3c3d6ea4a16460c78

Summary:
TSAN flagged the following lock inversion below. The reason for this lock inversion
is because of the following example usage:

1. set request context B, save old one A (lock B, lock A)
2. later on, reset request context back to A, get the old one back B (lock A, lock B)

In the two calls, we always locked the the new context first, resulting
in a lock inversion. The fix is to always acquire them in a consistent
order using `folly::acquireLocked`

TSAN report:

```
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=1574307)
  Cycle in lock order graph: M159 (0x7b1400000138) => M157 (0x7b14000000e8) => M159

  Mutex M157 acquired here while holding mutex M159 in main thread:
    #0 0x48f18c in AnnotateRWLockAcquired ??:?
    #1 0x315495 in folly::SharedMutexImpl<false, void, std::atomic, false>::lock_shared() ./folly/SharedMutex.h:385
    #2 0x1b0eb8 in folly::detail::LockTraitsImpl<folly::SharedMutexImpl<false, void, std::atomic, false>, (folly::detail::MutexLevel)1, false>::lock_shared(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:157
    #3 0x1b0e68 in std::integral_constant<bool, true> folly::LockPolicyShared::lock<folly::SharedMutexImpl<false, void, std::atomic, false> >(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:493
    #4 0x1d63a9 in LockedPtrBase ./folly/Synchronized.h:1026
    #5 0x1d6258 in LockedPtr ./folly/Synchronized.h:1326
    #6 0x1c9a17 in folly::SynchronizedBase<folly::Synchronized<folly::RequestContext::State, folly::SharedMutexImpl<false, void, std::atomic, false> >, (folly::detail::MutexLevel)1>::rlock() const ./folly/Synchronized.h:123
    #7 0x1c7530 in folly::RequestContext::setContext(std::shared_ptr<folly::RequestContext>) ./folly/io/async/Request.cpp:243
    #8 0x36a7b in folly::RequestContext::create() ./folly/io/async/Request.h:119
    #9 0x34916 in RequestContextScopeGuard ./folly/io/async/Request.h:269

  Mutex M159 previously acquired by the same thread here:
    #0 0x48f18c in AnnotateRWLockAcquired ??:?
    #1 0x315495 in folly::SharedMutexImpl<false, void, std::atomic, false>::lock_shared() ./folly/SharedMutex.h:385
    #2 0x1b0eb8 in folly::detail::LockTraitsImpl<folly::SharedMutexImpl<false, void, std::atomic, false>, (folly::detail::MutexLevel)1, false>::lock_shared(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:157
    #3 0x1b0e68 in std::integral_constant<bool, true> folly::LockPolicyShared::lock<folly::SharedMutexImpl<false, void, std::atomic, false> >(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:493
    #4 0x1d63a9 in LockedPtrBase ./folly/Synchronized.h:1026
    #5 0x1d6258 in LockedPtr ./folly/Synchronized.h:1326
    #6 0x1c9a17 in folly::SynchronizedBase<folly::Synchronized<folly::RequestContext::State, folly::SharedMutexImpl<false, void, std::atomic, false> >, (folly::detail::MutexLevel)1>::rlock() const ./folly/Synchronized.h:123
    #7 0x1c7516 in folly::RequestContext::setContext(std::shared_ptr<folly::RequestContext>) ./folly/io/async/Request.cpp:242
    #8 0x36a7b in folly::RequestContext::create() ./folly/io/async/Request.h:119
    #9 0x34916 in RequestContextScopeGuard ./folly/io/async/Request.h:269

  Mutex M159 acquired here while holding mutex M157 in main thread:
    #0 0x48f18c in AnnotateRWLockAcquired ??:?
    #1 0x315495 in folly::SharedMutexImpl<false, void, std::atomic, false>::lock_shared() ./folly/SharedMutex.h:385
    #2 0x1b0eb8 in folly::detail::LockTraitsImpl<folly::SharedMutexImpl<false, void, std::atomic, false>, (folly::detail::MutexLevel)1, false>::lock_shared(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:157
    #3 0x1b0e68 in std::integral_constant<bool, true> folly::LockPolicyShared::lock<folly::SharedMutexImpl<false, void, std::atomic, false> >(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:493
    #4 0x1d63a9 in LockedPtrBase ./folly/Synchronized.h:1026
    #5 0x1d6258 in LockedPtr ./folly/Synchronized.h:1326
    #6 0x1c9a17 in folly::SynchronizedBase<folly::Synchronized<folly::RequestContext::State, folly::SharedMutexImpl<false, void, std::atomic, false> >, (folly::detail::MutexLevel)1>::rlock() const ./folly/Synchronized.h:123
    #7 0x1c7530 in folly::RequestContext::setContext(std::shared_ptr<folly::RequestContext>) ./folly/io/async/Request.cpp:243
    #8 0x34b99 in ~RequestContextScopeGuard ./folly/io/async/Request.h:278

  Mutex M157 previously acquired by the same thread here:
    #0 0x48f18c in AnnotateRWLockAcquired ??:?
    #1 0x315495 in folly::SharedMutexImpl<false, void, std::atomic, false>::lock_shared() ./folly/SharedMutex.h:385
    #2 0x1b0eb8 in folly::detail::LockTraitsImpl<folly::SharedMutexImpl<false, void, std::atomic, false>, (folly::detail::MutexLevel)1, false>::lock_shared(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:157
    #3 0x1b0e68 in std::integral_constant<bool, true> folly::LockPolicyShared::lock<folly::SharedMutexImpl<false, void, std::atomic, false> >(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:493
    #4 0x1d63a9 in LockedPtrBase ./folly/Synchronized.h:1026
    #5 0x1d6258 in LockedPtr ./folly/Synchronized.h:1326
    #6 0x1c9a17 in folly::SynchronizedBase<folly::Synchronized<folly::RequestContext::State, folly::SharedMutexImpl<false, void, std::atomic, false> >, (folly::detail::MutexLevel)1>::rlock() const ./folly/Synchronized.h:123
    #7 0x1c7516 in folly::RequestContext::setContext(std::shared_ptr<folly::RequestContext>) ./folly/io/async/Request.cpp:242
    #8 0x34b99 in ~RequestContextScopeGuard ./folly/io/async/Request.h:278

```

Reviewed By: igorsugak

Differential Revision: D9797715

fbshipit-source-id: a0f991ae0ec8f3e9d33af6e05db49eceaf3a5174

Summary:
There is a lock inversion between VirtualEventBase and the fibers library:

1. During program shutdown, `GlobalCache::getImpl` acquires its `mutex_`, then it calls `VirtualEventBase::runOnDestruction` which acquires the SharedMutex on `onDestructionCallbacks_`
2. During normal program execution, `VirtualEventBase::destroyImpl()` acquires the `onDestructionCallbacks_` lock and then invokes the user-supplied callbacks, which will invoke `GlobalCache::eraseImpl` and acquire `mutex_`

The reason for this lock inversion is because in step 2, we are holding a mutex while invoking user-supplied callbacks.
The fix is to NOT hold the lock while we invoke these callbacks.

Reviewed By: andriigrynenko

Differential Revision: D9797816

fbshipit-source-id: 861a915f22138f0c8d3f8bca4d7bf4b9a0aa3d26

Summary:
When running tests with a TSAN-annotated version of folly::SharedMutex,
I discovered at least 3 types of lock inversions in folly::Singleton code.
These inversions are probably benign and deadlock is not possible, but we need to
suppress these inversions to prevent too much noise from TSAN.

Lock inversions:

1. The users can supply arbitrary create() functions for Singletons. If the create() function
   happens to acquire a mutex, then this introduces a lock ordering from
   SingletonVault -> mutex. Later on, when the user code runs in normal operations, we might
   hold the mutex and try to create a singleton, resulting in the mutex -> SingletonVault ordering
2. SingletonVault::destroyInstances() holds a mutex, and then invokes the user-supplied destroy()
   functions. In the user supplied destroy functions, we might attempt to create a singleton and
   acquire the lock in SingletonHolder, resulting in SingletonVault -> SingletonHolder lock ordering.
   In normal operations, whenever a singleton is created, we have the SingletonHolder -> SingletonVault
   lock ordering in createInstance()
3. doEagerInit() holds the eagerInitSingleton mutex and then acquires the SingletonVault lock
   during createInstance(). addEagerInitSingleton() holds the SingletonVault lock and then acquires
   the eagerInitSingleton lock.

The source of these errors is that we are invoking user-supplied callbacks while holding mutexes.
However, these lock inversions cannot actually deadlock based on higher-level knowledge of folly::Singleton
state transitions.

To suppress these errors, use the new `folly::SharedMutexSuppressTSAN` mutex instead, which will
not be annotated by TSAN. If we are not building with TSAN, then this mutex is equivalent to a normal
`folly::SharedMutex`.

Reviewed By: andriigrynenko

Differential Revision: D9797988

fbshipit-source-id: 82b5850fe189f7d1dcaca0e3562fabcfafd0cef5

Summary:
This diff adds TSAN annotations to `folly::SharedMutex` to allow TSAN to find
lock inversions with this type of mutex. Note that only the WritePriority version
is annotated, and the ReadPriority version is not. See the comments in the source code
for an explanation of this.

Some notes about how the annotation was done:

- We always call the RELEASED annotations at the beginning of unlock(), and the ACQUIRED annotations at the end of lock().
  This prevents a double locking interleaving where thread 1 has unlocked it and before RELEASED is called, another thread
  has locked it and calls ACQUIRED first.
- These annotations treat upgrade locks as equivalent to a shared lock. This prevents any false positives, and keeps
  the annotations simple.
- We need to keep the constructor for SharedMutex as `constexpr` to avoid static initialization bugs. As a result, we need
  to lazily annotate creation of the mutex. To do this, this adds an extra bit to the `SharedMutex` state to keep track if
  initialization has been called. In TSAN builds, we have an array of mutexes to enforce that initialization is called
  at most once.
- This diff introduces a new template param AnnotateForThreadSanitizer (default is False). This allows users to use a new
  folly::SharedMutexSuppressTSAN version to avoid lock inversion detection if there are noisy lock inversions.
  However, this should be the exception and not the norm. In normal build modes, this is equivalent to a normal SharedMutex.

Reviewed By: nbronson

Differential Revision: D9677058

fbshipit-source-id: b0f5719a75024937fb81672435fb1c9802f255d7

Summary:
This change is part of a bigger effort to make `folly::SharedMutex`
compatible with TSAN so that TSAN can find lock inversions with these
types of mutexes.

In order to do this, we need to annotate the mutex function calls
accordingly. The annotation functions are provided by the TSAN
runtime library.

Upcoming changes to `folly::SharedMutex`:

1. provide annotation helper macros (this change)
2. fix or suppress known issues in existing libraries (these macros will allow us to suppress issues)
3. once all the issues are fixed, annotate `SharedMutex` accordingly

Reviewed By: nbronson

Differential Revision: D9797582

fbshipit-source-id: 12306db98505fe31a8d8590113d10585840fbe6d

Summary:
`AccessSpreader::cachedCurrent()` caches the result of `AccessSpreader::getcpuFunc()` for 32 calls in a thread-local. The cached function takes 2 ns, where the current call takes 12 ns. This comes at the cost of being imprecise when threads migrate to a new cpu.

I chose 32 as the number of calls because it only has a 10% overhead over never refreshing (2.05 ns vs 1.83 ns), where 16 has a 30% overhead, and it performs just as well as 64.

Reviewed By: ot

Differential Revision: D10151009

fbshipit-source-id: 07ed292dfdcdcedcb74c24279f7773a80ad09348

Summary:
* The atomic load was being optimized out of the benchmark.
* The initializer wasn't being called for the test tags, so `AccessSpreader<ThreadLocalTag>::current()` and `AccessSpreader<PthreadSelfTag>::current()` were always returning 0.

Reviewed By: ot

Differential Revision: D10154155

fbshipit-source-id: 76be9865c9ee591d24aedc64e6faf4fd24350282

Summary: ALPN is the preferred way to do this.  We should only use that and not provide NPN support anymore.

Reviewed By: knekritz

Differential Revision: D10110626

fbshipit-source-id: 5ec91fe51252bed0a8d8047e481df76a395e1286

Summary:
Two things
- Use thenValue instead of then in one specific occurrence as later one has been deprecated
- Build googletest on `release-1.8.1` as it is the one that compiles with c++11

Reviewed By: cenzhao

Differential Revision: D10178139

fbshipit-source-id: 069578716e5e55ef0a25ea87e9ef5ceded1ff9d6

Summary: There's no need for this to be inheritable

Reviewed By: yfeldblum

Differential Revision: D10139559

fbshipit-source-id: ae2db0d13e33c383fd73c0563b127e62e54a7094

Summary: Add IOBuf::takeOwnershipIov method

Reviewed By: yfeldblum, djwatson

Differential Revision: D10135962

fbshipit-source-id: 86a22d73aa1b72645640cf026d2dc2f38b02989c

Summary:
Use heterogenous lookup for dynamic::getDefault for
StringPiece/dynamic keys.

Reviewed By: shixiao

Differential Revision: D10127452

fbshipit-source-id: 3cf28a28373211428ac5352136cf073e6c377b16

Summary:
dynamic::setDefault already calls heterogenous `emplace(...)`, however it does not behave correctly for static constexpr strings.
Add a specialization for StringPiece which solves this.

The current set of tests does not invoke setDefault with an object as a key.
This diffs adds a test that does that.

Reviewed By: shixiao

Differential Revision: D10007262

fbshipit-source-id: 386a44bf6b0a24f81e843695d177d85b280b907d

Summary: Document that insert may evaluate its arguments after rehashing.

Reviewed By: shixiao

Differential Revision: D10132409

fbshipit-source-id: ba5996a309ae201ae08808272edd1fb0b981cc64

Summary: Consume `folly::dynamic` and if it matches JSON patch format, build a sequence of patch operations. Application is to be done separately, e.g. by a method of `folly::dynamic` by walking the operations vector (obtained after parsing the patch) and applying operations to the document. This mimics the approach we used previously with `folly::json_pointer`.

Reviewed By: yfeldblum

Differential Revision: D10017234

fbshipit-source-id: 1d0b9f0967e528b81abf7998c88abdaf70a2fce8

Summary: There are no other methods, and icc gets confused otherwise

Reviewed By: yfeldblum

Differential Revision: D10116562

fbshipit-source-id: 7664501b910e5857e32337d3583dcc948caddd5d

Summary: icc doesn't support it.

Reviewed By: yfeldblum

Differential Revision: D10116547

fbshipit-source-id: 1ed522bd373ab30507f713d855ef02027ce90a91

Summary:
Having Indestructible field in SettingsCore makes it leak memory when
threads die.

Reviewed By: yfeldblum

Differential Revision: D10118572

fbshipit-source-id: 68696983128049525018a911b03cb7ea8add53ab

Summary: [Folly] Declare deleted `new` and `delete` operators in pairs in `ScopeGuard`.

Reviewed By: Orvid

Differential Revision: D10119848

fbshipit-source-id: 3588ed299248efd124d216bfaf5e472795f6a7aa

Summary: [Folly] Closer guarantee of `constexpr` Fingerprint tables computation. Works around a limitation that `template <...> constexpr` functions are only sometimes `constexpr` and may be non-`constexpr` for some instantiations with no warning by enforcing `constexpr` for the instantiations used.

Differential Revision: D10116556

fbshipit-source-id: f7014e5b2fce411c27c331874a7fd33841e14df3

Summary: This fixes a linker issue

Reviewed By: tuomaspelkonen

Differential Revision: D10109866

fbshipit-source-id: bf4193eafc8eb6f75b84241ff6e2ef89ef866642

Summary: SSL_CTX_new can fail if OpenSSL is not initialized.  This ensures it is.

Reviewed By: bolekk

Differential Revision: D10105302

fbshipit-source-id: 4da40d4714747b0e6a359a52fa594fc29bb93c0a

Summary:
Current thread was previously blocked waiting on the Future to be completed anyway, so there's no reason to not treat it like one of the ObserverManager threads (if it isn't one already) and run the computation inline.
Without this change nested makeObserver calls could result in a deadlock if recursion depth was greater than the number of ObserverManager threads.

Reviewed By: yfeldblum

Differential Revision: D10097363

fbshipit-source-id: eb781ed5aea0ad9415da40fbc7f1c15ed481e92f

Summary:
This allows a caller to throw away unwanted parts e.g.

```
std::string user;
folly::split('/', "ads_lla/admarket/adfox", std::ignore, user, std::ignore);

```

Reviewed By: yfeldblum

Differential Revision: D10057247

fbshipit-source-id: 8198d4cb207bbd2007635db1e5796759dccf367c

Summary:
SettingsMetadata provides a type-erased view, so the default value is stored as a string representation.
This method allows getting the actual original default value.

Differential Revision: D10077879

fbshipit-source-id: 2c6bf15eba57da5cb33ad1c657bc9bd51bcd2946

Summary: This forces users to take a Snapshot and makes global settings updates both atomic and more explicit.

Differential Revision: D10055254

fbshipit-source-id: 73ce34acdadad19d5b75e74ad26f13dc7763228d

Summary:
See the unit test for example usage.

```
folly::settings::Snapshot snapshot;

// Now changes to `FOLLY_SETTING(project, name)` and `snapshot(FOLLY_SETTING(project, name))`
// are independent from each other.
// Any updates to the settings in the snapshot can be published via snapshot.publish()
```

Basic version with a global lock, there are ways to optimize this later.

Differential Revision: D9926070

fbshipit-source-id: 66ee7f28539c242a08b688eab4a18e4a42a8b0ff

Summary: [Folly] Various `noexcept` specifiers in `IOBuf`.

Reviewed By: Orvid

Differential Revision: D10081705

fbshipit-source-id: 5516e7deca4edd50ea99e61e10cd413f3c0f17ae

Summary: [Folly] Compute fingerprint tables at constexpr time, v.s. as a separate build step which generates a source file.

Reviewed By: Orvid

Differential Revision: D10081043

fbshipit-source-id: f01f7587aafd29fc1c44515730b759325e1566d8

Summary: Currently for `ElfFile::openNoThrow`, if `init` fails (due to mal-formatted content etc.) it won't unmap and close the file. They will only be unmapped / closed on destruction. This Diff calls `reset()` on that case.

Reviewed By: yfeldblum

Differential Revision: D9997690

fbshipit-source-id: c30907fb793c813a5ad40d249919dd1752a2d711

Summary:
The signal handler might use nearly 9KB stack for symbolization. That would
cause stack overflow if the signal handler is running on a small alternative
stack set by sigaltstack. We have seen that in Rust programs. The Rust stdlib
sets up [8KB](https://github.com/rust-lang/libc/blob/4bd52f5e3ef5d6c8abdd1fbaf7c9975800a0dc93/src/unix/notbsd/linux/other/b64/x86_64.rs#L505) stacks [using sigaltstack](https://github.com/rust-lang/rust/blob/55b54a999bcdb0b1c1f42b6e1ae670beb0717086/src/libstd/sys/unix/stack_overflow.rs#L173) for all threads created by Rust on x64 Linux.

Talked with jsgf, we'd still like to get a nice symbolized stack trace if
segfault happens in a binary including both Rust and C++ code, since it's
more likely the C++ part being wrong, so C++ symbol demangling is more
important than Rust demangling.

This diff detects the small sigaltstack case, then allocates larger stack for
symbolization. Note the stack manipulation logic is not signal-safe, so it's
not used for non-small-sigaltstack cases. Programs not using sigaltstack, or
use sigaltstack with >9KB stacks won't be affected.

As we're here, teach the signal handler to save and restore errno. Also add
a note about golang's SA_ONSTACK requirement.

Reviewed By: luciang

Differential Revision: D9846902

fbshipit-source-id: f0dc81b1c1249d8a724e112906e0ff0b14c14ea9

Summary: This demonstrates a deadlock that's possible when using InlineTask.

Reviewed By: lbrandy

Differential Revision: D9883432

fbshipit-source-id: 45bb14181f6cfec41851315c0369232a30ba13d9