Summary: Avoid SIGSEGV if getIOExecutor() returns null

Reviewed By: yfeldblum

Differential Revision: D16859157

fbshipit-source-id: 00c7a45677c61d84a10486f11b4f792093ac2ff5

Summary:
It's spammy and is just informational.

Created from Diffusion's 'Open in Editor' feature.

Reviewed By: yfeldblum

Differential Revision: D16886741

fbshipit-source-id: 29aecb8a62cac9a60cbc4cabc596fd1ee00cfc04

Summary:
[Folly] Cut IsRelocatable markings of containers, removing forward declarations of the containers to avoid over-inclusion and to avoid undefined behavior.

> 16.5.4.2.1 [namespace.std]\1
> Unless otherwise specified, the behavior of a C++ program is undefined if it adds declarations or definitions to namespace std or to a namespace within namespace std.

Closes #1203.
Closes #1204.

Reviewed By: igorsugak

Differential Revision: D16855473

fbshipit-source-id: 0e6acc4f589707aa79b0001b415bcadfab652d62

Summary: [Folly] Revise `atomic_fetch_set`, `atomic_fetch_reset` comments, enforcing structure and switching comment style.

Reviewed By: aary

Differential Revision: D16877442

fbshipit-source-id: 0c4c163f2daa14616f20c4e0cac1002c83ec5fac

Summary:
http://man7.org/linux/man-pages/man2/mlock.2.html

       MLOCK_ONFAULT
              Lock pages that are currently resident and mark the entire
              range so that the remaining nonresident pages are locked when
              they are populated by a page fault.

The kernel still checks a few things for `MLOCK_ONFAULT`: rlimits RLIMIT_MEMLOCK / CAP_IPC_LOCK => EPERM or ENOMEM

`TRY_LOCK` & `MUST_LOCK` are only different in in their handling of EPERM, ENOMEN: whether to fail or ignore the errors, so I added a separate enum.

https://github.com/torvalds/linux/blob/v5.0/mm/mlock.c#L29-L37
```
bool can_do_mlock(void)
{
	if (rlimit(RLIMIT_MEMLOCK) != 0)
		return true;
	if (capable(CAP_IPC_LOCK))
		return true;
	return false;
}
EXPORT_SYMBOL(can_do_mlock);
```

https://github.com/torvalds/linux/blob/v5.0/mm/mlock.c#L671-L714
```
static __must_check int do_mlock(unsigned long start, size_t len, vm_flags_t flags)
{
	unsigned long locked;
	unsigned long lock_limit;
	int error = -ENOMEM;

	if (!can_do_mlock())
		return -EPERM;

	len = PAGE_ALIGN(len + (offset_in_page(start)));
	start &= PAGE_MASK;

	lock_limit = rlimit(RLIMIT_MEMLOCK);
	lock_limit >>= PAGE_SHIFT;
	locked = len >> PAGE_SHIFT;

	if (down_write_killable(&current->mm->mmap_sem))
		return -EINTR;

	locked += current->mm->locked_vm;
	if ((locked > lock_limit) && (!capable(CAP_IPC_LOCK))) {
		/*
		 * It is possible that the regions requested intersect with
		 * previously mlocked areas, that part area in "mm->locked_vm"
		 * should not be counted to new mlock increment count. So check
		 * and adjust locked count if necessary.
		 */
		locked -= count_mm_mlocked_page_nr(current->mm,
				start, len);
	}

	/* check against resource limits */
	if ((locked <= lock_limit) || capable(CAP_IPC_LOCK))
		error = apply_vma_lock_flags(start, len, flags);

	up_write(&current->mm->mmap_sem);
	if (error)
		return error;

	error = __mm_populate(start, len, 0);
	if (error)
		return __mlock_posix_error_return(error);
	return 0;
}
```

https://github.com/torvalds/linux/blob/v5.0/mm/mlock.c#L721-L732
```
SYSCALL_DEFINE3(mlock2, unsigned long, start, size_t, len, int, flags)
{
	vm_flags_t vm_flags = VM_LOCKED;

	if (flags & ~MLOCK_ONFAULT)
		return -EINVAL;

	if (flags & MLOCK_ONFAULT)
		vm_flags |= VM_LOCKONFAULT;

	return do_mlock(start, len, vm_flags);
}

```

Reviewed By: yfeldblum

Differential Revision: D16760748

fbshipit-source-id: 54937046f5faaceac2cfb9dd6a0f207ad653fe6b

Summary:
Currently the timekeeper callback set by the within method isn't cleaned up when the future resolves and always waits the full duration. This can cause OOM under high QPS and long timeout situations.

This resolves this by cancelling the callback.

Reviewed By: mpark

Differential Revision: D16835816

fbshipit-source-id: c400afc37dbc66447aaa8fd557a770d717d9ab02

Summary:
RocksDB's TSAN version was not able to instrument atomic_fetch_set and
atomic_fetch_reset; and as a result the DistributedMutex tests were failing
under TSAN

Also TSAN seems to have some issue when and exception_ptr is placed into an
aligned_storage_t instance. This verifiably correct program (I think) also fails
with the same TSAN error P78696506. So
disable that test

This is the code with the same false-negative
```
namespace {
class ExceptionWithConstructionTrack : public std::exception {
 public:
  explicit ExceptionWithConstructionTrack(int id)
      : id_{folly::to<std::string>(id)}, constructionTrack_{id} {}

  const char* what() const noexcept override {
    return id_.c_str();
  }

 private:
  std::string id_;
  TestConstruction constructionTrack_;
};

template <typename Storage, typename Atomic>
void transferCurrentException(Storage& storage, Atomic& produced) {
  assert(std::current_exception());
  new (&storage) std::exception_ptr(std::current_exception());
  produced->store(true, std::memory_order_release);
}

void concurrentExceptionPropagationStress(
    int numThreads,
    std::chrono::milliseconds milliseconds) {
  auto&& stop = std::atomic<bool>{false};
  auto&& exceptions = std::vector<std::aligned_storage<48, 8>::type>{};
  auto&& produced = std::vector<std::unique_ptr<std::atomic<bool>>>{};
  auto&& consumed = std::vector<std::unique_ptr<std::atomic<bool>>>{};
  auto&& consumers = std::vector<std::thread>{};
  for (auto i = 0; i < numThreads; ++i) {
    produced.emplace_back(new std::atomic<bool>{false});
    consumed.emplace_back(new std::atomic<bool>{false});
    exceptions.push_back({});
  }

  auto producer = std::thread{[&]() {
    auto counter = std::vector<int>(numThreads, 0);
    for (auto i = 0; true; i = ((i + 1) % numThreads)) {
      try {
        throw ExceptionWithConstructionTrack{counter.at(i)++};
      } catch (...) {
        transferCurrentException(exceptions.at(i), produced.at(i));
      }

      while (!consumed.at(i)->load(std::memory_order_acquire)) {
        if (stop.load(std::memory_order_acquire)) {
          return;
        }
      }

      consumed.at(i)->store(false, std::memory_order_release);
    }
  }};

  for (auto i = 0; i < numThreads; ++i) {
    consumers.emplace_back([&, i]() {
      auto counter = 0;
      while (true) {
        while (!produced.at(i)->load(std::memory_order_acquire)) {
          if (stop.load(std::memory_order_acquire)) {
            return;
          }
        }
        produced.at(i)->store(false, std::memory_order_release);

        try {
          auto storage = &exceptions.at(i);
          auto exc = folly::launder(
            reinterpret_cast<std::exception_ptr*>(storage));
          auto copy = std::move(*exc);
          exc->std::exception_ptr::~exception_ptr();
          std::rethrow_exception(std::move(copy));
        } catch (std::exception& exc) {
          auto value = std::stoi(exc.what());
          EXPECT_EQ(value, counter++);
        }

        consumed.at(i)->store(true, std::memory_order_release);
      }
    });
  }

  std::this_thread::sleep_for(milliseconds);
  stop.store(true);
  producer.join();
  for (auto& thread : consumers) {
    thread.join();
  }
}
} // namespace
```

Reviewed By: yfeldblum

Differential Revision: D16739396

fbshipit-source-id: 0eba191ffc07d2e64a3ea550c5fe9a7d01ce2f18

Summary: As title

Reviewed By: yfeldblum

Differential Revision: D16821444

fbshipit-source-id: 14f0655caf0f53ef47fa65b9b6ba57f0f190df12

Summary: [Folly] `#define FOLLY_ERASE FOLLY_ALWAYS_INLINE FOLLY_ATTR_VISIBILITY_HIDDEN`.

Reviewed By: vitaut

Differential Revision: D16686469

fbshipit-source-id: cedaee61955a40ae28ad2b0da565fc9fb385bc6e

Summary:
Update the generated `run_cmake.py` script to allow the caller to specify that
they just want to run a build without the install step.

Reviewed By: wez

Differential Revision: D16778007

fbshipit-source-id: 1859aca2b80fa7b099b4790682a6508e0185f2a0

Summary:
This cleans up how the `CMAKE_ENV` and `CMAKE_DEFINE_ARGS` variables are
written in the generated `run_cmake.py` script that we emit for CMake-based
projects.

We now emit each entry in these variables on separate lines, just to improve
readability.  (Both of these variables tend to have a number of entries and
are very long if emitted on a single line.)

This also replaces the `-DCMAKE_INSTALL_PREFIX` entry in `CMAKE_DEFINE_ARGS`
to have it correctly honor the `INSTALL_DIR` variable defined in
`run_cmake.py`.  This makes `run_cmake.py` still do the right thing if someone
manually edits it to change the `INSTALL_DIR` value.

Reviewed By: wez

Differential Revision: D16778006

fbshipit-source-id: fee5d25748b87b5d9c57ee2edf8de5e586e872ee

Summary:
Add arguments to getdeps.py to allow overriding the source, build, and install
directories a per-project basis.  The arguments take the form `[PROJECT:]PATH`
If the `PROJECT` portion is omitted, it defaults to the current project being
built.

In particular this makes it possible to specify `--src-dir .` to tell
getdeps.py to find the project sources from the current directory rather than
downloading them.

Reviewed By: wez

Differential Revision: D16778011

fbshipit-source-id: f33b87213ace04abb66334f588babdf59df91964

Summary:
This makes getdeps.py no longer crash if used in the folly repository.
Folly does not depend on any other Facebook projects, so it does not
include a `build/deps` directory.

Reviewed By: wez

Differential Revision: D16778009

fbshipit-source-id: 41be53d862f41b62154b081eb90ddba8742658fe

Summary:
Most of the getdeps subcommands operate on a single project, and some of the
argument parsing and initial logic to load the project is largely the same.
This consolidates that logic into a base class so that we can share this code
across subcommands, instead of repeating it.

This also unifies the behavior so that by default all commands enable tests on
the specified project, and disable test on dependent projects.  Making sure
all commands use the same behavior here is important as whether are not tests
are enabled can affect the project configuration, and therefore affect its
getdeps hash.

Reviewed By: wez

Differential Revision: D16778010

fbshipit-source-id: 044f99ad6cdd4a56f843276cec8ead786249ee7a

Reviewed By: yfeldblum

Differential Revision: D16785914

fbshipit-source-id: 73ade28f7aa300ce5e278e2ff2f56f4aafef9533

Summary: As title

Reviewed By: yfeldblum

Differential Revision: D16808375

fbshipit-source-id: 0efbb64c0ad9403f4efb1e2a885b21d476ff7cac

Summary: Useful to disable this on some platforms.

Reviewed By: yfeldblum

Differential Revision: D16153556

fbshipit-source-id: ab5916c929f5987dad7daa0c66b229eaf7404c88

Summary:
- Add allocator-extended copy constructors
- Add allocator-extended move constructors
- Add allocator-extended initializer-list constructor
- Add get_allocator accessor
- Make emplace/emplace_hint use the allocator
- pmr tests catch allocations that "escape" by setting the default resource to null

Reviewed By: nbronson

Differential Revision: D16623448

fbshipit-source-id: c10682f98d1c234fcad95608643b2f3ccea64a41

Summary: This function should be marked const.

Reviewed By: yfeldblum

Differential Revision: D16793980

fbshipit-source-id: 3e98f999c3e00caa0d238fa29c082ab6e819bf2a

Summary:
Fix the BuildOptions class to correctly honor the `host_type` parameter that
it was constructed with when constructing the manifest evaluation context.
I accidentally broke this behavior in D16477396, and incorrectly had this code
path default to using the current host system rather than the value passed in
from the command line.

Reviewed By: wez

Differential Revision: D16779579

fbshipit-source-id: de911daaa643f6303fd35149775ab25d3f64d34f

Summary:
This adds standalone tests for the heterogeneous key features
of callWithExtractedKey and callWithConstructedKey.  These code
(metacode?) paths were already tested somewhat by F14MapTest and
F14SetTest.

Reviewed By: yfeldblum

Differential Revision: D16742421

fbshipit-source-id: 1f7e5065cad2216b3db4d4cb33fc73ac72d43c06

Summary:
Fixes compilation on non x86.
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Pull Request resolved: https://github.com/facebook/folly/pull/1202

Differential Revision: D16760427

Pulled By: yfeldblum

fbshipit-source-id: 6f31d5a17b9fe9c1f91d884eb94967ba06756f3e

Summary:
This diff fixes the initial conditions for SwapTrackingAlloc,
which fixes F14 tests on platforms that do not have intrinsics enabled.

Differential Revision: D16755857

fbshipit-source-id: a76d8093100cbc0c5ce8723daa0976673d942299

Summary:
This diff extracts the key destructuring logic in F14's emplace
functions into standalone helpers that can be used by other container
types (like sorted_vector_{set,map}).  It also adds better PMR compliance, by
using a stateful allocator to construct a temporary if that is required
to be able to perform the search.

This diff also cleans up some of the testing code in folly/container/test
that had issues if included in multiple places.

Reviewed By: yfeldblum

Differential Revision: D16655131

fbshipit-source-id: 5a6f57ac346d1fd5928e0f737110760b0c518974

Summary:
In response to review feedback for D16477400 and D16477401, update
`ManifestLoader.load_all_manifests()` to only update its data for projects
that have not previously been loaded.  This helps ensure that code using a
single `ManifestLoader` object cannot have two in-memory `Manifest` objects
for the same project, and that existing data (such as project hashes) can't be
invalidated if a manifest is later loaded from updated on-disk data.

Reviewed By: pkaush

Differential Revision: D16586682

fbshipit-source-id: 50b1979ec55f2ad6901629cd852293a8f6ca903f

Summary:
This is a missing piece from https://github.com/facebook/folly/pull/1178, but more nasty as it can cause runtime rather than compile-time failures.

Given an interface `ICat` with `void meow() const`, if we have `struct cat` with `void meow() const noexcept`, then constructing a `folly::Poly<ICat const&>` for an instance of `cat` will have a null pointer in the vtable entry for the `meow` method.

The culprit seems to be in the template specialization for [`ThunkFn` ](https://github.com/cstratopoulos/folly/blob/feature/poly-update/folly/detail/PolyDetail.h#L539) which may be dispatched on `IsConstMember`.

I've added a fairly trivial test case to `PolyTest.cpp`; but notably it is the only instance of _any_ test case in that file having a `noexcept` method. Without the change in `PolyDetail.h`, the call to `cref->meow()` will cause a null pointer dereference. If something more explicit is desired, maybe we could `EXPECT` that the corresponding vtable entry is non-null, but that doesn't exactly fit the pattern of other test cases in that file.

(Aside: In the linked PR I used `__cpp_noexcept_function_type` directly, but I have since realized that a macro for this is already present in [`Portability.h`](https://github.com/facebook/folly/blob/master/folly/Portability.h#L503) so I'm using that one now)
Pull Request resolved: https://github.com/facebook/folly/pull/1201

Reviewed By: ericniebler

Differential Revision: D16686139

Pulled By: yfeldblum

fbshipit-source-id: 73c7ac132d6e772ffd49f450b5fcd0f8a0dbdb26

Summary:
Added boundary support in folly/io/Cursor.h.

The bounded Cursor can have a boundary that is shorter than the underlying IOBuf. This gives a custom view of the IOBuf.

Reviewed By: spalamarchuk

Differential Revision: D16347838

fbshipit-source-id: 16d91e42e2acee788b31bc2a9b6cc8716b1d22f2

Summary: Switch defer to use inline continuations by default, if bound executors match.

Reviewed By: andriigrynenko

Differential Revision: D16643666

fbshipit-source-id: d3cdf86761c378d7ca138b46322c55a9755bdc48

Summary: As both coro.h and futures.h defines the function getExecutor, redefinition error occurs when both of them are included. This change moves the getExecutor inline function to executor.h so it can be shared.

Reviewed By: yfeldblum

Differential Revision: D16676456

fbshipit-source-id: 33f7558a3d5dbc996c4efef4fe90918de32ad8b2

Summary:
When inserting an item with a hint, the item will be always copied, even
if it is already present in the map. This could cause an exception to be
thrown when it doesn't need to be. The standard doesn't require this, but
it is an optimization anyways.

Reviewed By: yfeldblum

Differential Revision: D16585971

fbshipit-source-id: f0a8eeb0bd86f07ef110f91cbef8ab422ff55480

Summary:
The current folly ExceptionTracer only keeps the stack traces in thread local storage for as long as the exception is currently active or caught. When using folly futures we almost never handle exceptions within a catch block or even the same thread with thenError.

With C++ 11, we have exception_ptr which ref counts the exception so being able to keep the stack trace as long as something still holds a pointer to the exception is very useful. This allows us to get a stack trace for an exception returned by futures.

This tracer adds a callback that runs when an exception is thrown and captures the stack trace in a synchronized map. To manage the lifetime, we override the deleter function with our own wrapper that will remove the trace from the map when the exception is no longer needed.

Two things of note:
* We're attempting to allocate memory in the exception thrown callback so if we run out of memory this code won't be able to handle it and trigger a terminate via the noexcept.
* Every exception thrown requires acquiring a lock. We should be able to mitigate this via folly::ConcurrentHashMap if it becomes a problem. Initial benchmarking seems fine as is. I don't believe it'll become a problem since we only throw exceptions in exceptional cases and they shouldn't be performance sensitive.

Reviewed By: yfeldblum

Differential Revision: D16533296

fbshipit-source-id: 7d8832c24c79bde98d1bb213d4b70f259fb3d4bc

Summary: We can't use futures::WaitExecutor here, because it may drop some work if detached, which is not supported by coro::Task.

Reviewed By: lewissbaker

Differential Revision: D16664921

fbshipit-source-id: 9797b2c59af156ec96e64244d5810a5924cfb3cc

Summary: More broadly compatible implementation of variant in future core using a boost variant.

Reviewed By: andriigrynenko

Differential Revision: D16676402

fbshipit-source-id: 19784ab1815e81b9afc49e46b5425e55111cd530

Summary: Add PicoSpinLock TSAN annotations

Reviewed By: yfeldblum

Differential Revision: D16653917

fbshipit-source-id: d388a31824f1278fe79ab4a42e21a66781b879df

Summary: [Folly] Cut enforcement of copy-ctor counts in sorted-vector sets/maps tests, since whether the copy-ctor is elidable, and therefore the actual values of the counts, depends on the compiler and on the optimizations in use.

Reviewed By: nbronson

Differential Revision: D16638747

fbshipit-source-id: b7a03a46afc065bd93595026df1aab46c9e1295e

To reduce risk of stack overflow when catching up with deferred work, make blocking operation runOnMainContext

Summary: Shift wait execution off of a (potential) fiber stack to reduce risk of stack overflows while running deferred work.

Reviewed By: andriigrynenko

Differential Revision: D16643654

fbshipit-source-id: 3ab1fe762b8f93c011cdb926e7689281018f54bc

Reviewed By: yfeldblum

Differential Revision: D16639497

fbshipit-source-id: ddefeb12e2f83bc86e410092233efb2cdeaacbd0

Summary: This can be useful to avoid the cost of re-throwing exceptions.

Reviewed By: yfeldblum, LeeHowes

Differential Revision: D16628722

fbshipit-source-id: 434842d99369d0293c4eeb894e6feac15ec16173

Summary: Defer uses inline continuations by default, if bound executors match.

Reviewed By: andriigrynenko

Differential Revision: D15300706

fbshipit-source-id: fc17c70ed9956442007a1cc2f6a082f75c555ad4

Summary:
After this diff, DeferredExecutor participates consistently in executor inline behaviour by being special cased in the core.

DeferredExecutor is no longer an executor, and is hence no longer special cased in the Future code. This is replaced with a variant of DeferredExecutor and Executor in Core.

Reviewed By: yfeldblum, andriigrynenko

Differential Revision: D15836529

fbshipit-source-id: 8324ba1de57e85fc757ecc3b431bf71858868a0d