Summary:
The `folly::coro::Task` coroutine type now has an associated `CancellationToken` that is implicitly passed down to child operations that it awaits.

This is a first step towards supporting cancellation of persistent async streams.

This adds a new `co_withCancellation()` customisation point that awaitable types can customise to allow them to opt-in to cancellation. Currently only `Task` customises this operation.

Also provided a new `co_current_cancellation_token` object that can be awaited within a `Task` to retrieve the current `CancellationToken`.

Note that I have not yet hooked up the `Future` or `SemiFuture` to integrate with this cancellation mechanism (most `Future`-based code is not cancellation-aware). So a coroutine that is currently suspended awaiting for a `Future` may not currently respond to a cancellation request.

Reviewed By: andriigrynenko

Differential Revision: D16610810

fbshipit-source-id: 72a31d7a3ba4c281db54c1942ab41d4ea2f34d21

Summary: `LockFlags` is a set of flags that mirrors `mlock2(..., int flags)`, but since this is not a C API we don't need a bitset, we can just use a more idiomatic struct of bools (also used for the other options in the class). Furthermore, having an entry with value 0 (`LOCK_PREFAULT`) does not make sense in a bitset.

Reviewed By: yfeldblum

Differential Revision: D16900127

fbshipit-source-id: 6477854121c0626e0f36cb7a50d50df73323bd4e

Summary:
This simplifies things for the OSS CI situation in
the short term, especially since Eden isn't quite buildable
or usable outside FB at this time.

Reviewed By: chadaustin

Differential Revision: D16953346

fbshipit-source-id: 028150bb3cfc9a8ccbe1197321214b56ad359463

Summary: [Folly] Drop workaround for clang-3.6 in `Range` ctor.

Reviewed By: gkmhub, Orvid

Differential Revision: D16945307

fbshipit-source-id: d70d7560375d0c020b19fbac227767605459a834

Summary:
bzero is deprecated and can be compile time disabled for uClibc-ng
Pull Request resolved: https://github.com/facebook/folly/pull/1195

Reviewed By: Orvid

Differential Revision: D16451837

Pulled By: yfeldblum

fbshipit-source-id: 929a8e8cd7f8ddb3a904c1b1956a5f8ca115244f

Summary:
Simplified the interface for consuming an AsyncGenerator to now just have a single async next() method instead of an iterator-based API.

The iterator-based API was added to integrate with the 'for co_await' syntax present in the Coroutines TS, however this syntax has been removed from C++20 to allow for future exploration of the AsyncRanges design-space.

The `co_await gen.next()` expression produces an optional-like object that lets you query whether the result is a sentinel or contains a value.

Old:
```
Task<void> consume(AsyncGenerator<T> gen) {
  for (auto it = co_await gen.begin();
       it != gen.end();
       co_await ++it) {
    use(*it);
  }
}
```

New:
```
Task<void> consume(AsyncGenerator<T> gen) {
  while (auto item = co_await gen.next()) {
    use(*item);
  }
}
```

Reviewed By: andriigrynenko, kirkshoop

Differential Revision: D16586151

fbshipit-source-id: 4b0bf31ba9291d894a18e9553513eddee0cde33a

Reviewed By: Orvid

Differential Revision: D16640245

fbshipit-source-id: 3a184e4d877ea27a9400de5993b494095f6bd5e5

Summary:
does not affect the github build.
this is for fb CI

Reviewed By: dddmello

Differential Revision: D16935982

fbshipit-source-id: 61656fa600f19438b5aa89486486ed4ee103dea4

Summary:
folly::Hash did not work with pointer-like types like std::unique_ptr<T>,
std::shared_ptr<T> and T*.  Add specializations for those

Reviewed By: yfeldblum

Differential Revision: D16134842

fbshipit-source-id: 78d3cf9c6f5ef4f49b008c0d1c489d3c6203ba89

Summary: As far as I can tell, folly/test/FBVectorTestUtil.h does not need to include folly/Benchmark.h. Move it to folly/testFBVectorBenchmark where it is actually used.

Reviewed By: yfeldblum

Differential Revision: D16931137

fbshipit-source-id: 8ef5f99a783537672d55883adadd08270e318a61

Summary: Fix : iterating through ConcurrentHashMapSIMD results in infinite loop

Reviewed By: nbronson

Differential Revision: D16923826

fbshipit-source-id: c8b98ef13034240c162c155cf0e5c3f097b73c7f

Summary: Underflow/overflow of signed integers is fraught with dangers. So this diff removes the potential for overflow while adding or subtracting buckets in folly stats.

Reviewed By: yfeldblum

Differential Revision: D16898755

fbshipit-source-id: 4d3df1024b4c01ea0b276278105071e62e27fa27

Summary:
In the Facebook infrastructure the testpilot runner talks
to a service to determine whether a given test should be run locally.

The remote service was returning an error for some tests because they
didn't have enough configuration specified.

This diff associates some configuration with the tests based on the
host type and makes things happier.

Reviewed By: Ben0mega

Differential Revision: D16894682

fbshipit-source-id: b6f59b112991160ef26cc126e98041d8908a6273

Summary:
Sized deallocation via ::operator delete with size == 0 is
broken in some allocators.  This diff adds a unit test that ensures we
are either disabling sized deallocator or running with a good allocator.

Reviewed By: interwq

Differential Revision: D10857654

fbshipit-source-id: 981c7bcc1b021d4cd1ea1725ccf510d1958842fd

Reviewed By: pixelb

Differential Revision: D15301144

fbshipit-source-id: f94a805427b6fc8b20e7c01fcf6e1853fbebf7f1

Summary: Make `AsyncServerSocket` not fail on `EINPROGRESS` in `bind` and fix handling of `errno` there.

Reviewed By: yfeldblum

Differential Revision: D16791566

fbshipit-source-id: 1aeaefb6bae47bf41fd2d20238b716b84d0566d1

Summary: T52681488

Reviewed By: Orvid

Differential Revision: D16901429

fbshipit-source-id: 0a5fc218edf6ef29b100365163102dbdc2abf313

Summary:
[Folly] Suppress a spurious MSVC warning in `folly/Traits.h`.

```
folly\traits.h(645): warning C4287: '>': unsigned/negative constant mismatch
folly\traits.h(646): warning C4287: '<=': unsigned/negative constant mismatch
folly\traits.h(647): warning C4287: '<': unsigned/negative constant mismatch
```

Reviewed By: Orvid

Differential Revision: D16889045

fbshipit-source-id: 31edd58bf046143092a5f006ee0ac5b071b539a0

Summary:
[Folly] Fix MSVC conversion warning in `Optional` test.

```
folly\optional.h(372): warning C4244: 'argument': conversion from 'double' to 'int', possible loss of data
folly\optional.h(358): note: see reference to function template instantiation 'void folly::Optional<Type>::construct<std::initializer_list<_Ty>&,double>(std::initializer_list<_Ty> &,double &&)' being compiled
          with
          [
              _Ty=int
          ]
folly\optional.h(359): note: see reference to function template instantiation 'void folly::Optional<Type>::construct<std::initializer_list<_Ty>&,double>(std::initializer_list<_Ty> &,double &&)' being compiled
          with
          [
              _Ty=int
          ]
folly\optional.h(449): note: see reference to function template instantiation 'folly::Optional<Type>::Optional<std::initializer_list<_Ty>&,double>(folly::Optional<Type>::PrivateConstructor,std::initializer_list<_Ty> &,double &&) noexcept' being compiled
          with
          [
              _Ty=int
          ]
folly\optional.h(451): note: see reference to function template instantiation 'folly::Optional<Type>::Optional<std::initializer_list<_Ty>&,double>(folly::Optional<Type>::PrivateConstructor,std::initializer_list<_Ty> &,double &&) noexcept' being compiled
          with
          [
              _Ty=int
          ]
folly\test\optionaltest.cpp(659): note: see reference to function template instantiation 'folly::Optional<Type> folly::make_optional<Type,int,double>(std::initializer_list<_Ty>,double &&)' being compiled
          with
          [
              _Ty=int
          ]
```

Reviewed By: Orvid

Differential Revision: D16889085

fbshipit-source-id: c59f5e3fa0d4402e1b836841dff66d73b111235f

Summary:
Building a project that uses folly and `target_compile_features` may fail
because folly requires C++14 and CMake doesn't know about it. So if a project
has only C++11 features in `target_compile_features` CMake will pick
`-std=gnu++11` or similar and compilation will fail on an assertion in
`folly/Portability.h`. To fix this, add `target_compile_features` to the
`folly` target pushing the minimum required standard to C++14.

Reviewed By: Orvid

Differential Revision: D16884888

fbshipit-source-id: 0e5ac4abde10b1ea757011bcd05b17d508859b4f

Summary:
Add some CMake functions for building standalone executables from Python
source files.  This generates executables similar to PEX
(https://github.com/pantsbuild/pex).

In the future this could potentially be leveraged to directly build XAR files
(https://github.com/facebookincubator/xar).

The main advantages of these functions is that they allow easily defining
"libraries" of python files, and their dependencies, which can then be used
and packaged as part of multiple different standalone executables.

Reviewed By: wez

Differential Revision: D16722499

fbshipit-source-id: e1d829b911dc428e5438b5cf9cebf99b3fb6ce24

Summary: Avoid SIGSEGV if getIOExecutor() returns null

Reviewed By: yfeldblum

Differential Revision: D16859157

fbshipit-source-id: 00c7a45677c61d84a10486f11b4f792093ac2ff5

Summary:
It's spammy and is just informational.

Created from Diffusion's 'Open in Editor' feature.

Reviewed By: yfeldblum

Differential Revision: D16886741

fbshipit-source-id: 29aecb8a62cac9a60cbc4cabc596fd1ee00cfc04

Summary:
[Folly] Cut IsRelocatable markings of containers, removing forward declarations of the containers to avoid over-inclusion and to avoid undefined behavior.

> 16.5.4.2.1 [namespace.std]\1
> Unless otherwise specified, the behavior of a C++ program is undefined if it adds declarations or definitions to namespace std or to a namespace within namespace std.

Closes #1203.
Closes #1204.

Reviewed By: igorsugak

Differential Revision: D16855473

fbshipit-source-id: 0e6acc4f589707aa79b0001b415bcadfab652d62

Summary: [Folly] Revise `atomic_fetch_set`, `atomic_fetch_reset` comments, enforcing structure and switching comment style.

Reviewed By: aary

Differential Revision: D16877442

fbshipit-source-id: 0c4c163f2daa14616f20c4e0cac1002c83ec5fac

Summary:
http://man7.org/linux/man-pages/man2/mlock.2.html

       MLOCK_ONFAULT
              Lock pages that are currently resident and mark the entire
              range so that the remaining nonresident pages are locked when
              they are populated by a page fault.

The kernel still checks a few things for `MLOCK_ONFAULT`: rlimits RLIMIT_MEMLOCK / CAP_IPC_LOCK => EPERM or ENOMEM

`TRY_LOCK` & `MUST_LOCK` are only different in in their handling of EPERM, ENOMEN: whether to fail or ignore the errors, so I added a separate enum.

https://github.com/torvalds/linux/blob/v5.0/mm/mlock.c#L29-L37
```
bool can_do_mlock(void)
{
	if (rlimit(RLIMIT_MEMLOCK) != 0)
		return true;
	if (capable(CAP_IPC_LOCK))
		return true;
	return false;
}
EXPORT_SYMBOL(can_do_mlock);
```

https://github.com/torvalds/linux/blob/v5.0/mm/mlock.c#L671-L714
```
static __must_check int do_mlock(unsigned long start, size_t len, vm_flags_t flags)
{
	unsigned long locked;
	unsigned long lock_limit;
	int error = -ENOMEM;

	if (!can_do_mlock())
		return -EPERM;

	len = PAGE_ALIGN(len + (offset_in_page(start)));
	start &= PAGE_MASK;

	lock_limit = rlimit(RLIMIT_MEMLOCK);
	lock_limit >>= PAGE_SHIFT;
	locked = len >> PAGE_SHIFT;

	if (down_write_killable(&current->mm->mmap_sem))
		return -EINTR;

	locked += current->mm->locked_vm;
	if ((locked > lock_limit) && (!capable(CAP_IPC_LOCK))) {
		/*
		 * It is possible that the regions requested intersect with
		 * previously mlocked areas, that part area in "mm->locked_vm"
		 * should not be counted to new mlock increment count. So check
		 * and adjust locked count if necessary.
		 */
		locked -= count_mm_mlocked_page_nr(current->mm,
				start, len);
	}

	/* check against resource limits */
	if ((locked <= lock_limit) || capable(CAP_IPC_LOCK))
		error = apply_vma_lock_flags(start, len, flags);

	up_write(&current->mm->mmap_sem);
	if (error)
		return error;

	error = __mm_populate(start, len, 0);
	if (error)
		return __mlock_posix_error_return(error);
	return 0;
}
```

https://github.com/torvalds/linux/blob/v5.0/mm/mlock.c#L721-L732
```
SYSCALL_DEFINE3(mlock2, unsigned long, start, size_t, len, int, flags)
{
	vm_flags_t vm_flags = VM_LOCKED;

	if (flags & ~MLOCK_ONFAULT)
		return -EINVAL;

	if (flags & MLOCK_ONFAULT)
		vm_flags |= VM_LOCKONFAULT;

	return do_mlock(start, len, vm_flags);
}

```

Reviewed By: yfeldblum

Differential Revision: D16760748

fbshipit-source-id: 54937046f5faaceac2cfb9dd6a0f207ad653fe6b

Summary:
Currently the timekeeper callback set by the within method isn't cleaned up when the future resolves and always waits the full duration. This can cause OOM under high QPS and long timeout situations.

This resolves this by cancelling the callback.

Reviewed By: mpark

Differential Revision: D16835816

fbshipit-source-id: c400afc37dbc66447aaa8fd557a770d717d9ab02

Summary:
RocksDB's TSAN version was not able to instrument atomic_fetch_set and
atomic_fetch_reset; and as a result the DistributedMutex tests were failing
under TSAN

Also TSAN seems to have some issue when and exception_ptr is placed into an
aligned_storage_t instance. This verifiably correct program (I think) also fails
with the same TSAN error P78696506. So
disable that test

This is the code with the same false-negative
```
namespace {
class ExceptionWithConstructionTrack : public std::exception {
 public:
  explicit ExceptionWithConstructionTrack(int id)
      : id_{folly::to<std::string>(id)}, constructionTrack_{id} {}

  const char* what() const noexcept override {
    return id_.c_str();
  }

 private:
  std::string id_;
  TestConstruction constructionTrack_;
};

template <typename Storage, typename Atomic>
void transferCurrentException(Storage& storage, Atomic& produced) {
  assert(std::current_exception());
  new (&storage) std::exception_ptr(std::current_exception());
  produced->store(true, std::memory_order_release);
}

void concurrentExceptionPropagationStress(
    int numThreads,
    std::chrono::milliseconds milliseconds) {
  auto&& stop = std::atomic<bool>{false};
  auto&& exceptions = std::vector<std::aligned_storage<48, 8>::type>{};
  auto&& produced = std::vector<std::unique_ptr<std::atomic<bool>>>{};
  auto&& consumed = std::vector<std::unique_ptr<std::atomic<bool>>>{};
  auto&& consumers = std::vector<std::thread>{};
  for (auto i = 0; i < numThreads; ++i) {
    produced.emplace_back(new std::atomic<bool>{false});
    consumed.emplace_back(new std::atomic<bool>{false});
    exceptions.push_back({});
  }

  auto producer = std::thread{[&]() {
    auto counter = std::vector<int>(numThreads, 0);
    for (auto i = 0; true; i = ((i + 1) % numThreads)) {
      try {
        throw ExceptionWithConstructionTrack{counter.at(i)++};
      } catch (...) {
        transferCurrentException(exceptions.at(i), produced.at(i));
      }

      while (!consumed.at(i)->load(std::memory_order_acquire)) {
        if (stop.load(std::memory_order_acquire)) {
          return;
        }
      }

      consumed.at(i)->store(false, std::memory_order_release);
    }
  }};

  for (auto i = 0; i < numThreads; ++i) {
    consumers.emplace_back([&, i]() {
      auto counter = 0;
      while (true) {
        while (!produced.at(i)->load(std::memory_order_acquire)) {
          if (stop.load(std::memory_order_acquire)) {
            return;
          }
        }
        produced.at(i)->store(false, std::memory_order_release);

        try {
          auto storage = &exceptions.at(i);
          auto exc = folly::launder(
            reinterpret_cast<std::exception_ptr*>(storage));
          auto copy = std::move(*exc);
          exc->std::exception_ptr::~exception_ptr();
          std::rethrow_exception(std::move(copy));
        } catch (std::exception& exc) {
          auto value = std::stoi(exc.what());
          EXPECT_EQ(value, counter++);
        }

        consumed.at(i)->store(true, std::memory_order_release);
      }
    });
  }

  std::this_thread::sleep_for(milliseconds);
  stop.store(true);
  producer.join();
  for (auto& thread : consumers) {
    thread.join();
  }
}
} // namespace
```

Reviewed By: yfeldblum

Differential Revision: D16739396

fbshipit-source-id: 0eba191ffc07d2e64a3ea550c5fe9a7d01ce2f18

Summary: As title

Reviewed By: yfeldblum

Differential Revision: D16821444

fbshipit-source-id: 14f0655caf0f53ef47fa65b9b6ba57f0f190df12

Summary: [Folly] `#define FOLLY_ERASE FOLLY_ALWAYS_INLINE FOLLY_ATTR_VISIBILITY_HIDDEN`.

Reviewed By: vitaut

Differential Revision: D16686469

fbshipit-source-id: cedaee61955a40ae28ad2b0da565fc9fb385bc6e

Summary:
Update the generated `run_cmake.py` script to allow the caller to specify that
they just want to run a build without the install step.

Reviewed By: wez

Differential Revision: D16778007

fbshipit-source-id: 1859aca2b80fa7b099b4790682a6508e0185f2a0

Summary:
This cleans up how the `CMAKE_ENV` and `CMAKE_DEFINE_ARGS` variables are
written in the generated `run_cmake.py` script that we emit for CMake-based
projects.

We now emit each entry in these variables on separate lines, just to improve
readability.  (Both of these variables tend to have a number of entries and
are very long if emitted on a single line.)

This also replaces the `-DCMAKE_INSTALL_PREFIX` entry in `CMAKE_DEFINE_ARGS`
to have it correctly honor the `INSTALL_DIR` variable defined in
`run_cmake.py`.  This makes `run_cmake.py` still do the right thing if someone
manually edits it to change the `INSTALL_DIR` value.

Reviewed By: wez

Differential Revision: D16778006

fbshipit-source-id: fee5d25748b87b5d9c57ee2edf8de5e586e872ee

Summary:
Add arguments to getdeps.py to allow overriding the source, build, and install
directories a per-project basis.  The arguments take the form `[PROJECT:]PATH`
If the `PROJECT` portion is omitted, it defaults to the current project being
built.

In particular this makes it possible to specify `--src-dir .` to tell
getdeps.py to find the project sources from the current directory rather than
downloading them.

Reviewed By: wez

Differential Revision: D16778011

fbshipit-source-id: f33b87213ace04abb66334f588babdf59df91964

Summary:
This makes getdeps.py no longer crash if used in the folly repository.
Folly does not depend on any other Facebook projects, so it does not
include a `build/deps` directory.

Reviewed By: wez

Differential Revision: D16778009

fbshipit-source-id: 41be53d862f41b62154b081eb90ddba8742658fe

Summary:
Most of the getdeps subcommands operate on a single project, and some of the
argument parsing and initial logic to load the project is largely the same.
This consolidates that logic into a base class so that we can share this code
across subcommands, instead of repeating it.

This also unifies the behavior so that by default all commands enable tests on
the specified project, and disable test on dependent projects.  Making sure
all commands use the same behavior here is important as whether are not tests
are enabled can affect the project configuration, and therefore affect its
getdeps hash.

Reviewed By: wez

Differential Revision: D16778010

fbshipit-source-id: 044f99ad6cdd4a56f843276cec8ead786249ee7a

Reviewed By: yfeldblum

Differential Revision: D16785914

fbshipit-source-id: 73ade28f7aa300ce5e278e2ff2f56f4aafef9533

Summary: As title

Reviewed By: yfeldblum

Differential Revision: D16808375

fbshipit-source-id: 0efbb64c0ad9403f4efb1e2a885b21d476ff7cac

Summary: Useful to disable this on some platforms.

Reviewed By: yfeldblum

Differential Revision: D16153556

fbshipit-source-id: ab5916c929f5987dad7daa0c66b229eaf7404c88

Summary:
- Add allocator-extended copy constructors
- Add allocator-extended move constructors
- Add allocator-extended initializer-list constructor
- Add get_allocator accessor
- Make emplace/emplace_hint use the allocator
- pmr tests catch allocations that "escape" by setting the default resource to null

Reviewed By: nbronson

Differential Revision: D16623448

fbshipit-source-id: c10682f98d1c234fcad95608643b2f3ccea64a41

Summary: This function should be marked const.

Reviewed By: yfeldblum

Differential Revision: D16793980

fbshipit-source-id: 3e98f999c3e00caa0d238fa29c082ab6e819bf2a