Summary: Underflow/overflow of signed integers is fraught with dangers. So this diff removes the potential for overflow while adding or subtracting buckets in folly stats.

Reviewed By: yfeldblum

Differential Revision: D16898755

fbshipit-source-id: 4d3df1024b4c01ea0b276278105071e62e27fa27

Summary:
In the Facebook infrastructure the testpilot runner talks
to a service to determine whether a given test should be run locally.

The remote service was returning an error for some tests because they
didn't have enough configuration specified.

This diff associates some configuration with the tests based on the
host type and makes things happier.

Reviewed By: Ben0mega

Differential Revision: D16894682

fbshipit-source-id: b6f59b112991160ef26cc126e98041d8908a6273

Summary:
Sized deallocation via ::operator delete with size == 0 is
broken in some allocators.  This diff adds a unit test that ensures we
are either disabling sized deallocator or running with a good allocator.

Reviewed By: interwq

Differential Revision: D10857654

fbshipit-source-id: 981c7bcc1b021d4cd1ea1725ccf510d1958842fd

Reviewed By: pixelb

Differential Revision: D15301144

fbshipit-source-id: f94a805427b6fc8b20e7c01fcf6e1853fbebf7f1

Summary: Make `AsyncServerSocket` not fail on `EINPROGRESS` in `bind` and fix handling of `errno` there.

Reviewed By: yfeldblum

Differential Revision: D16791566

fbshipit-source-id: 1aeaefb6bae47bf41fd2d20238b716b84d0566d1

Summary: T52681488

Reviewed By: Orvid

Differential Revision: D16901429

fbshipit-source-id: 0a5fc218edf6ef29b100365163102dbdc2abf313

Summary:
[Folly] Suppress a spurious MSVC warning in `folly/Traits.h`.

```
folly\traits.h(645): warning C4287: '>': unsigned/negative constant mismatch
folly\traits.h(646): warning C4287: '<=': unsigned/negative constant mismatch
folly\traits.h(647): warning C4287: '<': unsigned/negative constant mismatch
```

Reviewed By: Orvid

Differential Revision: D16889045

fbshipit-source-id: 31edd58bf046143092a5f006ee0ac5b071b539a0

Summary:
[Folly] Fix MSVC conversion warning in `Optional` test.

```
folly\optional.h(372): warning C4244: 'argument': conversion from 'double' to 'int', possible loss of data
folly\optional.h(358): note: see reference to function template instantiation 'void folly::Optional<Type>::construct<std::initializer_list<_Ty>&,double>(std::initializer_list<_Ty> &,double &&)' being compiled
          with
          [
              _Ty=int
          ]
folly\optional.h(359): note: see reference to function template instantiation 'void folly::Optional<Type>::construct<std::initializer_list<_Ty>&,double>(std::initializer_list<_Ty> &,double &&)' being compiled
          with
          [
              _Ty=int
          ]
folly\optional.h(449): note: see reference to function template instantiation 'folly::Optional<Type>::Optional<std::initializer_list<_Ty>&,double>(folly::Optional<Type>::PrivateConstructor,std::initializer_list<_Ty> &,double &&) noexcept' being compiled
          with
          [
              _Ty=int
          ]
folly\optional.h(451): note: see reference to function template instantiation 'folly::Optional<Type>::Optional<std::initializer_list<_Ty>&,double>(folly::Optional<Type>::PrivateConstructor,std::initializer_list<_Ty> &,double &&) noexcept' being compiled
          with
          [
              _Ty=int
          ]
folly\test\optionaltest.cpp(659): note: see reference to function template instantiation 'folly::Optional<Type> folly::make_optional<Type,int,double>(std::initializer_list<_Ty>,double &&)' being compiled
          with
          [
              _Ty=int
          ]
```

Reviewed By: Orvid

Differential Revision: D16889085

fbshipit-source-id: c59f5e3fa0d4402e1b836841dff66d73b111235f

Summary:
Building a project that uses folly and `target_compile_features` may fail
because folly requires C++14 and CMake doesn't know about it. So if a project
has only C++11 features in `target_compile_features` CMake will pick
`-std=gnu++11` or similar and compilation will fail on an assertion in
`folly/Portability.h`. To fix this, add `target_compile_features` to the
`folly` target pushing the minimum required standard to C++14.

Reviewed By: Orvid

Differential Revision: D16884888

fbshipit-source-id: 0e5ac4abde10b1ea757011bcd05b17d508859b4f

Summary:
Add some CMake functions for building standalone executables from Python
source files.  This generates executables similar to PEX
(https://github.com/pantsbuild/pex).

In the future this could potentially be leveraged to directly build XAR files
(https://github.com/facebookincubator/xar).

The main advantages of these functions is that they allow easily defining
"libraries" of python files, and their dependencies, which can then be used
and packaged as part of multiple different standalone executables.

Reviewed By: wez

Differential Revision: D16722499

fbshipit-source-id: e1d829b911dc428e5438b5cf9cebf99b3fb6ce24

Summary: Avoid SIGSEGV if getIOExecutor() returns null

Reviewed By: yfeldblum

Differential Revision: D16859157

fbshipit-source-id: 00c7a45677c61d84a10486f11b4f792093ac2ff5

Summary:
It's spammy and is just informational.

Created from Diffusion's 'Open in Editor' feature.

Reviewed By: yfeldblum

Differential Revision: D16886741

fbshipit-source-id: 29aecb8a62cac9a60cbc4cabc596fd1ee00cfc04

Summary:
[Folly] Cut IsRelocatable markings of containers, removing forward declarations of the containers to avoid over-inclusion and to avoid undefined behavior.

> 16.5.4.2.1 [namespace.std]\1
> Unless otherwise specified, the behavior of a C++ program is undefined if it adds declarations or definitions to namespace std or to a namespace within namespace std.

Closes #1203.
Closes #1204.

Reviewed By: igorsugak

Differential Revision: D16855473

fbshipit-source-id: 0e6acc4f589707aa79b0001b415bcadfab652d62

Summary: [Folly] Revise `atomic_fetch_set`, `atomic_fetch_reset` comments, enforcing structure and switching comment style.

Reviewed By: aary

Differential Revision: D16877442

fbshipit-source-id: 0c4c163f2daa14616f20c4e0cac1002c83ec5fac

Summary:
http://man7.org/linux/man-pages/man2/mlock.2.html

       MLOCK_ONFAULT
              Lock pages that are currently resident and mark the entire
              range so that the remaining nonresident pages are locked when
              they are populated by a page fault.

The kernel still checks a few things for `MLOCK_ONFAULT`: rlimits RLIMIT_MEMLOCK / CAP_IPC_LOCK => EPERM or ENOMEM

`TRY_LOCK` & `MUST_LOCK` are only different in in their handling of EPERM, ENOMEN: whether to fail or ignore the errors, so I added a separate enum.

https://github.com/torvalds/linux/blob/v5.0/mm/mlock.c#L29-L37
```
bool can_do_mlock(void)
{
	if (rlimit(RLIMIT_MEMLOCK) != 0)
		return true;
	if (capable(CAP_IPC_LOCK))
		return true;
	return false;
}
EXPORT_SYMBOL(can_do_mlock);
```

https://github.com/torvalds/linux/blob/v5.0/mm/mlock.c#L671-L714
```
static __must_check int do_mlock(unsigned long start, size_t len, vm_flags_t flags)
{
	unsigned long locked;
	unsigned long lock_limit;
	int error = -ENOMEM;

	if (!can_do_mlock())
		return -EPERM;

	len = PAGE_ALIGN(len + (offset_in_page(start)));
	start &= PAGE_MASK;

	lock_limit = rlimit(RLIMIT_MEMLOCK);
	lock_limit >>= PAGE_SHIFT;
	locked = len >> PAGE_SHIFT;

	if (down_write_killable(&current->mm->mmap_sem))
		return -EINTR;

	locked += current->mm->locked_vm;
	if ((locked > lock_limit) && (!capable(CAP_IPC_LOCK))) {
		/*
		 * It is possible that the regions requested intersect with
		 * previously mlocked areas, that part area in "mm->locked_vm"
		 * should not be counted to new mlock increment count. So check
		 * and adjust locked count if necessary.
		 */
		locked -= count_mm_mlocked_page_nr(current->mm,
				start, len);
	}

	/* check against resource limits */
	if ((locked <= lock_limit) || capable(CAP_IPC_LOCK))
		error = apply_vma_lock_flags(start, len, flags);

	up_write(&current->mm->mmap_sem);
	if (error)
		return error;

	error = __mm_populate(start, len, 0);
	if (error)
		return __mlock_posix_error_return(error);
	return 0;
}
```

https://github.com/torvalds/linux/blob/v5.0/mm/mlock.c#L721-L732
```
SYSCALL_DEFINE3(mlock2, unsigned long, start, size_t, len, int, flags)
{
	vm_flags_t vm_flags = VM_LOCKED;

	if (flags & ~MLOCK_ONFAULT)
		return -EINVAL;

	if (flags & MLOCK_ONFAULT)
		vm_flags |= VM_LOCKONFAULT;

	return do_mlock(start, len, vm_flags);
}

```

Reviewed By: yfeldblum

Differential Revision: D16760748

fbshipit-source-id: 54937046f5faaceac2cfb9dd6a0f207ad653fe6b

Summary:
Currently the timekeeper callback set by the within method isn't cleaned up when the future resolves and always waits the full duration. This can cause OOM under high QPS and long timeout situations.

This resolves this by cancelling the callback.

Reviewed By: mpark

Differential Revision: D16835816

fbshipit-source-id: c400afc37dbc66447aaa8fd557a770d717d9ab02

Summary:
RocksDB's TSAN version was not able to instrument atomic_fetch_set and
atomic_fetch_reset; and as a result the DistributedMutex tests were failing
under TSAN

Also TSAN seems to have some issue when and exception_ptr is placed into an
aligned_storage_t instance. This verifiably correct program (I think) also fails
with the same TSAN error P78696506. So
disable that test

This is the code with the same false-negative
```
namespace {
class ExceptionWithConstructionTrack : public std::exception {
 public:
  explicit ExceptionWithConstructionTrack(int id)
      : id_{folly::to<std::string>(id)}, constructionTrack_{id} {}

  const char* what() const noexcept override {
    return id_.c_str();
  }

 private:
  std::string id_;
  TestConstruction constructionTrack_;
};

template <typename Storage, typename Atomic>
void transferCurrentException(Storage& storage, Atomic& produced) {
  assert(std::current_exception());
  new (&storage) std::exception_ptr(std::current_exception());
  produced->store(true, std::memory_order_release);
}

void concurrentExceptionPropagationStress(
    int numThreads,
    std::chrono::milliseconds milliseconds) {
  auto&& stop = std::atomic<bool>{false};
  auto&& exceptions = std::vector<std::aligned_storage<48, 8>::type>{};
  auto&& produced = std::vector<std::unique_ptr<std::atomic<bool>>>{};
  auto&& consumed = std::vector<std::unique_ptr<std::atomic<bool>>>{};
  auto&& consumers = std::vector<std::thread>{};
  for (auto i = 0; i < numThreads; ++i) {
    produced.emplace_back(new std::atomic<bool>{false});
    consumed.emplace_back(new std::atomic<bool>{false});
    exceptions.push_back({});
  }

  auto producer = std::thread{[&]() {
    auto counter = std::vector<int>(numThreads, 0);
    for (auto i = 0; true; i = ((i + 1) % numThreads)) {
      try {
        throw ExceptionWithConstructionTrack{counter.at(i)++};
      } catch (...) {
        transferCurrentException(exceptions.at(i), produced.at(i));
      }

      while (!consumed.at(i)->load(std::memory_order_acquire)) {
        if (stop.load(std::memory_order_acquire)) {
          return;
        }
      }

      consumed.at(i)->store(false, std::memory_order_release);
    }
  }};

  for (auto i = 0; i < numThreads; ++i) {
    consumers.emplace_back([&, i]() {
      auto counter = 0;
      while (true) {
        while (!produced.at(i)->load(std::memory_order_acquire)) {
          if (stop.load(std::memory_order_acquire)) {
            return;
          }
        }
        produced.at(i)->store(false, std::memory_order_release);

        try {
          auto storage = &exceptions.at(i);
          auto exc = folly::launder(
            reinterpret_cast<std::exception_ptr*>(storage));
          auto copy = std::move(*exc);
          exc->std::exception_ptr::~exception_ptr();
          std::rethrow_exception(std::move(copy));
        } catch (std::exception& exc) {
          auto value = std::stoi(exc.what());
          EXPECT_EQ(value, counter++);
        }

        consumed.at(i)->store(true, std::memory_order_release);
      }
    });
  }

  std::this_thread::sleep_for(milliseconds);
  stop.store(true);
  producer.join();
  for (auto& thread : consumers) {
    thread.join();
  }
}
} // namespace
```

Reviewed By: yfeldblum

Differential Revision: D16739396

fbshipit-source-id: 0eba191ffc07d2e64a3ea550c5fe9a7d01ce2f18

Summary: As title

Reviewed By: yfeldblum

Differential Revision: D16821444

fbshipit-source-id: 14f0655caf0f53ef47fa65b9b6ba57f0f190df12

Summary: [Folly] `#define FOLLY_ERASE FOLLY_ALWAYS_INLINE FOLLY_ATTR_VISIBILITY_HIDDEN`.

Reviewed By: vitaut

Differential Revision: D16686469

fbshipit-source-id: cedaee61955a40ae28ad2b0da565fc9fb385bc6e

Summary:
Update the generated `run_cmake.py` script to allow the caller to specify that
they just want to run a build without the install step.

Reviewed By: wez

Differential Revision: D16778007

fbshipit-source-id: 1859aca2b80fa7b099b4790682a6508e0185f2a0

Summary:
This cleans up how the `CMAKE_ENV` and `CMAKE_DEFINE_ARGS` variables are
written in the generated `run_cmake.py` script that we emit for CMake-based
projects.

We now emit each entry in these variables on separate lines, just to improve
readability.  (Both of these variables tend to have a number of entries and
are very long if emitted on a single line.)

This also replaces the `-DCMAKE_INSTALL_PREFIX` entry in `CMAKE_DEFINE_ARGS`
to have it correctly honor the `INSTALL_DIR` variable defined in
`run_cmake.py`.  This makes `run_cmake.py` still do the right thing if someone
manually edits it to change the `INSTALL_DIR` value.

Reviewed By: wez

Differential Revision: D16778006

fbshipit-source-id: fee5d25748b87b5d9c57ee2edf8de5e586e872ee

Summary:
Add arguments to getdeps.py to allow overriding the source, build, and install
directories a per-project basis.  The arguments take the form `[PROJECT:]PATH`
If the `PROJECT` portion is omitted, it defaults to the current project being
built.

In particular this makes it possible to specify `--src-dir .` to tell
getdeps.py to find the project sources from the current directory rather than
downloading them.

Reviewed By: wez

Differential Revision: D16778011

fbshipit-source-id: f33b87213ace04abb66334f588babdf59df91964

Summary:
This makes getdeps.py no longer crash if used in the folly repository.
Folly does not depend on any other Facebook projects, so it does not
include a `build/deps` directory.

Reviewed By: wez

Differential Revision: D16778009

fbshipit-source-id: 41be53d862f41b62154b081eb90ddba8742658fe

Summary:
Most of the getdeps subcommands operate on a single project, and some of the
argument parsing and initial logic to load the project is largely the same.
This consolidates that logic into a base class so that we can share this code
across subcommands, instead of repeating it.

This also unifies the behavior so that by default all commands enable tests on
the specified project, and disable test on dependent projects.  Making sure
all commands use the same behavior here is important as whether are not tests
are enabled can affect the project configuration, and therefore affect its
getdeps hash.

Reviewed By: wez

Differential Revision: D16778010

fbshipit-source-id: 044f99ad6cdd4a56f843276cec8ead786249ee7a

Reviewed By: yfeldblum

Differential Revision: D16785914

fbshipit-source-id: 73ade28f7aa300ce5e278e2ff2f56f4aafef9533

Summary: As title

Reviewed By: yfeldblum

Differential Revision: D16808375

fbshipit-source-id: 0efbb64c0ad9403f4efb1e2a885b21d476ff7cac

Summary: Useful to disable this on some platforms.

Reviewed By: yfeldblum

Differential Revision: D16153556

fbshipit-source-id: ab5916c929f5987dad7daa0c66b229eaf7404c88

Summary:
- Add allocator-extended copy constructors
- Add allocator-extended move constructors
- Add allocator-extended initializer-list constructor
- Add get_allocator accessor
- Make emplace/emplace_hint use the allocator
- pmr tests catch allocations that "escape" by setting the default resource to null

Reviewed By: nbronson

Differential Revision: D16623448

fbshipit-source-id: c10682f98d1c234fcad95608643b2f3ccea64a41

Summary: This function should be marked const.

Reviewed By: yfeldblum

Differential Revision: D16793980

fbshipit-source-id: 3e98f999c3e00caa0d238fa29c082ab6e819bf2a

Summary:
Fix the BuildOptions class to correctly honor the `host_type` parameter that
it was constructed with when constructing the manifest evaluation context.
I accidentally broke this behavior in D16477396, and incorrectly had this code
path default to using the current host system rather than the value passed in
from the command line.

Reviewed By: wez

Differential Revision: D16779579

fbshipit-source-id: de911daaa643f6303fd35149775ab25d3f64d34f

Summary:
This adds standalone tests for the heterogeneous key features
of callWithExtractedKey and callWithConstructedKey.  These code
(metacode?) paths were already tested somewhat by F14MapTest and
F14SetTest.

Reviewed By: yfeldblum

Differential Revision: D16742421

fbshipit-source-id: 1f7e5065cad2216b3db4d4cb33fc73ac72d43c06

Summary:
Fixes compilation on non x86.
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Pull Request resolved: https://github.com/facebook/folly/pull/1202

Differential Revision: D16760427

Pulled By: yfeldblum

fbshipit-source-id: 6f31d5a17b9fe9c1f91d884eb94967ba06756f3e

Summary:
This diff fixes the initial conditions for SwapTrackingAlloc,
which fixes F14 tests on platforms that do not have intrinsics enabled.

Differential Revision: D16755857

fbshipit-source-id: a76d8093100cbc0c5ce8723daa0976673d942299

Summary:
This diff extracts the key destructuring logic in F14's emplace
functions into standalone helpers that can be used by other container
types (like sorted_vector_{set,map}).  It also adds better PMR compliance, by
using a stateful allocator to construct a temporary if that is required
to be able to perform the search.

This diff also cleans up some of the testing code in folly/container/test
that had issues if included in multiple places.

Reviewed By: yfeldblum

Differential Revision: D16655131

fbshipit-source-id: 5a6f57ac346d1fd5928e0f737110760b0c518974

Summary:
In response to review feedback for D16477400 and D16477401, update
`ManifestLoader.load_all_manifests()` to only update its data for projects
that have not previously been loaded.  This helps ensure that code using a
single `ManifestLoader` object cannot have two in-memory `Manifest` objects
for the same project, and that existing data (such as project hashes) can't be
invalidated if a manifest is later loaded from updated on-disk data.

Reviewed By: pkaush

Differential Revision: D16586682

fbshipit-source-id: 50b1979ec55f2ad6901629cd852293a8f6ca903f

Summary:
This is a missing piece from https://github.com/facebook/folly/pull/1178, but more nasty as it can cause runtime rather than compile-time failures.

Given an interface `ICat` with `void meow() const`, if we have `struct cat` with `void meow() const noexcept`, then constructing a `folly::Poly<ICat const&>` for an instance of `cat` will have a null pointer in the vtable entry for the `meow` method.

The culprit seems to be in the template specialization for [`ThunkFn` ](https://github.com/cstratopoulos/folly/blob/feature/poly-update/folly/detail/PolyDetail.h#L539) which may be dispatched on `IsConstMember`.

I've added a fairly trivial test case to `PolyTest.cpp`; but notably it is the only instance of _any_ test case in that file having a `noexcept` method. Without the change in `PolyDetail.h`, the call to `cref->meow()` will cause a null pointer dereference. If something more explicit is desired, maybe we could `EXPECT` that the corresponding vtable entry is non-null, but that doesn't exactly fit the pattern of other test cases in that file.

(Aside: In the linked PR I used `__cpp_noexcept_function_type` directly, but I have since realized that a macro for this is already present in [`Portability.h`](https://github.com/facebook/folly/blob/master/folly/Portability.h#L503) so I'm using that one now)
Pull Request resolved: https://github.com/facebook/folly/pull/1201

Reviewed By: ericniebler

Differential Revision: D16686139

Pulled By: yfeldblum

fbshipit-source-id: 73c7ac132d6e772ffd49f450b5fcd0f8a0dbdb26

Summary:
Added boundary support in folly/io/Cursor.h.

The bounded Cursor can have a boundary that is shorter than the underlying IOBuf. This gives a custom view of the IOBuf.

Reviewed By: spalamarchuk

Differential Revision: D16347838

fbshipit-source-id: 16d91e42e2acee788b31bc2a9b6cc8716b1d22f2

Summary: Switch defer to use inline continuations by default, if bound executors match.

Reviewed By: andriigrynenko

Differential Revision: D16643666

fbshipit-source-id: d3cdf86761c378d7ca138b46322c55a9755bdc48

Summary: As both coro.h and futures.h defines the function getExecutor, redefinition error occurs when both of them are included. This change moves the getExecutor inline function to executor.h so it can be shared.

Reviewed By: yfeldblum

Differential Revision: D16676456

fbshipit-source-id: 33f7558a3d5dbc996c4efef4fe90918de32ad8b2

Summary:
When inserting an item with a hint, the item will be always copied, even
if it is already present in the map. This could cause an exception to be
thrown when it doesn't need to be. The standard doesn't require this, but
it is an optimization anyways.

Reviewed By: yfeldblum

Differential Revision: D16585971

fbshipit-source-id: f0a8eeb0bd86f07ef110f91cbef8ab422ff55480