Summary:
A long time ago, getdeps scheduled each build up to the number of hardware threads. For some builds, that was too heavy, so it got throttled back to only ever use half the hardware threads. This left parallelism on the table for CPU-bound projects that don't use much RAM per compilation.

This commit makes better use of the hardware with finer-grained logic that allows each manifest to specify a `job_weight_mib` estimate in MiB, and limit concurrency to `available_ram / job_weight`.

Reviewed By: ahornby

Differential Revision: D33754018

fbshipit-source-id: 785bed6c6cfe3c473244e0806a77cec1fc119e1f

Summary:
The flip operation, like the set and reset operations, are optimizable on x86-related architectures using the `btc` instruction, compared with `bts` and `btr` for set and reset operations.

The optimization is done for gnu-like compilers. However, for MSVC, there is no interlocked intrinsic for `btc`, so the optimization is not done.

Reviewed By: luciang

Differential Revision: D32049844

fbshipit-source-id: 5f60f8c4c534fb0ee41d76c5d766f96f7ff94892

Summary:
checkout@v2 is the current action version, and the old one may have
security issues. Let's just run with the latest version.

Reviewed By: ahornby

Differential Revision: D33834847

fbshipit-source-id: fba0733577bf23c208c9824aa3815bd01a813d9a

Summary:
I'm about to upgrade from checkout v1 to checkout v2. First, update
the GitHub action workflows for all projects.

Reviewed By: ahornby

Differential Revision: D33834784

fbshipit-source-id: f4cba24f1a589588dbfaf7ad7ada7122dd8284cb

Summary:
The public_autocargo is for external builds, not internal ones as they contain
references to git repository, while the non-public_autocargo refers to path in
fbcode. In Sandcastle, git repositories can fail to be accessed due to lack fo
proxies, thus we shouldn't access them.

Reviewed By: kmancini

Differential Revision: D33907864

fbshipit-source-id: 3f2006607676cebd92201560365d179231214db4

Summary: Release notes are: https://github.com/google/googletest/releases/tag/release-1.11.0.

Reviewed By: bigfootjon

Differential Revision: D33868235

fbshipit-source-id: 57b0e118d6ebadbaf634caccec50a0dd87f34fc8

Summary: Workaround for getdeps builds of edenfs

Reviewed By: fanzeyi

Differential Revision: D33896917

fbshipit-source-id: 9aed609ca6d5b209cda4fc7eecb52e5cfa9f6dac

Summary:
Move cargo logic to separate file as builder.py was getting a bit large

It's just a move of source file for CargoBuilder.  Diff looks a bit big as I did hg cp to preserve history

Differential Revision: D33888925

fbshipit-source-id: 2d57343535ab087e09876edba4d00a6f0234fcd0

Summary: Title

Reviewed By: yfeldblum

Differential Revision: D33873064

fbshipit-source-id: 65b16ad30a7bf9f2b9e23857f072d33c94098899

Summary: Generate cargo files for hgclientconf

Reviewed By: yancouto

Differential Revision: D33843589

fbshipit-source-id: d9b3452be684fcaa727c0c0a1175c1d8954bf9f2

Summary:
Since clang has a mode which rejects the append-va-arg construct as implemented in `FOLLY_PP_DETAIL_APPEND_VA_ARG`.

Do this by avoiding appending va-args to other args. Instead, take advantage of `operator()` as a workaround.

Avoids:
```
folly/Preprocessor.h:72:46: error: token pasting of ',' and __VA_ARGS__ is a GNU extension [-Werror,-Wgnu-zero-variadic-macro-arguments]
#define FOLLY_PP_DETAIL_APPEND_VA_ARG(...) , ##__VA_ARGS__
                                             ^
```

Reviewed By: luciang

Differential Revision: D33863906

fbshipit-source-id: 14912aca73e332ab811db4c0ac772a2d224a0666

Summary: So that we may easily inspect the generated code.

Reviewed By: ot, luciang

Differential Revision: D32161049

fbshipit-source-id: 00ca6f4196518389f7b1b18d9278085e1a3b43d2

Summary:
Function implements move and nuke operations via a type-erased pointer to an exec function. Both operations are explicitly noexcept but the function pointer which these operations call is not marked as noexcept. So the compiler emits unnecessary exception-handling code/data around the call. The exception-handling stuff make generated code larger and in theory may also hinder possible optimizations.

But when the exec function pointer as marked as `noexcept`, the compiler may elide the exception-handling stuff, shrinking generated code and in theory permitting a class of optimizations.

It is not possible for the underlying move operation to throw an exception since, if the move operation of the contained callable could throw an exception, the contained callable would be on-heap and not in-situ, and the underlying move operation would just be an infallible relocation of the in-situ `Function` state.

It is possible for the underlying nuke operation to throw an exception since the destructor of the contained callable could throw an exception. But this is rare. In this case, the exec function would contain exception-handling stuff; but in general, it would not. And this would merely moves where the exception-handling stuff is for the nuke operation, from one side of the type-erasure to the other side.

Reviewed By: ot, philippv, luciang

Differential Revision: D32904616

fbshipit-source-id: 5853a2d81a30bef565b350b72f5f35e6c9c9a45b

Summary:
Avoids:
```
folly/experimental/symbolizer/StackTrace.cpp:192:15: error: the pointer incremented by 68719476736 refers past the last possible element for an array in 32-bit address space containing 64-bit (8-byte) elements (max possible 536870912 elements) [-Werror,-Warray-bounds]
              normalStackFrame + kMaxExpectedStackFrameSize)) {
              ^                  ~~~~~~~~~~~~~~~~~~~~~~~~~~
```

Reviewed By: kennyyu, luciang

Differential Revision: D33837703

fbshipit-source-id: 0ca6b7ba6c2f76d0be3c54c9784c305125b525df

Summary: To measure `atomic_fetch_set` across sizes and behaviors.

Reviewed By: luciang

Differential Revision: D32262813

fbshipit-source-id: 84510e85368cb5e012ddc8bbf2ca159444659f20

Reviewed By: ot

Differential Revision: D33857528

fbshipit-source-id: 265457c5771684d6830a16a3b7390e9bd99b277d

Summary:
Currently, `getSingletonStackTrace` is in the logically private `detail`
namespace. This adds `symbolizer::getStackTrace` to replace `detail::getSingletonStackTrace`,
and this new location mirrors the existing `symbolizer::getAsyncStackTrace`.

The changes will be done in multiple parts:
- Add the new version
- Modify all internal callers to use the new version
- Delete the old version

Reviewed By: yfeldblum

Differential Revision: D33783972

fbshipit-source-id: 137a1d654aea5a451bb90ca12e1916f91589b0df

Summary: Like `FOLLY_SAFE_CHECK(true, ...)`.

Reviewed By: luciang

Differential Revision: D33801512

fbshipit-source-id: c9023f4b14e0b045c654c15a8f2b9ea0cc384617

Reviewed By: yns88

Differential Revision: D33850004

fbshipit-source-id: f01c3508c91049c3fc4f72bec14683b50332df77

Summary:
Originally set in D1315038, 16x overhead for stacksize for ASAN was too
pessimistic. The assumption that pre-allocating fiber stacks does not increase
process RSS was incorrect. As a result, fiber-heavy services consume way too
much memory when ASAN is enabled.

Reviewed By: yfeldblum

Differential Revision: D33754702

fbshipit-source-id: 7d0f3eb8aff8f3b7d17ac4a4960f5072822df05f

Summary:
- Use `ThreadIdWorkerProvider` in `IOThreadPoolExecutor` for Thrift dogpiles
- Only enable thread ID collection in `IOThreadPoolExecutor` if `THRIFT_FLAG(enable_io_queue_lag_detection)` is enabled

Reviewed By: yfeldblum

Differential Revision: D32815984

fbshipit-source-id: 9c923e44ca0e3ac93697f7332ac34a95b4c784cb

Summary:
In buck builds, fmt is at 8.0.1 which isn't entirely compatible with fmt 7 that
getdeps uses. Let's sync up the version.

Reviewed By: chadaustin

Differential Revision: D33832051

fbshipit-source-id: 7de71c4b675f220c0aa5b418c3466d65d78e9292

Summary: Pull Request resolved: https://github.com/facebook/folly/pull/1697

Reviewed By: Gownta

Differential Revision: D33483753

Pulled By: Orvid

fbshipit-source-id: d08291b5d02c61e461322edd332bb5a6e744ff96

Summary: Don't set the "fb" cargo feature via cmake unless we are in fb build.

Reviewed By: quark-zju

Differential Revision: D33781554

fbshipit-source-id: 39c61dc1631ebfd8a2c858b5e214c6a09ce0b005

Summary:
SSLAcceptRunnerFiberCloseSessionCb was leaking the SSL_SESSION because it set a
custom server SSL session callback handler that did not properly manage the
SSL_SESSION that it was given.

Reviewed By: modiking

Differential Revision: D33807657

fbshipit-source-id: 5eeaecfef5df1f1d281abe2918f9926a969ab84d

Summary:
Populate the new environment variable if relevant path for it found.

For the OSS getdeps builds of things depending on fb303 this will allow them to be able to be able to find the fb303 thrift files from the declared fb303 dependency, rather than needing duplicates/stubs inside each package.

Also adds python as dependency to fbthrift manifest,  found it needed it when testing

Reviewed By: HarveyHunt

Differential Revision: D33771979

fbshipit-source-id: 9635221c0e6a2ac225d7d0babe76f1c919b583de

Summary: The same logic that was gated for MSVC is applicable to GCC as well (note that it had to be fixed).

Reviewed By: yfeldblum

Differential Revision: D33799604

fbshipit-source-id: 34bd7159c0cc37b5d49d631f715339512bfc4ddd

Summary: Avoid using `bool` directly in `IOThreadPoolExecutor` constructor

Reviewed By: yfeldblum, praihan

Differential Revision: D33769477

fbshipit-source-id: ba16d676d2130c7680c960cd27f43174f06d0475

Summary: The test looks for every function in its list of functions to be present in the example stack-trace. But, depending on optimizations, the compiler may transform some of the calls to tail calls, in which case the calling functions would be elided from the stack-trace. Prevent those tail calls.

Reviewed By: kennyyu

Differential Revision: D33797857

fbshipit-source-id: 1c20c19bc612cebbc6a30466ef8deb54713b1497

Summary:
Fix a bunch of issues that a blocking migration to fmt 8.x:

* Missing includes
* Use of deprecated APIs
* Ambiguous calls to unqualified `format` (can be fmt or folly)
* Unused result of `fmt::format`
* Missing path formatter
* Passing runtime format string

Reviewed By: xavierd

Differential Revision: D33767880

fbshipit-source-id: 3976c4cc0551f3fb42c05a04c29e1c9723245749

Summary:
clang is particular about not treating strings as booleans, but it offers an "out" by using `expr && "string"` specifically for the use cases of `assert` like ours

**Before**
```
assert(!"Should never happen!"); <-- clang warning: -Wstring-conversion
```

**After**
```
assert(false && "Should never happen!");  <-- no clang warning
```

Reviewed By: yfeldblum

Differential Revision: D33796849

fbshipit-source-id: bd48139d0db04d50572e9ada29da777551774f84

Summary: Bring fboss in line with other oss builds

Reviewed By: shri-khare

Differential Revision: D33620345

fbshipit-source-id: 26e51230e4ae3c950eca2cb366a9fa55d5c1dfee

Summary:
Need to specify both gtest and gmock includes like folly does

Centos stream 8 doesn't have libmnl-static, and needed different alias for nl-route-3

Reviewed By: shri-khare

Differential Revision: D33537568

fbshipit-source-id: 506ef13224e71b874f8a43b52474f39da14a8d67

Reviewed By: magedm

Differential Revision: D33763949

fbshipit-source-id: 779c2a3d0337a357395a2554e810e7b0903a86d6

Summary:
Per standard pointer arithmetic is only defined when keeping pointers within the bounds of an array object.

 Applying non-zero offset to nullptr (or making non-nullptr a nullptr by subtracting pointer's integral value from the pointer itself) is undefined behavior.

Since https://reviews.llvm.org/D66608 `[InstCombine] icmp eq/ne (gep inbounds P, Idx..), null -> icmp eq/ne P, null) LLVM middle-end uses those guarantees for transformations.` and mis-compilations have been observed:
- https://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20190826/687838.html
- https://github.com/google/filament/pull/1566

To prevent help weed out bugs before they lead to future miscompilations a new UBSAN check has been added: `nullptr-with-nonzero-offset`
- https://reviews.llvm.org/D67122 [UBSan][clang][compiler-rt] Applying non-zero offset to nullptr is undefined behaviour

`folly::io::Cursor` does this type of operations when checking if `crtPos_ + N <= crtEnd_`: when it's empty it becomes `nullptr + N <= nullptr`:
  const uint8_t* crtBegin_{nullptr};
  const uint8_t* crtEnd_{nullptr};
  const uint8_t* crtPos_{nullptr};

Switch to `uintptr_t` space where the math is no longer UB.

Reviewed By: yfeldblum

Differential Revision: D33737556

fbshipit-source-id: 588b91ac1387112a6f183edfda5555ca1b7193d8

Summary: Exec ops include move and nuke ops.

Reviewed By: ot, philippv

Differential Revision: D32904615

fbshipit-source-id: cb6ffa81f034c2dd8072ebd605dc8caabac2f199

Summary:
[Folly] A place for compiler hints in `folly/lang/Hint.h`.

Add long-form names which explicitly reveal intentions:
* `compiler_may_unsafely_assume` v.s. `assume`
* `compiler_may_unsafely_assume_unreachable` v.s. `assume_unreachable`
* `compiler_must_not_elide` v.s. `doNotOptimizeAway`
* `compiler_must_not_predict` v.s. `makeUnpredictable`

Reviewed By: Gownta

Differential Revision: D20240392

fbshipit-source-id: 5f13fafc6a426fcd66fa61465eb1e2ffe15b4187

Summary: We have dataclasses dependency now in eden_scm, which means python 3.7 or later

Reviewed By: quark-zju

Differential Revision: D33713759

fbshipit-source-id: 6f915238326201d7408f885804714981505b7fa8

Summary:
Add support for overriding os, distro and distro version to command line when inspecting system packages so one can requested see ubuntu 18.04 package from other OS.  Makes testing easier

Used shlex to shell unquote the value to be tested in the getdeps expression evaluator. getdeps expression parser didn't tolerate 18.04 as . is special char to getdeps expressions, needed to be "18.04"

Reviewed By: quark-zju

Differential Revision: D33741323

fbshipit-source-id: d83397c7fb5180a4d985d0d8ae7b3ff33b72f828

Summary: ThreadManager.cpp and CPUThreadPoolExecutor.cpp both define the same implemenation of a ThreadIdCollector. Let's create one implementation that can be shared.

Reviewed By: yfeldblum, amlannayak

Differential Revision: D33065836

fbshipit-source-id: b17ef7abbba29cc7bad3779f20c76a3b08773963