Summary: Improve IOBuf::clone and IOBuf::cloneOne() perf

Reviewed By: yfeldblum

Differential Revision: D20882203

fbshipit-source-id: ce37c2bf3c0fbdbe3c316856dc431a0108e4b9bf

Summary:
Fix folly::AsyncUDPSocket::gro_.value() check - GRO is enabled if the value is > 0

(Note: this ignores all push blocking failures!)

Reviewed By: kevin-vigor

Differential Revision: D20930544

fbshipit-source-id: 6672b65cafe3f9859f86ad34a6c3a2d65c2dc355

Summary: This prevents one acquire/release.

Reviewed By: lewissbaker

Differential Revision: D20924353

fbshipit-source-id: 8c5135bd2ebc03b736b1ea3dbb385e01218cffbc

Summary:
This is a follow up from D16427231; after testing the code in a similar situation on a mac (with `clang` and `libc++`) I realized there were still standards compliance issues.

In that diff, the guarantee we provide is:

```
 *   - insert() single key variants, emplace(), and emplace_hint() only provide
 *     the strong exception guarantee (unchanged when exception is thrown) when
 *     std::is_nothrow_move_constructible<value_type>::value is true.
```

The implementation is eventually backed by `std::vector`.

The guarantees provided by `std::vector` in the standard are slightly weaker (https://en.cppreference.com/w/cpp/container/vector/insert):

```
If an exception is thrown when inserting a single element at the end, and T is CopyInsertable or std::is_nothrow_move_constructible<T>::value is true, there are no effects (strong exception guarantee).
```

i.e. it only provides this safety guarantee if the element is inserted at the end, and that's exactly what the new test shows.

After discussion with nbronson we discovered that `libstdc++` and `libc++` are using different techniques to handle the corner case of `v.insert(0, v[1])` for `insert(iter, value_type const&)`. `libstdc++` makes the copy into a temporary, and then moves it in, so it does in fact provide a stronger guarantee than necessary. `libc++` checks the address of the reference it is given, and if it is in the moved range it adjusts it (!) and then uses the copy assignment operator. This means on copy assignment exception there will be a moved-from element at the desired index.

Section 26.2.1 para 11 says that `insert` or `emplace` will have no effects if an exception is thrown, unless otherwise specified. 26.3.11.5 is the `vector` modifiers, but that only mentions `insert`. That means we can fix this by merely switch our uses of `std::vector::insert` to `std::vector::emplace` inside `sorted_vector_map/set`. On `libstdc++` this should't have any effect, since both insert and emplace make a temporary copy for the non-end insertion case (in different but similar code paths). On `libc++` this will avoid the address-adjusting trick and use the temporary copy technique like in `libstdc++`.

Reviewed By: yfeldblum

Differential Revision: D20785383

fbshipit-source-id: 290c3c7c061dedf1660da9534f4b9cc338da6224

Summary: This should reduce refcounting

Reviewed By: lewissbaker

Differential Revision: D20911114

fbshipit-source-id: be941a5e0a095652c916e4704b0dd29874e0979d

Summary:
Migration from Future-returning executor-erasing collectX forms to
SemiFuture-returning forms, that are less risky in particular with coroutines.

Earlier changes added collectXSemiFuture and collectXUnsafe as a migration
path. We then migrated collectX callsites to collectXSemiFuture or
collectXUnsafe and switched the implementation of collectX to the SemiFuture
form.

This diff adds deprecation flag to collectXSemiFuture during the migration to
collectX and and removes from folly tests that are not specific to testing those
operations.

Reviewed By: yfeldblum

Differential Revision: D20841855

fbshipit-source-id: 30818711362c88c3296cfc976cd75fbd94e1c15c

Summary:
VS2019 changes the internal implementation of basic_string and vector.
This diff adds support for the new internals, and fixes a linkage
problem on VS2017.  It also adds the UninitializedMemoryHacksTest to
the getdeps-built tests.

For basic_string a single version handles both old and new.
basic_string::_Eos changed from public to private, so code that uses
the template specialization hack to invoke it when it is private will
work for the older public method as well.

vector does not have any suitable internal methods, so we need to adapt
to the new internal structure.

This diff fixes the issues addressed by
https://github.com/facebook/folly/pull/1345 , but uses an alternate
strategy to avoid reinterpret_cast.

Reviewed By: yfeldblum

Differential Revision: D20838799

fbshipit-source-id: eba7db2bd6feade1349d51be224c481a9156732b

Reviewed By: simpkins

Differential Revision: D20885314

fbshipit-source-id: 8c3a5ccbfd6630107b421b0d6953f17a93da2412

Summary:
Reduce the cost of copying by keeping a vector of cleared RequestData references instead of keeping a hash map of the state of all RequestData references.

Also, added FOLLY_ALWAYS_INLINE to functions called in copying of request contexts.

Reviewed By: yfeldblum

Differential Revision: D20863318

fbshipit-source-id: 29cc38ac6e8abde3acaffcf8fa4d1e7ccbd45e87

Summary: Use memcpy for copying maps with the same capacity instead of rehashing.

Reviewed By: yfeldblum

Differential Revision: D20863257

fbshipit-source-id: 11e52f9f2246440fb724a94c2611e44e253e462d

Summary:
When incrementally adding in MasterPtr to an existing code-base there are some shared_ptr<> (eg. returned from factories) that need to be stored in MasterPtr.

since unique_ptr converts to shared_ptr, just take and store a shared_ptr

Yes, this removes the guarantee that ~T() will run in join(). I am open to suggestions

Reviewed By: andriigrynenko

Differential Revision: D19583122

fbshipit-source-id: 3daf0436d56b9c396b5e82d397ebaf93ed1a2a2d

Summary:
when T derives from EnableSharedFromThis<T> and T is placed in a MasterPtr<T> then users of T have access to MasterPtr<T> functionality.

`EnableSharedFromThis<T>::masterLockFromThis()`
`EnableSharedFromThis<T>::masterRefFromThis()`

Reviewed By: andriigrynenko

Differential Revision: D19583025

fbshipit-source-id: ee12e9de30fd844b5be36c39d7e1ade830e27bbb

Summary:
The read of globalConfigData outside of sm.lock() races with any other writer that is modifying
globalConfigData inside of sm.lock(). This is a documentation only change.
Pull Request resolved: https://github.com/facebook/folly/pull/1348

Reviewed By: paulmckrcu

Differential Revision: D20864073

Pulled By: yfeldblum

fbshipit-source-id: 31b7ed469716acb35913df9a71585c2a59e50bea

Summary:
Add microbenchmark for ShallowCopyRequestContextScopeGuard that involves copying RequestContext-s that hold references to multiple RequestData objects.

Also, removed microbenchmark results for less common patterns, and updated results after switching to the hazard pointer-based implementation (D19145252).

Reviewed By: A5he

Differential Revision: D20862753

fbshipit-source-id: b4fc36a548761d4281c7c55ec97f731c7b4bc98d

Summary: [Folly] Revise references to `TAsyncSocket`.

Differential Revision: D20826538

fbshipit-source-id: 4448f19d1cc64b88dbccc1eda2be63355d15bf42

Summary:
This diff moves a struct definition out from the inside of
a closure, where it seems to have been causing a compilation issue
under VS2019.

Reviewed By: yfeldblum

Differential Revision: D20845042

fbshipit-source-id: 7b32c898490a8ba59f5e77f21e890097d9069a4f

Summary: Avoid calling ::io_uring_register_files_update again in case of failure

Reviewed By: danobi

Differential Revision: D20847977

fbshipit-source-id: 9dedecb4e815b3bb0bd41b0ba908a0c2496a67d5

Summary:
This diff fixes a couple small issues in some folly tests that
are exposed when compiling with VS2019.

Reviewed By: yfeldblum

Differential Revision: D20845004

fbshipit-source-id: 44a3e140f2bb4334c4eb81ae80297da4975a13ae

Summary: Call addTimerFd only after a callback to avoid issuing duplicate poll requests

Reviewed By: danobi

Differential Revision: D20843253

fbshipit-source-id: 9aa538dcc4cc63f1d1a3431d41366ce3fa3644cb

Summary: Handle spurious poll returning 0 when we have high load at process startup time

Reviewed By: danobi

Differential Revision: D20821506

fbshipit-source-id: 93ec91f0908f44e5210974e1cf3f30ac7aa38950

Summary:
gcc-9.3 complains about folly::Format:

```
array subscript -1 is outside array bounds of ‘char [69]’
```

But the warning is incorrect; my guess is that it is confused because
it doesn't realize that higher-level logic guarantees that integer
formatting will run out of digits before it runs out of buffer.

Changing some code from `array + x` to `&array[x]` works around it.

Reviewed By: pixelb

Differential Revision: D20694399

fbshipit-source-id: 1176858054cdec9e415de4829371647958955103

Summary:
In coverage collection mode a special module loader is prepended to
`sys.meta_path`. In very specific conditions this module loader can end up
returning a loader pointing to a _completely wrong module_. When importing
symbols from the wrong module errors occur.

The conditions to trigger the bug are:

- running in coverage collection mode, enabling the custom loader
- the test binary is a zip (e.g. par_style=fastzip)
- having a module name where the end part matches the name of a builtin Python
  module

When these conditions were met, the special loader would return the builtin
Python module instead of the expected module. E.g. when loading a module like
`myapp.somemod.platform` in a zip style binary.

The custom loader first calls `imp.find_module()` to find the module it wants
to return a wrapped loader for. This fails for modules included in the test
binary, because the builtin `imp` module can not load from zips. This was the
trigger leading to the call to the buggy code.

When the initial call to `imp.find_module()` failed, the custom loader would
try a second call, asking the internal loader to this time try any path on
`sys.path`. For most module names this call would also fail, making the custom
loader return `None`, after which Python tries other loaders on `sys.path`.
However, when the final part of the module that was asked to load matches the
name of a Python builtin module, then the second call to the `imp` module would
succeed, returning a loader for the builtin module. E.g. `platform` when asking
for `myapp.somemod.platform`.

This diff fixes the issue by removing the broken second call to the internal
loader. This will never have worked, we just have not triggered or noticed
triggering the wrong loading before.

Differential Revision: D20798119

fbshipit-source-id: dffb54e308106a81af21b63c5ee64c6ca2041920

Summary:
https://www.openssl.org/source/openssl-1.1.1b.tar.gz is gone

changelog:
https://www.openssl.org/news/openssl-1.1.1-notes.html

Reviewed By: udippant

Differential Revision: D20810020

fbshipit-source-id: 0ed385f49b2187ec149defd79feb86e2c8b492d2

Summary:
Change to SemiFuture returning collect, which checks for nullptr passed to via, breaks this test which produced a default nullptr.

Instead remove default and force passing of a valid executor.

Reviewed By: yfeldblum

Differential Revision: D20795552

fbshipit-source-id: 5411fea910d9f85eb5a4ba825a1cfd48a72b8a44

Summary: Add support for specifying which fatal signals to handle

Reviewed By: simpkins

Differential Revision: D20672281

fbshipit-source-id: 16696abcf551efcd932f5b56b336d6c5bc0d7a55

Summary: io_uring backend error handling

Reviewed By: kevin-vigor

Differential Revision: D20737416

fbshipit-source-id: 23eebc98326bcd07ee308e8f71f972c7374f6108

Summary:
I was testing vs2019 vs vs2017 and realized that
we weren't reconfiguring when the toolchain was changed;
this resolves that.

Reviewed By: genevievehelsel

Differential Revision: D20795118

fbshipit-source-id: db80f090367cacfcc6b53887b77cf949f9cef0e6

Summary:
We see flakey builds to conflicting writes to .pdb files; the error
message suggests enabling `/FS` which is already enabled.  Some serious
internet research reveals that we can put the debug info into `.obj` files
instead of having everyone contend with the same `.pdb` file.

```
C:\PROGRA~2\MIB055~1\2017\BUILDT~1\VC\Tools\MSVC\1416~1.270\bin\Hostx64\x64\cl.exe  /nologo /TP -DBOOST_CONFIG_SUPPRESS_OUTDATED_MESSAGE -DGFLAGS_IS_A_DLL=1 -DWIN32_LEAN_AND_MEAN -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_WARNINGS -D_ENABLE_EXTENDED_ALIGNED_STORAGE -D_SCL_SECURE_NO_WARNINGS -D_STL_EXTRA_DISABLED_WARNINGS="4774 4987" -IZ:\shipit\folly -I. -IZ:\installed\boost-6XBjAEHS6zoZVAY7uQHhSxYamF-vUNWngO1YaEIiiko\include\boost-1_69 -IZ:\installed\double-conversion-8uX8TcZDEz7gXyY70bMXqPUF2uLqH2OTWBffkLfU7FU\include -IZ:\installed\gflags-Md0yNU5drHUQ6MCdsB6iC0WS5YX-RFZk_eTHNKjkcPs\include -IZ:\installed\glog-_3NtzkcZDkkSFkgC70vwofqrnao05-mPxEa0E546a9g\include -IZ:\installed\libevent-TIUwVyvnVZZVHDsesjdIidM_iH4er-_6pEE0X4dzuL4\include -IZ:\installed\openssl-0G_glZ7y5BplM752sREIHKuam0GcoPqw7m6yjEGefyE\include -IZ:\installed\zlib-lxujVMVR4hhdh4-S0SCxkb55DAWh5ISCLWbfL7m7hwg\include -IZ:\installed\zstd-M8csiME1A2X7ZGECWeSGmWbqM1glHd6xoskhyPHgpaw\include -IZ:\installed\snappy-Z3WzzGXxgX1-6zSn4NdRg8Z3coQaz7Gob3gGoomN95E\include -IZ:\installed\fmt-W9-akKzyOlf4GoHcFsw1_rCxt3y0ghyjO3gb5Xve1Gc\include /DWIN32 /D_WINDOWS /W3 /GR  /MD /Zi /O2 /Ob1 /DNDEBUG   /EHs /GF /Zc:referenceBinding /Zc:rvalueCast /Zc:implicitNoexcept /Zc:strictStrings /Zc:threadSafeInit /Zc:throwingNew /permissive- /std:c++latest /bigobj /favor:blend /Zc:inline /Wall /MP /Gw /Gy /Qpar /Oi /Ot /wd4191 /wd4291 /wd4309 /wd4310 /wd4366 /wd4587 /wd4592 /wd4628 /wd4723 /wd4724 /wd4868 /wd4996 /wd4068 /wd4091 /wd4146 /wd4800 /wd4018 /wd4365 /wd4388 /wd4389 /wd4100 /wd4459 /wd4505 /wd4701 /wd4702 /wd4061 /wd4127 /wd4200 /wd4201 /wd4296 /wd4316 /wd4324 /wd4355 /wd4371 /wd4435 /wd4514 /wd4548 /wd4571 /wd4574 /wd4582 /wd4583 /wd4619 /wd4623 /wd4625 /wd4626 /wd4643 /wd4647 /wd4668 /wd4706 /wd4710 /wd4711 /wd4714 /wd4820 /wd5026 /wd5027 /wd5031 /wd5045 /we4099 /we4129 /we4566 /showIncludes /FoCMakeFiles\folly_base.dir\folly\Format.cpp.obj /FdCMakeFiles\folly_base.dir\ /FS -c Z:\shipit\folly\folly\Format.cpp

Z:\shipit\folly\folly\Format.cpp: fatal error C1041: cannot open program database 'Z:\build\folly\CMakeFiles\folly_base.dir\vc140.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS

ninja: build stopped: subcommand failed.
```

Reviewed By: yfeldblum

Differential Revision: D20752325

fbshipit-source-id: 4f20871d54cc0973b3d025734dd0886c202c839b

Summary:
This diff exposes the maximum number of internal stripes
supported by the AccessSpreader, increases that value from 128 to 256
for server platforms, and uses bulk string operations to speed up the
internal initialization. This change only affects the spreading results
on machines with more than 128 hardware threads.

(Note: this ignores all push blocking failures!)

Reviewed By: yfeldblum

Differential Revision: D20772443

fbshipit-source-id: 5c7142efb2bb9ba07cc423195074fc33aeea6dcb

Summary: This helps avoid TSAN false positives.

Reviewed By: yfeldblum

Differential Revision: D20787056

fbshipit-source-id: e62537235e3ad876e3ec5266f9c449d0d8340330

Summary: Detach by destruction is common, and for SemiFutures drops the continuation. We can instead make detach an explicit operation, parameterised optionally by an executor, and phase out implicit detach on destruction of SemiFuture.

Reviewed By: yfeldblum, andriigrynenko

Differential Revision: D20683652

fbshipit-source-id: 909d2ffcadbe71501edcddbf0847a9cb12aa3cce

Summary:
Migration from Future-returning executor-erasing collectX forms to SemiFuture-returning forms, that are less risky in particular with coroutines.

Earlier changes added collectXSemiFuture and collectXUnsafe as a migration path. This diff changes the behaviour of collectX to return SemiFuture.

Reviewed By: yfeldblum

Differential Revision: D20260537

fbshipit-source-id: 9e01536e4d9af68ff429f9fcd10aea4881d62041

Summary:
Currently `folly/memory/UninitializedMemoryHacks.h`'s `resizeWithoutInitialization()` is able to do its job for `std::vector<T>` and `std::string` only. This PR extends it to support `std::basic_string<T>`.

Effectively, replicated the machinery pattern used to achieve the same for any `std::vector<T>`.

Added test cases too, however the test executable is not being built by the `CMakeLists.txt` (had to enable it locally for testing.)

Is there any particular reason why these tests for `folly/memory/UninitializedMemoryHacks.h` are not built/run by default?
`folly/memory/test/UninitializedMemoryHacksODR.cpp`
`folly/memory/test/UninitializedMemoryHacksTest.cpp`
Pull Request resolved: https://github.com/facebook/folly/pull/1339

Reviewed By: Mizuchi

Differential Revision: D20603510

Pulled By: yfeldblum

fbshipit-source-id: dc92ca71cab6a76e2d93c2bbcb4a325bcd70468c

Summary:
On Windows the build artifacts cannot be easily run directly from the build
output directory without installing them.  The `$PATH` environment variable
needs to be set correctly so that the runtime library dependencies can be
found.

This updates the builder code to emit a `run.ps1` wrapper script in the build
output directory that sets `$PATH` to support running build artifacts directly
from the build directory.

Additionally, this updates the CMake-specific builder to set properly when
running the tests with `ctest`.

Reviewed By: wez

Differential Revision: D20688290

fbshipit-source-id: 5d0f4d685692bca7e37370bd88309cf7634d87f0

Summary:
This still requires support from EdenFS in order to do much
of use, but it takes us a step closer:

* Pull in cpptoml when building with Eden support
* On Windows, when we locate the `.eden` directory, load and parse
  the config file in order to determine the socket path
* If the EdenView constructor throws, treat it as a terminal error
  so that we don't fallback to the regular filesystem watcher.
  This is important because current EdenFS builds don't implement
  the journal thrift API endpoint yet.

Reviewed By: pkaush

Differential Revision: D20504752

fbshipit-source-id: 48bbad49f1641698aa7d7b85674e3ddf4d4e617d

Summary:
We have a global `--install-prefix` argument that can be used to set
the prefix for all projects, but that is only suitable if you are running with
sufficient privileges to install each of the deps to that location during the
build.  Cmake dependency resolution won't work from the build directory in that
situation; it can only see the final installed location and it will error out
if those files are not present, or link against the currently installed version
instead of the version we just built; not great!

This commit adds a project specific `--project-install-prefix` that can be used
on just the leaf project in a set of deps.  That sidesteps the dependency
concern because only the last stage is built in that mode.  This option
can technically be applied to an arbitrary set of projects, but in light
of the above, in practice it only makes sense to use it for the final
cmake project.  Only the CMakeBuilder respects this option.

In the watchman repo, this commit adjusts the autogen.sh script to allow
specifying the installation prefix; it defaults to `/usr/local` as you
might expect.

refs: https://github.com/facebook/watchman/issues/760

Reviewed By: yfeldblum

Differential Revision: D20674439

fbshipit-source-id: 52799dbd47f3c295e2d6469ee2b74cedeaa20138

Summary:
The math that figures out the biggest possible formatted
number, for a stack buffer size, computed 65 bytes for "binary" format:

```
    // #b: 0b prefix + 64 bytes = 65 bytes
```

But with that 2-byte "0b" prefix this of course sums to 66 bytes.

This turns out to be harmless, rather than causing a buffer overrun,
because the buffer size additionally reserves two more bytes for
"prefix shenanigans", which are redundant with the two prefix bytes
explicitly reserved for "0b" (but in this case were covering up the
off-by-one error).

I cleaned up a few things while I was there:

- I fixed the size computation for binary buffers, including
  making it use kMaxBinaryLength, which is based on `sizeof(uintmax_t)`,
  rather than hardcoding `64`.
- I removed the redundant prefix shenanigans reservation.
- I fixed the places that were reserving two bytes for a prefix
  (like "0x") right after asserting that no prefix is allowed,
  e.g. for chars and decimal numbers.
- I stopped reserving a byte for NUL, because these days this code
  uses folly::StringPiece and never NUL-terminates.

Reviewed By: yfeldblum

Differential Revision: D20694395

fbshipit-source-id: 6658675d9993d2fe223675db55c8040ddee143a8

Summary:
Not sure if this is a transient issue, but the URL we've been using to obtain boost has started to return 403 errors.
Switch to a sourceforge download link instead.

Refs: https://github.com/facebook/watchman/pull/797

Reviewed By: chadaustin

Differential Revision: D20739351

fbshipit-source-id: 47483c675d59201a410c9d2a8f6db0f63ea5da69

Summary:
My recent change to ensure that we were using python3 to launch everything failed on windows: the GH actions environment has `python.exe` in the path and it is python version 3.  There is no such thing as `python3` in that environment, although there is a `python2`.

Refs: https://github.com/facebook/watchman/pull/797

Reviewed By: chadaustin

Differential Revision: D20740411

fbshipit-source-id: 0e40590ccedc18e327ebb84901e2509588fdd0ff

Summary:
This diff is a more complete fix for the problem addressed
by D19561643, in which private inheritance caused an incorrect
inaccessibility error for a type that was derived via an alternate
construction from a template param.

Reviewed By: lnicco

Differential Revision: D20718329

fbshipit-source-id: 778d246d07030f6abb9e08e127a98d48c736a7d7