1. 07 Jul, 2020 2 commits
    • Sotirios Delimanolis's avatar
      Add FOLLY_OPENSSL_PREREQ macro function · 4d04d3d7
      Sotirios Delimanolis authored
      Summary:
      This diff introduces a new macro function to resolve whether the current OpenSSL version available is greater than or equal to the given (major, minor, fix) version tuple.
      
      We used the rules defined in the [OPENSSL_VERSION_NUMBER man page](https://www.openssl.org/docs/man1.1.1/man3/OPENSSL_VERSION_NUMBER.html), ie. nibbles for each of
      ```
       MNNFFPPS: major minor fix patch status
      ```
      
      We choose to ignore `patch` and `status` for simplicity and because we don't expect to build against such versions.
      
      The existing `FOLLY_OPENSSL_IS_110` variable already checks for greater than or equal to version number 1.1.0, so I didn't feel the need to modify it (nor the other variables). We can do some clean up in a future diff.
      
      Reviewed By: yfeldblum, mingtaoy
      
      Differential Revision: D22346692
      
      fbshipit-source-id: 156ee69ecd619de12319d7d63239f28c323820a4
      4d04d3d7
    • Kirk Shoop's avatar
      create explicit instantiations for folly::Unit · 3e9186f1
      Kirk Shoop authored
      Summary:
      move template instantiations for folly::Unit of several types into explicit instantiations in cpp files.
      
      `folly::Future<folly::Unit>`
      `folly::SemiFuture<folly::Unit>`
      `folly::Promise<folly::Unit>`
      `folly::SharedPromise<folly::Unit>`
      `folly::futures::detail::FutureBase<folly::Unit>`
      `folly::futures::detail::Core<folly::Unit>`
      `folly::Try<folly::Unit>`
      
      Reviewed By: yfeldblum, ericniebler
      
      Differential Revision: D22092858
      
      fbshipit-source-id: b8faa1c8aedfc193c17e2c3bc2e5b0f429ae6b0f
      3e9186f1
  2. 06 Jul, 2020 5 commits
    • Zeyi (Rice) Fan's avatar
      add retry flag to getdeps test · 9426d302
      Zeyi (Rice) Fan authored
      Summary: This commit adds a flag `--retry` to getdeps and teach it to run retry failed test. This allows us to still pass the tests when there are some flaky tests presents.
      
      Reviewed By: wez
      
      Differential Revision: D22291063
      
      fbshipit-source-id: 572af48a52ceb4a9abbf530cc0154ded0120c0de
      9426d302
    • Pranav Thulasiram Bhat's avatar
      Add Task<void> specialization for taskWait · f9dec043
      Pranav Thulasiram Bhat authored
      Summary:
      `folly::coro::blockingWait(...)` returns `void` for `Task<void>`.
      
      This diff implements a template specialization on taskWait for `Task<void>`
      
      Differential Revision: D22265598
      
      fbshipit-source-id: ff13d2b18b8706a6e8257ef706e407b315daf98a
      f9dec043
    • Pranav Thulasiram Bhat's avatar
      Faster collectAll(args...) · bd41a1ed
      Pranav Thulasiram Bhat authored
      Summary:
      The variadic implementation of collectAll uses the futures APIs, and is likely to be slow.
      
      This diff adds a fiber only version similar to fibers::foreach
      
      Reviewed By: yfeldblum
      
      Differential Revision: D22212232
      
      fbshipit-source-id: f71765c5411d2c6ca4388fe373baafad65f1b01c
      bd41a1ed
    • Zeyi (Rice) Fan's avatar
      avoid using relative path in fb_py_win_main · 18372331
      Zeyi (Rice) Fan authored
      Summary:
      After some experimenting, it is a little awkward if we want to specify a relative path based on the executable location. We'd need to add a bunch of path calculations to make it right, and I don't think the added complexity is really worth the effort.
      
      As a result, let's just remove the use of relative path, and if we ever want to ship a copy of Python distribution, we can place it under the same directory as the binary.
      
      Reviewed By: chadaustin
      
      Differential Revision: D22394180
      
      fbshipit-source-id: 86d27f6d16a03fe08826b5e5eafcef2a1c77997f
      18372331
    • Shai Szulanski's avatar
      fix clang-format nits · 5b5359fe
      Shai Szulanski authored
      Reviewed By: yfeldblum
      
      Differential Revision: D22393040
      
      fbshipit-source-id: 920a7dfbf1d4334aecb475add96d3799c89aa211
      5b5359fe
  3. 03 Jul, 2020 2 commits
    • Giuseppe Ottaviano's avatar
      IOBufQueue::moveAsValue() · fc6f6e43
      Giuseppe Ottaviano authored
      Summary: Added for symmetry with `IOBuf::cloneAsValue()`
      
      Reviewed By: luciang
      
      Differential Revision: D21737695
      
      fbshipit-source-id: 40f5695565ffc7d7c389c9953bffc3ab85508ed1
      fc6f6e43
    • Lukas Piatkowski's avatar
      eden/scm: provide getdeps.py way of building eden/scm on GitHub · 63977f90
      Lukas Piatkowski authored
      Summary:
      In order to do what the title says, this diff does:
      1. Add the `eden/oss/.../third-party/rust/.../Cargo.toml` files. As mentioned in the previous diff, those are required by GitHub so that the third party dependencies that are local in fbsource are properly defined with a "git" dependency in order for Cargo to "link" crates properly.
      2. Changes to `eden/scm/Makefile` to add build/install commands for getdeps to invoke. Those command knowing that they are called from withing getdeps context they link the dependencies brought by getdeps into their proper places that match their folder layout in fbsource. Those Makefile commands also pass a GETDEPS_BUILD env to the setup.py invocations so that it knows it is being called withing a getdeps build.
      3. Changes to `eden/scm/setup.py` that add "thriftasset" that makes use of the getdeps.py provided "thrift" binary to build .py files out of thrift files.
      4. Changes to `distutils_rust` to use the vendored crates dir provided by getdeps.
      5. Changes to `getdeps/builder.py` and `getdeps/manifest.py` that enable more fine-grained configuratior of how Makefile builds are invoked.
      6. Changes to `getdeps/buildopts.py` and `getdeps/manifest.py` to disable overriding PATH and pkgconfig env, so that "eden/scm" builds in getdeps using system libraries rather than getdeps-provided ones (NOTE: I've tried to use getdeps provided libraries, but the trickiest bit was that Rust links with Python, which is currently not providable by getdeps, so if you try to build everything the system provided Python libraries will collide with getdeps provided ones)
      7. Added `opensource/fbcode_builder/manifests/eden_scm` for the getdeps build.
      
      Reviewed By: quark-zju
      
      Differential Revision: D22336485
      
      fbshipit-source-id: 244d10c9e06ee83de61e97e62a1f2a2184d2312f
      63977f90
  4. 02 Jul, 2020 3 commits
  5. 01 Jul, 2020 4 commits
    • Zeyi (Rice) Fan's avatar
      make fb_py_win_main to dynamically find Python3.dll · 59701670
      Zeyi (Rice) Fan authored
      Summary:
      In EdenFS's latest Windows package. We are seeing DLL import errors coming from `asyncio` as it requires a system native module `_overlapped.pyd`.
      
      The underlying cause is because when we build EdenFS CLI on Sandcastle, we are linking with Python 3.6.2. The Python36.dll shipped with the EdenFS package is also coming from that version.
      
      However, on Windows laptop. We have Python 3.6.3. Since we are not shipping the Python system libraries with us. It uses the libraries installed in the system, and it attempts to import the `_overlapped.pyd` located at `C:\Pythone36\DLLs\`. This version is compiled against Python 3.6.3, which is incompatible with the Python36.dll we are using.
      
      ----
      
      To resolve this, we need either ship an embedded copy of Python along with EdenFS, or teach EdenFS to use the Python distribution installed in the system. This commit tweaks the executable we prepend to the archive created with zipapp to locate `Python3.dll` dynamically. This allows us to use the Python installed in the system so we can avoid the version mismatch issue.
      
      With this setup, we can also be shipping an embedded Python version along with EdenFS, and the Python loader can look for that path. This is demonstrated with the relative DLL loading `..\python`.
      
      In theory, we can have a package structure like this:
      
      ```
      .
      ├── python
      │   ├── ....
      │   └── python3.dll
      └── bin
          ├── ...
          ├── edenfsctl.exe
          └── edenfs.exe
      ```
      
      Reviewed By: xavierd
      
      Differential Revision: D22325210
      
      fbshipit-source-id: 96a3f9503e7865a5f9d95710ff13f019afcf04f1
      59701670
    • Jon Maltiel Swenson's avatar
      Remove rsocket-cpp dependency from OSS projects depending on fbthrift · 0c4dd654
      Jon Maltiel Swenson authored
      Summary: fbthrift no longer depends on rsocket-cpp
      
      Reviewed By: simpkins
      
      Differential Revision: D22275231
      
      fbshipit-source-id: c1f217f1ce97591b8ebca002bf8ae3af701be641
      0c4dd654
    • Yedidya Feldblum's avatar
      Small benchmark for coarse_steady_clock · c3e79e82
      Yedidya Feldblum authored
      Summary: [Folly] Small benchmark for `coarse_steady_clock`.
      
      Differential Revision: D22320499
      
      fbshipit-source-id: dbd93e886b9e984d7469288258eefaa4a3fafb10
      c3e79e82
    • Giuseppe Ottaviano's avatar
      Fix overflow in EliasFanoReader · cf316095
      Giuseppe Ottaviano authored
      Summary:
      The upper bitvector size (in bits) can exceed the domain of `SizeType` even if the list otherwise respects the contract.
      
      Also improve some of the tests.
      
      Reviewed By: yfeldblum, philippv
      
      Differential Revision: D22304614
      
      fbshipit-source-id: bf846730c59451457d3e2d3cf14d90e3df012e0b
      cf316095
  6. 30 Jun, 2020 7 commits
    • Matthieu Martin's avatar
      Add addFiberRemote · 77a38ddb
      Matthieu Martin authored
      Summary: Complete support of the FiberManager::addTask*
      
      Reviewed By: pranavtbhat
      
      Differential Revision: D22305146
      
      fbshipit-source-id: 94e4a02f754b306d8b5fe4994832a7e2dc2cf4a5
      77a38ddb
    • Matthieu Martin's avatar
      Specify return type of async:: functions · 6284a28e
      Matthieu Martin authored
      Summary: Simplify API understanding for code reader
      
      Reviewed By: pranavtbhat
      
      Differential Revision: D22272546
      
      fbshipit-source-id: 341625ed9079f3bc4aee09851eadc02fd05b2434
      6284a28e
    • Phil Willoughby's avatar
      support clang compiler · 06786016
      Phil Willoughby authored
      Summary: `clang-cl` and some other invocations of `clang` on windows define `_MSC_VER`. However `clang` does not implement `#pragma optimize` so we need to use the other code-path here.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D22301661
      
      fbshipit-source-id: e51656497a5fc109bf04f7daac82f824ba8c5c8e
      06786016
    • Sergey Anpilov's avatar
      Add folly::coro::filter(AsyncGenerator) · 925ab325
      Sergey Anpilov authored
      Summary: Filtering is one of the basic stream operations and it would be convenient to have it for AsyncGenerators
      
      Reviewed By: yfeldblum
      
      Differential Revision: D22136080
      
      fbshipit-source-id: ca6a233a8c1aeddb63899aeaba8ca4c1b6dfa535
      925ab325
    • Swaroop Manjunath's avatar
      Eliminate need for reading upper bound at construction. · c9c55642
      Swaroop Manjunath authored
      Summary:
      # Note: This is a resubmission after reverting this change.
      The failing `DCHECK` has been fixed with an explicit cast to `size_t` to accommodate long lists.
      
      # Summary
      
      This diff introduces the following changes to the EliasFanoReader implementation.
      
      - `EliasFanoReader` no longer requires knowledge of the last value in the list at construction time. This removes the need to access the last byte of the encoded list.
      - Operations in `UpperBitsReader` are now responsible for ensuring validity.
      - Removes constraint that the last set bit in upperBits must be in the last byte of the encoded list.
      
      In addition, the diff also extends the unit-tests for Elias Fano coding to add arbitrary extensions to the upperBound at construction to ensure that additional 0-blocks at the end of the list do not affect the behavior of the reader.
      
      Reviewed By: ot, luciang
      
      Differential Revision: D22211304
      
      fbshipit-source-id: 0dbe904c9fd8cd9568a480355e5e6a4525922966
      c9c55642
    • Matthieu Martin's avatar
      Strictify async_inner_type · 01da43c2
      Matthieu Martin authored
      Summary: Only accept Async input now
      
      Reviewed By: yfeldblum
      
      Differential Revision: D22268635
      
      fbshipit-source-id: cd974af7a7a3adc25b3c90029efe4c6080c3739a
      01da43c2
    • Tristan Rice's avatar
      folly: disable zero copy in AsyncSSLSocket to avoid memory leak · 0ff36eaf
      Tristan Rice authored
      Summary: When zero copy is enabled on AsyncSSLSocket every IOBuf written via writeChain is accumulated in idZeroCopyBufInfoMap_ and never freed.
      
      Differential Revision: D22272411
      
      fbshipit-source-id: 576d16ddb4e5c7c19209b77f7346949f10fbf32d
      0ff36eaf
  7. 29 Jun, 2020 3 commits
    • Matthieu Martin's avatar
      Async interface for addTaskRemoteFuture · 7e6a92c1
      Matthieu Martin authored
      Summary:
      This provides a set of three functions under folly::fibers::async, to schedule then wait for work on a remote fiber manager.
      The functions cover those 3 (legitimate) needs:
      
      - Blocking current thread: `T executeOnRemoteFiberAndWait`
      - Blocking current fiber: `Async<T> executeOnRemoteFiber`
      - Awaitable (barebone version for now): `Future<T> addFiberRemoteFuture`
      
      See code docblock for more details
      
      Reviewed By: pranavtbhat
      
      Differential Revision: D22252598
      
      fbshipit-source-id: add9c23f5bdacb233e47f30b2639d376027167e3
      7e6a92c1
    • Dan Melnic's avatar
      More io_uring file read/write tests · 5b8e0513
      Dan Melnic authored
      Summary: More io_uring file read/write tests
      
      Reviewed By: kevin-vigor
      
      Differential Revision: D22236012
      
      fbshipit-source-id: 37abd398d07af2bb41cfe3b3b6dfeb40410554d5
      5b8e0513
    • Shrikrishna Khare's avatar
      Fix OpenNSA 6.5.19 hash as Brcm modified the tarball · e678005f
      Shrikrishna Khare authored
      Summary:
      Koray reported issue with OpenNSA while using build VM and observed that we
      don't hit the issue on devserver thanks to downloading the lfs cached file.
      
      The root cause:
      
      Broadcom added License notice and uploaded new 6.5.19 with same name.
      This changed the hash though - causing fbcode builder to complain.
      Fix it by adjusting the hash.
      
      See the first line in the recursive diff of before vs. after: P134467317
      
      (the broken symlinks is an unrelated existing issue, which we have reported to
      Broadcom).
      
      Reviewed By: bkoray
      
      Differential Revision: D22280971
      
      fbshipit-source-id: 781079df426f83901509225156cf77a3966d3301
      e678005f
  8. 27 Jun, 2020 1 commit
    • Matthieu Martin's avatar
      Move addFiberFuture from detail:: · 0c27031c
      Matthieu Martin authored
      Summary:
      After pursuing adoption and more discussion, it seems clear that this was probably trying to push "hiding fiber details" one step too far.
      See comment in code for more details
      
      Reviewed By: yfeldblum, pranavtbhat
      
      Differential Revision: D22259585
      
      fbshipit-source-id: 798cada6527332faf7e4db9942a964d6b7cca1e4
      0c27031c
  9. 26 Jun, 2020 2 commits
    • Pranav Thulasiram Bhat's avatar
      Switch await into a callable object · 10dd9f4b
      Pranav Thulasiram Bhat authored
      Summary: This diff converts `await` into a callable object.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D22175597
      
      fbshipit-source-id: 689e11c62161a2e4b82699bf52835e92c96beff7
      10dd9f4b
    • Pranav Thulasiram Bhat's avatar
      Mark lambdas in addFiber(Future) mutable · b18832a6
      Pranav Thulasiram Bhat authored
      Summary:
      If `func` is mutable, then the lambda applying `init_await` must be mutable too in order to call `func`'s non-const () operator.
      
      This diff marks the `init_await` applying lambdas mutable.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D22238824
      
      fbshipit-source-id: 20f202b121edd0bdfcf7a6f707a4f1d4df0181d8
      b18832a6
  10. 25 Jun, 2020 2 commits
    • Songqiao Su's avatar
      (folly/coro) using atomic_compare_exchange_weak_explicit in coro::Baton · df460658
      Songqiao Su authored
      Summary:
      to avoid hit TSAN issue when using coro::Baton
      
      (Note: this ignores all push blocking failures!)
      
      Reviewed By: lewissbaker
      
      Differential Revision: D22192037
      
      fbshipit-source-id: ee35179be79deb1d3b79b7b432c8a65c793f7d8f
      df460658
    • Lewis Baker's avatar
      Make sure coroutine final_suspend() methods are declared noexcept · 8a3d5b9c
      Lewis Baker authored
      Summary:
      The C++20 specification for coroutines included a late change that
      now requires the `co_await promise.final_suspend();` expression to
      not be potentially throwing (ie. it needs to be declared `noexcept`).
      
      This change updates all coroutine types in folly to make sure that
      `final_suspend()` methods and the methods on awaitables returned from
      these methods are all declared `noexcept` so that folly can be used
      with compilers that enforce this requirement.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D22203834
      
      fbshipit-source-id: 4a5bbcbad644349d2ff0b08f6460fa13c3165aec
      8a3d5b9c
  11. 24 Jun, 2020 9 commits
    • Pranav Thulasiram Bhat's avatar
      Async Readme · df45ceb8
      Pranav Thulasiram Bhat authored
      Summary: This diff adds a README for the async annotation library.
      
      Reviewed By: andriigrynenko, A5he
      
      Differential Revision: D22080510
      
      fbshipit-source-id: 2b1800fd44596e0e4bddff8a321a88fcd9304739
      df45ceb8
    • Mark Santaniello's avatar
      Simplify Arena destructor · aeb4ecf8
      Mark Santaniello authored
      Summary: Use the intrusive list's clear_and_dispose API to simplify things somewhat.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D22199152
      
      fbshipit-source-id: fb6ec4e206660bc7628ddd87edba3698d8a2af6d
      aeb4ecf8
    • Pranav Thulasiram Bhat's avatar
      FiberManager - executeOnFiber · 4485ef0c
      Pranav Thulasiram Bhat authored
      Summary:
      This diff implements a new API to schedule a function on a new fiber and block the existing fiber till completion.
      
      The API should be used sparingly to deal with fiber-stack-overflow issues
      
      Reviewed By: A5he
      
      Differential Revision: D22074021
      
      fbshipit-source-id: 893e578b1acd6dd5cca34187fc8431202cdb73da
      4485ef0c
    • Pranav Thulasiram Bhat's avatar
      collectAllFibers and collectFibers · ac28a012
      Pranav Thulasiram Bhat authored
      Summary:
      This diff implements a wrapper around `fibers::collectAll` to accept annotated functors.
      
      It also implements version of `collect` that accepts variadic arguments, and returns a tuple.
      
      Reviewed By: A5he
      
      Differential Revision: D22142416
      
      fbshipit-source-id: dab320f7e1718c3872c54e168c24ea4bc39051d5
      ac28a012
    • Pranav Thulasiram Bhat's avatar
      FiberManager - addFiber · 98759027
      Pranav Thulasiram Bhat authored
      Summary:
      This diff implements addFiber, a wrapper around fiberManager.addTask.
      
      It should mainly be used to 'fire-and-forget' a fiber.
      
      Reviewed By: A5he
      
      Differential Revision: D22073924
      
      fbshipit-source-id: 52516e86b7b03a7bc59166fae4db79f60cddffb7
      98759027
    • Pranav Thulasiram Bhat's avatar
      FiberManager - addFiberFuture and blockingWait · 7c81dc02
      Pranav Thulasiram Bhat authored
      Summary:
      This diff has two goals:
      - To provide an API that allows users to schedule async annotated functors to run on fibers (thereby removing the need to use `init_await` at the top of the stack)
      - To provide a boilerplate-free entry point into fiber code that should work for a majority of testing/benchmarking interactions with fiber manager.
      
      The diff implements two APIs:
      -addFiberFuture: wrapper around `fiberManager.addTaskFuture`. The differing name is to mainly avoid namespace collisions and avoid confusion with `coro::Task`
      - blockingWait: Fiber entry point helper
      
      Reviewed By: A5he
      
      Differential Revision: D22071005
      
      fbshipit-source-id: 24ceea005d32f6e2c84b4599042703207504bd0f
      7c81dc02
    • Shrikrishna Khare's avatar
      fbcode_builder: getdeps: Update OpenNSA to 6.5.19 · f8ca5b8f
      Shrikrishna Khare authored
      Summary: 6.5.19 is now available, switch OSS to pick that instead of old 6.5.17.
      
      Reviewed By: rsunkad
      
      Differential Revision: D22199286
      
      fbshipit-source-id: 231346df8d2f918d2226cfe17b01bde12c18a5a7
      f8ca5b8f
    • Lukas Piatkowski's avatar
      mononoke/configerator structs: change the OSS folder where configerator structs are saved · d0c01e87
      Lukas Piatkowski authored
      Summary:
      Due to Thrift design of "include" statements in fbcode the thrift structures has to be contained in folders that are identical to the folder layout inside fbcode.
      
      This diff changes the folder layout on Cargp.toml files and in fbcode_builder, there will be a next diff that changes this for ShipIt as well.
      
      Reviewed By: ikostia
      
      Differential Revision: D22208707
      
      fbshipit-source-id: 65f9cafed2f0fcc8398a3887dfa622de9e139f68
      d0c01e87
    • Andrii Grynenko's avatar
      Make sure global CPU executor is never joined on an application thread · 3b8891d6
      Andrii Grynenko authored
      Reviewed By: LeeHowes
      
      Differential Revision: D22181192
      
      fbshipit-source-id: 279c56398cd7e4c9274bc43c3564062588c9c81a
      3b8891d6