1. 05 Feb, 2021 7 commits
    • Dan Melnic's avatar
      Add support for io_uring min capacity · 737e6144
      Dan Melnic authored
      Summary: Add support for io_uring min capacity. This should allow us to create io_uring instances even memory is fragmented and the kernel cannot allocate contiguous physical memory for capacity CQ entries (SQ is usually much smaller).
      
      Reviewed By: kevin-vigor
      
      Differential Revision: D26252442
      
      fbshipit-source-id: f4e3bd4f00d21321b77d4e1452400689dbca1073
      737e6144
    • Yedidya Feldblum's avatar
      XLOG_FIRST_N · 7e4c8713
      Yedidya Feldblum authored
      Summary: Add `XLOG_FIRST_N` log function.
      
      Reviewed By: simpkins
      
      Differential Revision: D26184283
      
      fbshipit-source-id: 23081f759897bbf7949ce603f8e5edb29869b40d
      7e4c8713
    • Dan Melnic's avatar
      Remove FOLLY_UNLIKELY from IoUringBackend::eb_event_base_loop · 4eaba962
      Dan Melnic authored
      Summary: Remove FOLLY_UNLIKELY from IoUringBackend::eb_event_base_loop
      
      Reviewed By: yfeldblum
      
      Differential Revision: D26272113
      
      fbshipit-source-id: de280347908e5448cc97ffc1e1be3100cf6cd4b1
      4eaba962
    • Yedidya Feldblum's avatar
      reexport suspend_always, suspend_never · 317e8753
      Yedidya Feldblum authored
      Summary: Reexport `suspend_always`, `suspend_never` from `folly/experimental/coro/Coroutine.h`, which wraps inclusion of `experimental/coroutine`.
      
      Differential Revision: D26215623
      
      fbshipit-source-id: 5dbe5dba67e0c4dcb5a54b00753937c39833d3a7
      317e8753
    • Yedidya Feldblum's avatar
      rely on C++17 syntactic copy elision in DelayedInit · cc15e578
      Yedidya Feldblum authored
      Summary: Rely on C++17 syntactic copy elision in `DelayedInit` to simplify and write `try_emplace` in terms of `try_emplace_with`.
      
      Reviewed By: praihan
      
      Differential Revision: D26260351
      
      fbshipit-source-id: 590fdfed55ee325ed8a8ae4fa68c1db0bebf1545
      cc15e578
    • Yedidya Feldblum's avatar
      move and revise fmt/compile.h portability · b93c8daa
      Yedidya Feldblum authored
      Summary: Move the libfmt portability code into `folly/portability/` and revise it only to mutate folly's namespace.
      
      Reviewed By: Orvid
      
      Differential Revision: D26182824
      
      fbshipit-source-id: e0b72f523dfeac7ff7799ff2104a1146032dfbf3
      b93c8daa
    • Yedidya Feldblum's avatar
      let coro headers always be buildable · 4ab51101
      Yedidya Feldblum authored
      Summary: Let folly coro headers always be buildable whether or not coroutines are available. They may just be empty apart from transitive includes.
      
      Reviewed By: iahs
      
      Differential Revision: D26191449
      
      fbshipit-source-id: 5865ce5edb13f97624b9f60c45c045886fdf0bfb
      4ab51101
  2. 04 Feb, 2021 4 commits
    • Misha Shneerson's avatar
      python event loop consumes one task at a time from C++ notification queue · f133aef5
      Misha Shneerson authored
      Summary:
      Currently, when C++ threads communicate with python's event loop, they place
      the items on the queue, notify the event loop, and when even loops starts
      running, they would consume all items from this queue at once.
      The downside of this approach is that we are unable to deprioritize the
      upstream tasks to yield to internal python tasks. However, such ability to yield
      is fundamental to how Thrift approaches load shedding under overload.
      In this diff we change the approach to take only a single item from the queue at a
      time. Notice that if queue is not emptied, the consumer will notify the event loop to
      consume from this queue on the next iteration of the loop.
      
      Reviewed By: zhxchen17
      
      Differential Revision: D26228933
      
      fbshipit-source-id: 827d21df12b65d518109151cb150cd45e513da57
      f133aef5
    • Yedidya Feldblum's avatar
      tweaks to DelayedInit · d8ecccad
      Yedidya Feldblum authored
      Summary:
      Tweaks to `folly::DelayedInit`:
      * No potential for ambiguity in placement-new by using `::new` and casting the address to `void*`.
      * Helper functions `slot` and `store` simplify `try_emplace_with` and `try_emplace`.
      * `try_emplace_with` and `try_emplace` are no longer `const`-qualified, removing the need for `mutable` qualifiers.
      
      The emplacement members should not be `const`-qualified since they are not semantically non-modifying. The `const` qualifier signals semantic non-modification. While it is the case that there are customs around `const` qualifiers and concurrency, these customs are not the core meaning of `const`.
      
      Reviewed By: praihan
      
      Differential Revision: D25919779
      
      fbshipit-source-id: 95e5c4b4abf5285fbb00dddda1192998baf6ee59
      d8ecccad
    • Maged Michael's avatar
      ConcurrentHashMap: Catch use-after-destruction user bugs · b9cc811a
      Maged Michael authored
      Summary:
      The change aims to make it easier to catch user bugs of using the map during or after its destruction in optimized modes without ASAN.
      
      The changes are:
      - Clear the buckets_ and chunks_ pointers in the corresponding destructors.
      - Add DCHECK-s of buckets_ (or chunks_) in other member functions when non-null values are expected.
      
      Reviewed By: davidtgoldblatt
      
      Differential Revision: D26232735
      
      fbshipit-source-id: 14f0c11e10997773efb78c7275db99932dd19703
      b9cc811a
    • Nick Cooper's avatar
      RFC: Add direct support for relative benchmarking to the folly/Benchmark library. · 88ee330d
      Nick Cooper authored
      Summary:
      When modifying code that lacks alternative variations, there is only a 'HEAD' benchmark - so it can be useful to keep a version of these results for latter printing in the format of a relative benchmark.
      
      Add a new flag to simplify this use-case,
      --bm_relative_to: A flag taking the path to a JSON-verbose formatted dump, which will be used when printing (in non-JSON-mode) to produce output relative to the provided prior dump.
      
      Usage:
      - generate a JSON benchmark dump, use "--bm_json_verbose"
        $ your_benchmark_binary --benchmark --bm_json_verbose  old-json
      - run a benchmark printing a comparison to an old dump:
        $ your_benchmark_binary --benchmark --bm_relative_to old-json
      
      Note: You can also use bm_json_verbose and bm_relative_to at the same time, this will print relative results and update the JSON dump
      
      Reviewed By: luciang
      
      Differential Revision: D26214397
      
      fbshipit-source-id: 8134b74604b7bb5037934205e8eee5e3ffc27f15
      88ee330d
  3. 03 Feb, 2021 7 commits
    • Pranjal Raihan's avatar
      Use compact_once_flag in DelayedInit only if size can be reduced · d6d84f34
      Pranjal Raihan authored
      Summary: If `T` is greater than 4-byte aligned, then `compact_once_flag` brings no size improvements. We can use `once_flag` in these cases, which has a better mutex (`folly::SharedMutex` rather than `folly::MicroLock`).
      
      Reviewed By: yfeldblum
      
      Differential Revision: D26214262
      
      fbshipit-source-id: ea15376ea892df42cc9d3a6eac3da458fe0564a8
      d6d84f34
    • Orvid King's avatar
      Back out "Fix definition of folly::remainder for android" · eb1954c7
      Orvid King authored
      Summary: Original commit changeset: 149b780350bf
      
      Reviewed By: zaxy78
      
      Differential Revision: D26222637
      
      fbshipit-source-id: 29bc017be6619165cbd497c27c7843fd95e345b1
      eb1954c7
    • Andrew McFague's avatar
      folly: Backout changes to folly::DelayedInit and pass in a reference via folly::ConcurrentLazy · ca1e1523
      Andrew McFague authored
      Summary: This violates some expectations for how this function should be called, so explicitly allow this to be passed as a reference.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D26215425
      
      fbshipit-source-id: 29cff0407ec10ab61970d32da083110d1b609fbb
      ca1e1523
    • Orvid King's avatar
      Fix definition of folly::remainder for android · 57c897d3
      Orvid King authored
      Summary: Non-uclibc android wasn't getting folly::remainder defined, so fall back to the STL in that case.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D26196078
      
      fbshipit-source-id: 149b780350bf4aaf6917ecce5e4718c91815c744
      57c897d3
    • Nick Cooper's avatar
      Add usage description to BenchmarkCompare · 8f284aae
      Nick Cooper authored
      Summary: As above.
      
      Reviewed By: luciang
      
      Differential Revision: D26213601
      
      fbshipit-source-id: e2eb639d0059e20cfd6828be669c62491e5ad9c4
      8f284aae
    • Yedidya Feldblum's avatar
      coro safe_point · 229a0c46
      Yedidya Feldblum authored
      Summary:
      Add a new concept of a safe-point to folly::Task<...> with implementation-defined semantics.
      
      Example usage:
      ```lang=c++
      Task<> co_something() {
        while (is_work_available()) {
          co_await co_safe_point;
          do_work();
        }
      }
      ```
      
      Previously, a coroutine function which would cancel itself if cancellation is requested would need to spell cancellation pass-through a bit less elegantly:
      ```lang=c++
      Task<> co_something() {
        while (is_work_available()) {
          auto token = co_await co_current_cancellation_token;
          if (token.isCancellationRequested()) {
            co_yield co_cancel;
          }
          do_work();
        }
      }
      ```
      
      Initial semantics include:
      * If cancellation has been requested, finish the coroutine with cancellation.
      * Otherwise, continue the coroutine.
      
      In the future, it may be wise to add additional behavior:
      * If the coroutine has been running for too long since its previous resumption, reschedule it.
      
      Reviewed By: aary
      
      Differential Revision: D25861265
      
      fbshipit-source-id: 6da896f1529d652bfe222cf3a70e9dbe19778510
      229a0c46
    • Yedidya Feldblum's avatar
      use LOG_FIRST_N in MemoryIdler · bdbdd4b3
      Yedidya Feldblum authored
      Summary: Use `LOG_FIRST_N` in `MemoryIdler` rather than reimplement it.
      
      Reviewed By: praihan
      
      Differential Revision: D26185702
      
      fbshipit-source-id: c1045ef8bf6aabefa4c0806c35c156539d6e594f
      bdbdd4b3
  4. 02 Feb, 2021 1 commit
    • Dan Melnic's avatar
      Delay registration of the timer and signal fds until running the loop first time · 536086dc
      Dan Melnic authored
      Summary:
      Delay registration of the timer and signal fds until running the loop first time
      On some Linux kernel versions, io_uring will remove any fd registrations on thread exit. So creating the backend in a thread that exits later will cause us not to receive timer and signal fd events.
      
      Reviewed By: kevin-vigor
      
      Differential Revision: D26200805
      
      fbshipit-source-id: dc3025964deb3cf87bf3e5a27141abcdb8698caf
      536086dc
  5. 01 Feb, 2021 2 commits
  6. 30 Jan, 2021 2 commits
    • Misha Shneerson's avatar
      python event loop consumes one task at a time from C++ notification queue · 7c9f69f9
      Misha Shneerson authored
      Summary:
      Currently, when C++ threads communicate with python's event loop, they place
      the items on the queue, notify the event loop, and when even loops starts
      running, they would consume all items from this queue at once.
      The downside of this approach is that we are unable to deprioritize the
      upstream tasks to yield to internal python tasks. However, such ability to yield
      is fundamental to how Thrift approaches load shedding under overload.
      In this diff we change the approach to take only a single item from the queue at a
      time. Notice that if queue is not emptied, the consumer will notify the event loop to
      consume from this queue on the next iteration of the loop.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D26163397
      
      fbshipit-source-id: 48cd6cb57c48ea67dce4eb5f0de6db5b0feb75ac
      7c9f69f9
    • William McDonald's avatar
      Add conversion from const folly::Optional to std::optional · 19a84207
      William McDonald authored
      Summary:
      Previous version does not compile when converting from `const folly::Optional`
      to `std::optional`. This happens in a typical `for (const auto& x : vec)` where
      `x` is or contains `folly::Optional`.
      
      Add a new `const` version in case people case about `Value&` ctor as opposed to
      `const Value&`.
      
      Reviewed By: iahs
      
      Differential Revision: D26137442
      
      fbshipit-source-id: 649be596b67a01c6d1f4be341aad65f1099b74a1
      19a84207
  7. 29 Jan, 2021 3 commits
  8. 28 Jan, 2021 5 commits
    • Lee Howes's avatar
      Privately expose DefaultCPUExecutor as a form of InlineExecutor · cbb8cd2a
      Lee Howes authored
      Summary: DefaultCPUExecutor is the same as InlineExecutor, but hidden to avoid instantiating it to allow filtering on type, primarily in coro::task.
      
      Reviewed By: mpark, terrelln
      
      Differential Revision: D26077566
      
      fbshipit-source-id: 06704134f2446727cbb9f093df0b1954b46d2e49
      cbb8cd2a
    • Pranjal Raihan's avatar
      Fix MicroLock data storage when it is not word/half-word aligned · f60b081a
      Pranjal Raihan authored
      Summary:
      The current data storage API code assumes that the MicroLock is 4-byte aligned and that the lock lives in the lowest byte of `word()`. This assumption is incorrect (see added test). We should instead rely on `baseShift()`.
      
      This API is only used for `folly::compact_once_flag` which is new so low chance something broke.
      
      Reviewed By: andriigrynenko, davidtgoldblatt
      
      Differential Revision: D26091484
      
      fbshipit-source-id: c6e6e9aa7a963cada16e0ef7783fcc6bf6d2aed6
      f60b081a
    • Pranjal Raihan's avatar
      Disable MSAN for MicroLock functions · db4e040d
      Pranjal Raihan authored
      Summary: Even though `folly::MicroLock` is 1 byte, `detail::Futex<>` (aka `std::atomic<uint32_t>`) operates on 4 bytes. So 3 bytes are accessed but not used in any way. So we need to disable memory sanitizer.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D26082888
      
      fbshipit-source-id: fe585b60c96f48c02df69d182bd75a8eccde2a42
      db4e040d
    • Mingtao Yang's avatar
      Add ability to hint AsyncTransport to drop certificate · 6477caf1
      Mingtao Yang authored
      Summary:
      Many protocols often only require certificate information at the beginning of
      the connection (for authentication and authorization). Often, the
      protocol implementation will itself store its own representation of a transport
      identity.
      
      Certain certificates, such as X509 certificates, may be large and unnecessary
      to hold onto during the entirety of a connection past the initial connection
      establishment / handshaking phase. It is desirable to hint to transport
      implementations when certificate information is no longer needed.
      
      Reviewed By: AjanthanAsogamoorthy
      
      Differential Revision: D26031748
      
      fbshipit-source-id: de5164acfc141755debc0aa4f36474b1c7fd3109
      6477caf1
    • Philip Pronin's avatar
      skip unmodified keys in dynamic::merge_diff · 2a14eb88
      Philip Pronin authored
      Summary:
      `dynamic::merge_diff` currently fully replicates `target` within the
      patch.
      
      Reviewed By: psjanani, luciang
      
      Differential Revision: D26100408
      
      fbshipit-source-id: f6800e665281d87393fed0bb48bda888e24bbcfc
      2a14eb88
  9. 27 Jan, 2021 4 commits
    • Francesco Zoffoli's avatar
      atomic_wait expects the old value, not the new one · 141f44cb
      Francesco Zoffoli authored
      Summary: The linked paper, the implementation and the tests use the parameter as the old value, but the parameter name is called `expected`.
      
      Reviewed By: aary
      
      Differential Revision: D25975108
      
      fbshipit-source-id: bb67a95b057973e31512e9f972e1646e5a0f7d77
      141f44cb
    • Cameron Pickett's avatar
      Fix bug in folly coro GtestHelpers · 65c55155
      Cameron Pickett authored
      Differential Revision: D26084514
      
      fbshipit-source-id: ab7e94bb0f0da04d60af7112f34c7c221125feb7
      65c55155
    • Pedro Eugenio Rocha Pedreira's avatar
      Add getdeps.py manifest · d141da43
      Pedro Eugenio Rocha Pedreira authored
      Summary: Adding simple getdeps.py manifest for f4d.
      
      Reviewed By: amitkdutta
      
      Differential Revision: D25842791
      
      fbshipit-source-id: eabc33ec526c454f301f8cd401156f6250e5eaa1
      d141da43
    • Dan Melnic's avatar
      Fix flakey test · c21a0011
      Dan Melnic authored
      Summary: Fix flakey test
      
      Reviewed By: kevin-vigor
      
      Differential Revision: D26090290
      
      fbshipit-source-id: f03d6abf389815993e8742d5c27c2ccbcb13949a
      c21a0011
  10. 26 Jan, 2021 5 commits
    • Chad Austin's avatar
      substitute surrogates in log file (#8076) · 3aa8f275
      Chad Austin authored
      Summary:
      Pull Request resolved: https://github.com/facebookincubator/resctl/pull/8076
      
      D26025779 (https://github.com/facebook/folly/commit/4731bb0c37cc39a7a689a8369c9564eddeae146c) may have broken the getdeps build when logging
      surrogates. Use errors=surrogateescape to try to avoid that.
      
      Reviewed By: danobi
      
      Differential Revision: D26079717
      
      fbshipit-source-id: 98d938bfced417e0b39a6dcddb241225ffb69c06
      3aa8f275
    • Alexey Spiridonov's avatar
      Stop trying to build on Darwin, Windows · e8955d38
      Alexey Spiridonov authored
      Reviewed By: nilesr
      
      Differential Revision: D26059442
      
      fbshipit-source-id: d323ab38d7171b344c5a15863ec8d0a70b877853
      e8955d38
    • Pranjal Raihan's avatar
      Use compact_once_flag for DelayedInit · 1ab6a01f
      Pranjal Raihan authored
      Summary: Use the 1 byte version of `once_flag` such that there is no more memory overhead than `Optional`.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D25909346
      
      fbshipit-source-id: 1bb2f41200917ba58b20815bfafe3314137e9680
      1ab6a01f
    • Pranjal Raihan's avatar
      Refactor MicroLock and remove its slots · 6808ac5a
      Pranjal Raihan authored
      Summary:
      `folly::MicroLock` is actually 4 locks in one. Turns out that it's not very useful considering there is not a single use in fbcode. Instead `MicroLock` should allow a nicer API for exposing the 6 unused bits as user data.
      
      By limiting the lock to one slot (least significant 2 bits), we can make the assumption that the 6 contiguous most significant bits are user data. This means that all the masking logic can go away (see `slotMask` and `LockGuardWithDataSlots`).
      
      Now `lockAndLoad`, `unlockAndStore`, `LockGuardWithData` "just work" with needing to worry about user data interfering with the slots used for locking.
      
      Reviewed By: davidtgoldblatt
      
      Differential Revision: D25909068
      
      fbshipit-source-id: 3f050fa7393b40cff949ed6e30b691d7f40d0edf
      6808ac5a
    • Xiangyu Bu's avatar
      Fix typo about MutableStringPiece. · 3c764a2d
      Xiangyu Bu authored
      Summary:
      This diff corrects two references to `MutableStringPiece` that are misspelled
      as `MutablesStringPiece` which doesn't exist.
      
      Reviewed By: Orvid
      
      Differential Revision: D26011095
      
      fbshipit-source-id: 2503e13b84e04654c242bcd1f7bb17f9c1b757dc
      3c764a2d