1. 11 Jan, 2022 1 commit
    • Charles McCarthy's avatar
      Add tests for folly::dynamic ordering comparison operators on scalar/string... · 2e364fbb
      Charles McCarthy authored
      Add tests for folly::dynamic ordering comparison operators on scalar/string types and highlight some bugs
      
      Summary:
      # Summary
      Currently nullptr vs nullptr and int vs double ordering comparison operators (e.g. `operator<`) are inconsistent with their equality operators.
      
      This diff just adds tests to increase coverage and highlight the broken areas. The subsequent diffs will address the issues individually.
      
      The behavior on nulls at least has been present since the very beginning of when a CompareOp was added for it in 2017.
      
      More details are below.
      
      # Bug 1: nullptr vs nullptr
      If we have
      ```
      folly::dynamic a(nullptr);
      folly::dynamic b(nullptr);
      
      a < b // returns true
      a == b // returns true
      ```
      This violates that the two should be equal if `!(a < b) && !(b < a)` given that this expression will return `false`, whereas `a == b` returns `true`.
      
      This property is required by e.g. std::sets.
      If one puts a `folly::dynamic(nullptr)` in an std::set with the current behavior, when iterating it will be found, and will contribute to its size, but `std::set::find()` on it will return false. Hash-based sets are unaffected - given that the hash and `operator==` implementations are correct.
      
      The fix is straightforward.
      `struct dynamic::CompareOp<std::nullptr_t>` in dynamic-inl.h should return false.
      
      # Bug  2: int vs double (hash + ordering operators)
      The `folly::dynamic==` operator does numeric comparison for int64 vs double - whereas the `folly::dynamic<` operator does not - instead falling back on comparing against the Type enum value. Similar to the above this violates the principle of `operator<` being usable for checking equality - given that it is really just checking type equality - but has already been established in the code path that they're different types.
      
      Hashing is also inconsistent given that a hash of dynamic(2.0) will not equal dynamic(2) even though they are equal, given that each uses the hash of the underlying type. When ordering is fixed, hashing should be too so that hash based and ordering based containers have the same behavior.
      
      The fixes are straightforward.
      
      operator< in dynamic.cpp should pull the same logic from operator== with regards int vs double.
      See https://www.internalfb.com/code/fbsource/[4e70ca24b9ec]/fbcode/folly/dynamic.cpp?lines=97-126
      
      hashing can be resolved by having doubles use integer hashing in dynamic.cpp - https://www.internalfb.com/code/fbsource/[4e70ca24b9ec]/fbcode/folly/dynamic.cpp?lines=307
      
       ---
      
      Reviewed By: Gownta
      
      Differential Revision: D33465564
      
      fbshipit-source-id: 43b926837e20f4a9afe15ee0032d469864e360c0
      2e364fbb
  2. 08 Jan, 2022 1 commit
  3. 07 Jan, 2022 5 commits
    • Mark Santaniello's avatar
      Avoid copying ByteRange in OpenSSLHash hash_update() · 21329c4c
      Mark Santaniello authored
      Summary: This opportunity was found with the combination of Infer's PULSE_UNNECESSARY_COPY and Strobelight cycle counts.
      
      Reviewed By: Gownta
      
      Differential Revision: D33304127
      
      fbshipit-source-id: 7d3b54d26bc3e171cc7a0a831933fdbb9638f6a8
      21329c4c
    • Andrii Grynenko's avatar
      Replace mutexes in observers with SharexMutex to reduce memory usage · 73242d19
      Andrii Grynenko authored
      Summary: sizeof(SharedMutex) = 4, sizeof(std::mutex) = 40. This diff makes sure that we use smaller mutexes both in Observer Core and in Observables given that we may end up with many instances of these objects, yet there mutexes are accessed only on the update path.
      
      Reviewed By: praihan
      
      Differential Revision: D32271615
      
      fbshipit-source-id: ba7d04632a677a9eb4ae398d681dadac3b06f033
      73242d19
    • Mingda Zhang's avatar
      CO_SKIP_IF · de5021e9
      Mingda Zhang authored
      Reviewed By: yfeldblum
      
      Differential Revision: D33118848
      
      fbshipit-source-id: c69b1f8f415b130d5d7a6487e7027d5c1e49865f
      de5021e9
    • Maged Michael's avatar
      Baton and Futex: Add comments about async-signal-safety · e0204da8
      Maged Michael authored
      Summary: Add comments stating the expectations for Baton post and Futex wake native implementation to be async-signal-safe.
      
      Reviewed By: ot, luciang
      
      Differential Revision: D33461663
      
      fbshipit-source-id: 2855427cc94f577b566b19203a4523f54a41d0aa
      e0204da8
    • Alex Hornby's avatar
      extract get_dependencies method · e26fa87d
      Alex Hornby authored
      Summary:
      A number of places were extracting dependencies from manifests, but only one was adding in the implicit dependencies for build tools.
      
      Extract the logic to one place and use so that a change in a tool like cmake will now correctly affect all tools using cmake, as it will be taken into account as a dependency hash when the manifest's hash is computed.
      
      Tests for this change revealed that install_dirs needed to be populated in reverse order from the manifest topo-sort, so have also addressed that
      
      Reviewed By: wittgenst
      
      Differential Revision: D32730717
      
      fbshipit-source-id: 1b2a25e460de6085d274c99acfd391b3bd259264
      e26fa87d
  4. 06 Jan, 2022 3 commits
    • Zachary Collins's avatar
      change IOExecutorThreadPool such that observer logic can be overriden · c911d883
      Zachary Collins authored
      Summary: We want to have a custom `IOExecutorThreadPool` that performs different logic for handling observers being adding or not. This change makes it so the functions and structs necessary to do so can be properly overriden.
      
      Reviewed By: Gownta
      
      Differential Revision: D32101852
      
      fbshipit-source-id: 2f1c6bb4c00d41630891aa72b8e519708cb1a40f
      c911d883
    • Alex Hornby's avatar
      cmake needs openssl · e7eb2bb6
      Alex Hornby authored
      Summary: When we build cmake on linux the openssl headers are needed
      
      Reviewed By: fanzeyi
      
      Differential Revision: D33432903
      
      fbshipit-source-id: 2f869768f3a81b74e53391f5491781759c66b478
      e7eb2bb6
    • Kyle Nekritz's avatar
      Pass handshake exception to pending write and read callbacks. · c894977f
      Kyle Nekritz authored
      Summary:
      If we already have pending application reads or writes when we have a
      handshake error (eg when using a null connect callback), we should pass the
      handshake error to finishFail to give a more helpful error message.
      
      Reviewed By: mingtaoy
      
      Differential Revision: D33415008
      
      fbshipit-source-id: fc14ef774d09c0af975e177b24185f78c4e4139e
      c894977f
  5. 05 Jan, 2022 2 commits
  6. 04 Jan, 2022 4 commits
    • Jim Meyering's avatar
      folly/executors/ManualExecutor.h: mark overriding functions with "override" · bf2a54fe
      Jim Meyering authored
      Summary:
      This avoids the following errors:
      
        folly/executors/ManualExecutor.h:42:3: error: '~ManualExecutor' overrides a destructor but is not marked 'override' [-Werror,-Wsuggest-destructor-override]
      
      Reviewed By: r-barnes
      
      Differential Revision: D33397861
      
      fbshipit-source-id: f9bee89aeaa55d365155dc3fdc0b70b283980a2a
      bf2a54fe
    • Jim Meyering's avatar
      folly/io/async/ssl/OpenSSLTransportCertificate.h: mark overriding functions with "override" · 15549d1c
      Jim Meyering authored
      Summary:
      This avoids the following errors:
      
        folly/io/async/ssl/OpenSSLTransportCertificate.h:31:11: error: '~OpenSSLTransportCertificate' overrides a destructor but is not marked 'override' [-Werror,-Wsuggest-destructor-override]
      
      Reviewed By: r-barnes
      
      Differential Revision: D33397487
      
      fbshipit-source-id: 4237f5ad36166189b0bbd28dfcf268e0ef4b40f1
      15549d1c
    • Jim Meyering's avatar
      folly/experimental/coro/BlockingWait.h: mark overriding functions with "override" · cc060e3e
      Jim Meyering authored
      Summary:
      This avoids the following errors:
      
        folly/experimental/coro/BlockingWait.h:301:3: error: '~BlockingWaitExecutor' overrides a destructor but is not marked 'override' [-Werror,-Wsuggest-destructor-override]
      
      Reviewed By: r-barnes
      
      Differential Revision: D33397860
      
      fbshipit-source-id: 050c3c564e70dfc5dc1232e58b40f2c4cbcb660a
      cc060e3e
    • Jim Meyering's avatar
      folly/io/async/ScopedEventBaseThread.h: mark overriding functions with "override" · 395c5d94
      Jim Meyering authored
      Summary:
      This avoids the following errors:
      
        folly/io/async/ScopedEventBaseThread.h:49:3: error: '~ScopedEventBaseThread' overrides a destructor but is not marked 'override' [-Werror,-Wsuggest-destructor-override]
      
      Reviewed By: ryanthomasjohnson
      
      Differential Revision: D33397489
      
      fbshipit-source-id: 29210c0de0b850bdda5d2c27a5be8971fc17b40e
      395c5d94
  7. 31 Dec, 2021 2 commits
    • Alex Hornby's avatar
      update README.md to reference getdeps.py for build and tests · f2385299
      Alex Hornby authored
      Summary:
      Update the README.md to reflect that getdeps.py build and test is the method tested in CI and thus preferred.
      
      Where getdeps.py manifests encode dependencies I've removed the manually specified dependences from the README for consistency (e.g. googletest version in README was outdated, manifest is correct and is tested in CI)
      
      Also referenced main vs master and improved markdown header formatting a little
      
      Reviewed By: meyering
      
      Differential Revision: D33295900
      
      fbshipit-source-id: caec3b3795be0b37bf1efdf12f4dbf23a37a023c
      f2385299
    • Yicheng Wang's avatar
      Remove setMaxReadAtOnce() API in MeteredExecutor · 56aa5c9a
      Yicheng Wang authored
      Summary:
      There is no current use case of this API throughout the codebase
      and it's causing some confusions of how this class is used
      so removing it for now. It's easy to add it back if we need it
      in the future
      
      Reviewed By: yfeldblum
      
      Differential Revision: D33112001
      
      fbshipit-source-id: c711f5ab0ad2324e478f6eb18155d367d2cce1f2
      56aa5c9a
  8. 30 Dec, 2021 2 commits
  9. 29 Dec, 2021 1 commit
    • Maxim Georgiev's avatar
      Adding cert extension parsing methods to folly::OpenSSLCertUtils · 12c748a8
      Maxim Georgiev authored
      Summary:
      Adding methods to `folly::OpenSSLCertUtils` that allow to read extensions from X509 cert structures.
      - `folly::OpenSSLCertUtils::getExtension()` allows to query extensions by name
      - `folly::OpenSSLCertUtils::getAllExtensions()` returns everything including custom extensions.
      No extension name translation mechanism is provided for custom extensions.
      
      Reviewed By: mingtaoy
      
      Differential Revision: D31852457
      
      fbshipit-source-id: 22d34d2cf304ba135419c902d9c5c303f78cf781
      12c748a8
  10. 27 Dec, 2021 1 commit
  11. 23 Dec, 2021 1 commit
    • Nicholas Ormrod's avatar
      Fix timing-out ConcurrentAdd test · 5b5e5909
      Nicholas Ormrod authored
      Summary:
      This test regularly times out under load.
      
      I made a benchmark diff (prior to this in the stack, but not to be landed) which confirms (a) that the slowness of this test is due to spawning excessively-many threads, and (b) that the performance of ConcurrentSkipList doesn't unduly degrade with excessively-many threads. Conclusion: it is safe to simplify these tests.
      
      To simplify the test, I ratcheted down the number of threads, but also bumped the number of iterations. Much more work is being done (so this test should retain its race-condition-hunting properties), but the time spent is much less, even when the system is under load.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D33269899
      
      fbshipit-source-id: 9f7a0ec5f3c1077c91e5d09c89f3752d552a6320
      5b5e5909
  12. 22 Dec, 2021 1 commit
    • Nicholas Ormrod's avatar
      Fix broken singleton test · ec7f9f6f
      Nicholas Ormrod authored
      Summary: This test regularly fails on account of timing. Remove the timing upper bound.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D33271470
      
      fbshipit-source-id: 1369ef45bd49e500c35816bfdb21869f346b6f70
      ec7f9f6f
  13. 21 Dec, 2021 4 commits
    • Nicholas Ormrod's avatar
      Mark sorted_vector swap as conditionally noexcept · 6fcea2fc
      Nicholas Ormrod authored
      Summary: swap should be noexcept when possible
      
      Reviewed By: yfeldblum
      
      Differential Revision: D33198827
      
      fbshipit-source-id: f8d52a21087b0346332359b66e1dc00f982ddbbb
      6fcea2fc
    • Alex Hornby's avatar
      add BadgeTest.cpp to cmake build · 16a0583c
      Alex Hornby authored
      Summary:
      Externally this was failing on fedora 35's GCC 11.2.1
      
      Add test so we know what platforms it's good on
      
      Reviewed By: yfeldblum
      
      Differential Revision: D33192636
      
      fbshipit-source-id: 028255b6dbb34762b4e788122d948f26a449a57c
      16a0583c
    • Alex Hornby's avatar
      enable MemcpyTest in cmake · 65d40bbb
      Alex Hornby authored
      Summary: Try enabling this, need to handle linux ASM build case in cmake config
      
      Reviewed By: Gownta
      
      Differential Revision: D33191710
      
      fbshipit-source-id: 6d49a08106c8a0044380f592fd2862080cbea454
      65d40bbb
    • Alex Hornby's avatar
      folly: fixes to renable cmake tests (#1691) · 819becba
      Alex Hornby authored
      Summary:
      Having the tests is useful to be able to test the OSS builds. A few were failing for me locally so I've tagged them BROKEN in CMakeLists.txt which folly's cmake config filters out
      
      Pull Request resolved: https://github.com/facebook/folly/pull/1691
      
      Test Plan:
      tested with:
      
      ```
      ./build/fbcode_builder/getdeps.py --allow-system-packages build
      ./build/fbcode_builder/getdeps.py --allow-system-packages test
      ...
      100% tests passed, 0 tests failed out of 2736
      
      Total Test time (real) =  14.95 sec
      ````
      
      Reviewed By: yfeldblum, Gownta
      
      Differential Revision: D33169409
      
      Pulled By: ahornby
      
      fbshipit-source-id: 9c781a84b8873c295af96368dd8315254a78f096
      819becba
  14. 20 Dec, 2021 2 commits
  15. 19 Dec, 2021 1 commit
    • Yedidya Feldblum's avatar
      rename Function private helper ResultOf · 72229769
      Yedidya Feldblum authored
      Summary: `ResultOf` is no longer the most-suitable name. This helper has long been a SFINAE helper and its actual output did not matter, but now that it is always `void` the name is even less clear. Rename it.
      
      Reviewed By: luciang
      
      Differential Revision: D33155200
      
      fbshipit-source-id: a12842c6be38bd82bfffa85193ec265bc4d803ad
      72229769
  16. 18 Dec, 2021 3 commits
    • Yedidya Feldblum's avatar
      wrappers around operator new and operator delete · c465314c
      Yedidya Feldblum authored
      Summary:
      C++14 introduces sized overloads of `operator delete` and C++17 introduces aligned overloads of `operator new` and `operator delete`. Offer wrappers which call the best available versions.
      
      Using these overloads, asking for heap storage suitable for a non-over-aligned class can be done via these wrappers without requiring call-sites to test feature macros.
      
      Most potential use-cases should use `std::allocator<T>::allocate` and `std::allocator<T>::deallocate`. Reasons why use-cases may turn to these wrappers instead are:
      * These wrappers are not temnplates and using them does not instantiate class or function templates.
      * `std::allocator<T>::deallocate` may, but is not specified to, use sized deallocation when sized deallocation is available.
      
      Reviewed By: luciang
      
      Differential Revision: D33180956
      
      fbshipit-source-id: 944fe6a01ca4cfff72f3b92639158e4a5ac029b8
      c465314c
    • Yedidya Feldblum's avatar
      hide lambda return types from Function ctor symbols · 9a154f6e
      Yedidya Feldblum authored
      Summary: It is possible for return type names to be long. They are present in the ctor symbol names for SFINAE, but we can get around that by inserting a cast-to-void.
      
      Reviewed By: luciang
      
      Differential Revision: D32976770
      
      fbshipit-source-id: d5a972543f4d97f94abea9631a4bed35fa67f8f3
      9a154f6e
    • Yedidya Feldblum's avatar
      a std::decay_t lookalike · a61cf107
      Yedidya Feldblum authored
      Summary: Appears to be measurably faster to compile under clang as compared to `std::decay_t`.
      
      Reviewed By: Alfus, luciang
      
      Differential Revision: D32946561
      
      fbshipit-source-id: 028c79633095ae9d6dc75d967282b53b32986736
      a61cf107
  17. 17 Dec, 2021 6 commits
    • Andrew Smith's avatar
      Add support for rate limiting to transform and resumableTransform · eadf2192
      Andrew Smith authored
      Summary:
      The channels framework provides most of its memory savings by allowing channels to be used without long-lived coroutine frames consuming them. However, coroutines are still used for short-lived transformation functions. While these coroutines should not increase steady state memory (as they are short-lived), it is possible that a large number of transformations occur at exactly the same time. This can cause a large memory spike (due to the size of the coroutine frames), and lead to heap fragmentation.
      
      This diff solves this problem by adding support for rate limiting. An optional RateLimiter object can be provided to any transform or resumableTransform. A RateLimiter has a maximum concurrency specified on construction. For any transforms using the same rate limiter, the channels framework will ensure that the concurrency constraint is not violated. This limits the number of simultaneous coroutine frames (and therefore allows one to eliminate these spikes).
      
      Reviewed By: aary
      
      Differential Revision: D33037310
      
      fbshipit-source-id: f7c5d30e08d155db7825d01f1b4c4b7354b14def
      eadf2192
    • Andrew Smith's avatar
      Avoid having to catpure initialization arguments when using resumableTransform · 05e0144b
      Andrew Smith authored
      Summary:
      Currently, resumableTransform takes a paremeter-less initializeTransform method for initialization, and re-initializes the transform when the transformValue function throws an OnClosedException. This requires users to capture any initialization parameters in the initializeTransform lambda, so they can re-initialize with the same parameters when needed.
      
      However, it is often the case that the initialization arguments are stored elsewhere in the app, and passed along as part of each message coming through input receiver to the transform. (This is the case in all uses of resumableTransform so far.) If we could take advantage of already having the initialization arguments available, we would not need to waste memory capturing and storing them in the initializeTransform lambda.
      
      This diff does exactly that. Instead of triggering a reinitialization by throwing an OnCloseException, reinitialization is triggered by throwing a ReinitializeTransformException<InitializationArg>. This initialization argument is then passed to initializeTransform. By doing this, we no longer need to store initialization arguments in resumable transforms.
      
      (As a side benefit, this also makes the code slightly clearer to the reader. Rather than the reader having to know that throwing an OnClosedException triggers a reinitialization, throwing a ReinitializeTransform exception makes it clear.)
      
      Reviewed By: aary
      
      Differential Revision: D33037191
      
      fbshipit-source-id: 891394d79d3fc43a8e253031472256bb852c7716
      05e0144b
    • Andrew Smith's avatar
      Add closeSubscribers to FanoutChannel · ef8b5db6
      Andrew Smith authored
      Summary: This diff adds a new function to FanoutChannel that closes all existing subscribers, without closing the FanoutChannel itself.
      
      Reviewed By: aary
      
      Differential Revision: D33035825
      
      fbshipit-source-id: 6aa867344c280ad7a91bd8248e745aa2f27e7cc8
      ef8b5db6
    • Andrew Smith's avatar
      Add Transform overload that accepts a Transformer object · 9385c05a
      Andrew Smith authored
      Summary:
      This diff reduces the amount of memory that transforms use. Currently, transform explicitly takes an executor and a transform lambda (and resumableTransform takes an additional initialization lambda). Often, these lambdas end up taking pointers to some shared state object. In the case of resumableTransform, both lambdas need to capture a pointer to the same shared state object. Furthermore, the executor passed to transform is often present on the shared state object, making it redundant. This problem will get even worse when we add rate limiters (which also can be placed on the shared state object).
      
      To reduce memory waste, this diff adds a new overload for transform and resumableTransform that takes in a Transformer object. This transformer object must implement getExecutor, transformValue, and (for resumableTransform) initializeValue. This allows a transformer object to contain just one 8-byte pointer to a shared state object, rather than duplicating pointers for different lambda captures and executors (and eventually rate limiters). It also slightly simplifies code in certain cases, as the shared state can be accessed as a field from any function in the class (rather than captured and passed down as a parameter to all helper functions).
      
      We will still leave the existing overloads for cases where it is simpler for the user to not create a new object (when the user doesn't care as much about the memory savings).
      
      Reviewed By: aary
      
      Differential Revision: D33033365
      
      fbshipit-source-id: e57c16e76ad3795a7f93b39565f553e76623beb6
      9385c05a
    • Andrew Smith's avatar
      Change Transform template parameter names · b56cffe6
      Andrew Smith authored
      Summary: This diff changes Transform's parameter names to conform to folly's style guidelines.
      
      Reviewed By: Gownta
      
      Differential Revision: D33033366
      
      fbshipit-source-id: 3713ec7995df3dd913f4da11d21fbd9377c1f00f
      b56cffe6
    • Andrew Smith's avatar
      Change MergeChannel public interface · aa378653
      Andrew Smith authored
      Summary:
      The current MergeChannel public interface has a few problems that have become more apparent with usage.
      
      1. The output receiver does not contain the key associated with the input receiver that geneated the value. This often means that the user must transform input receivers before adding them to the merge channel to include the key, which wastes memory.
      2. There is no way for a consumer of the output receiver to know if an input receiver was closed without an exception.
      3. If an input receiver was closed with an exception, the entire merge channel is closed.
      
      This diff changes the public API to fix these problems. Instead of returning an output receiver that just has values, the output receiver contains MergeChannelEvent objects. Each such event object has the key of the corresponding input receiver associated with the event, along with the event contents. There are four types of events:
      
      1. A value was received through an existing input receiver.
      2. A new input receiver was added.
      3. An existing input receiver was removed.
      4. An existing input receiver was closed (with or without an exception).
      
      Note that we could theoretically avoid genering events for 2 and 3 (as they correspond with calls that the user makes to add and remove), but there are cases where it is useful to have 2 and 3. This way, an event is always received when an existing input receiver is removed, regardless of whether it was removed explicitly (through a call to removeReceiver) or implicitly (if the input receiver was closed).
      
      Reviewed By: aary
      
      Differential Revision: D33033374
      
      fbshipit-source-id: 34f74cba3f765c76a80d448647d733ad38a2195c
      aa378653