1. 01 Dec, 2020 3 commits
    • Xiao Shi's avatar
      heterogeneous lookup for ConcurrentHashMap · 89ad85e1
      Xiao Shi authored
      Summary:
      This diff adds heterogeneous lookup handling for `ConcurrentHashMap` (`find()`
      and `at()`), which allows lookups on keys that the hasher and key equal
      functors accept but are not exactly the same as the `KeyType` of the map.
      Common usage of the feature include looking up with a `std::string_view` key in
      a `map<string, X>`, thereby avoiding creating an unnecessary temporary string
      object.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D23279680
      
      fbshipit-source-id: 0537d23dd6b464cfea7a5b347ecbad9a5ec5bde4
      89ad85e1
    • Alexander Sklar's avatar
      Fix Folly to build with current MSVC (#1450) · 7dcf1524
      Alexander Sklar authored
      Summary:
      Fixes https://github.com/facebook/folly/issues/1449
      
      The current public VS releases (16.3 up to 16.7) have builtins support. In commit https://github.com/facebook/folly/commit/7f1bda25627503ca8af2536cc81298a7d4688375 the #ifdef that was gating the definition of builtins was removed, as 16.8 will remove it. However by removign this gate, it makes current VS versions unable to build folly which breaks projects like react-native-windows.
      
      Pull Request resolved: https://github.com/facebook/folly/pull/1450
      
      Reviewed By: yfeldblum
      
      Differential Revision: D23702071
      
      Pulled By: Orvid
      
      fbshipit-source-id: 6e05f63293e68783b6caae68cc5bd80ad3679766
      7dcf1524
    • Koray Polat's avatar
      Use half of available CPUs · 650dde9a
      Koray Polat authored
      Summary: During intensive builds using all available cores tends to crash the machine. Cutting default number of jobs by half prevents that.
      
      Reviewed By: wez
      
      Differential Revision: D25027757
      
      fbshipit-source-id: 5ea6c77fe4363b9ee09e50de1a5cc33281872e1d
      650dde9a
  2. 23 Nov, 2020 1 commit
  3. 20 Nov, 2020 5 commits
  4. 19 Nov, 2020 12 commits
    • Lewis Baker's avatar
      Add async-stack support to co_awaitTry() · 1d8a0445
      Lewis Baker authored
      Summary:
      Updates the co_awaitTry() adapters to support the co_withAsyncStack()
      CPO.
      
      This required updating the WithAsyncStackAwaitable to also support
      co_awaitTry() but having its wrapper awaiter type also forward through
      the .await_resume_try() method if one exists.
      
      Modified the is_awaiter trait check to no longer check for a valid
      await_suspend() call as some awaiters returned from co_withAsyncStack()
      now require coroutine_handle<Promise> and no longer accept the
      coroutine_handle<void> parameter previously checked by the trait.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24499833
      
      fbshipit-source-id: fdcb99a2b85c4235bcd4b5dcc12cf79cb414f034
      1d8a0445
    • Lewis Baker's avatar
      Improve stack-tracing for tasks launched by the merge() internals · 0249f352
      Lewis Baker authored
      Summary:
      The stack-traces for child worker tasks launched to process each
      incoming AsyncGenerator now have the same common consumer-task
      as the parent frame, which is then a detached-task, rather than
      each of the child workers being their own detached tasks.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24480161
      
      fbshipit-source-id: c02887abf676bde9b297ee6d8f0fcabe959616e3
      0249f352
    • Lewis Baker's avatar
      Add support for async-stacks to blockingWait() · d35391f4
      Lewis Baker authored
      Summary:
      This should allow stack-traces for a chain of async work that was launched
      using blockingWait() to continue tracing the normal stack-frames of the caller
      of blockingWait().
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24479522
      
      fbshipit-source-id: e0604ed43edbd9df51daf679e0adf06d934a3ca6
      d35391f4
    • Lewis Baker's avatar
      Update AsyncScope so that the first frame's return-address is the .add() callsite · cf7fad82
      Lewis Baker authored
      Summary:
      Modify the AsyncScope::add() method so that it records the call-site of .add()
      as the first frame's return address instead of recording .add() itself as the
      first frame.
      
      This should hopefully give a bit more insight into where the detached task was
      launched.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24478808
      
      fbshipit-source-id: 0723891e77af5e52c7f8c13e218e4fb9120867e3
      cf7fad82
    • Lewis Baker's avatar
      Hook up collectAll() algorithms to propagate async stack chains · 2e9437e4
      Lewis Baker authored
      Summary:
      The collectAll() family of algorithms now hooks up async-stack frames
      of child operations to ensure that they trace back up through to the
      caller of collectAll() rather than stopping at the call to .start()
      inside collectAll().
      
      This also adds the helper `folly::coro::detail::co_current_async_stack_frame` which
      allows the coroutine to get access to its own AsyncStackFrame
      object - which is needed for algorithms like collectAll() which
      then need to pass this on to the BarrierTask when it is started.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24466130
      
      fbshipit-source-id: f7058388ebc8fa212684eb0edc0a83deef6fc2a2
      2e9437e4
    • Lewis Baker's avatar
      Add support for async stacks to co_reschedule_on_current_executor · 036fea85
      Lewis Baker authored
      Summary:
      It now saves and restores the async stack frame natively which
      avoids creation of a wrapper coroutine to do this.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24464577
      
      fbshipit-source-id: 965893a7f2e30c0759fb1970784500c96ed8a8ac
      036fea85
    • Lewis Baker's avatar
      Add support for saving/restoring async stack frames to default co_viaIfAsync() implementation · e8eb0719
      Lewis Baker authored
      Summary:
      This should avoid creating two temporary wrapper coroutines to apply
      both co_viaIfAsync() and co_withAsyncStack() to awaitables when awaited
      within a Task/AsyncGenerator coroutine.
      
      The default co_viaIfAsync() implementation now customises the
      co_withAsyncStack() CPO and saves and restores the async frame
      itself if the wrapped awaitable does not support async stacks,
      or otherwise passes the async frame through to the child awaitable
      if it does support async stacks.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24464513
      
      fbshipit-source-id: f9751030f3c1d2725a00cb803ee46ee12dddb2ed
      e8eb0719
    • Lewis Baker's avatar
      Add async stack-trace support to DetachedInlineTask · beb3e640
      Lewis Baker authored
      Summary:
      The DetachedInlineTask coroutine type now has its own AsyncStackFrame.
      
      While this is largely an internal implementation detail used only
      for the TaskWithExecutor's `.start()` family of methods, this now
      means that detached tasks launched using TaskWithExecutor's
      `.start()` methods will now record the return-address of the call to
      `.start()` in the stack-trace and will also have the async-stack begin
      with the `detached_task` frame at the top-level.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24440817
      
      fbshipit-source-id: 861f1ca46ec4a28b5b7e388bb8b243e4d597ee0d
      beb3e640
    • Lewis Baker's avatar
      Add async stack support to AsyncGenerator · 155f5afa
      Lewis Baker authored
      Summary:
      AsyncGenerator coroutines now have their own AsyncStackFrame and so can
      participate in async call-stack tracing.
      
      AsyncGenerator now applies co_withAsyncStack() to all awaitables
      awaited from within the generator coroutine.
      
      The NextAwaiter now implements the co_withAsyncStack() CPO to advertise
      that it will save/restore the awaiting coroutine's AsyncStackFrame if
      it provides one.
      
      This allows the AsyncGenerator to symmetric-transfer to awaited Task
      coroutines and vica-versa. This fixes some stack-overflow issues with
      some synchronously-completing use-cases of AsyncGenerator.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24437278
      
      fbshipit-source-id: f9612d54558b961bb023460d4a118a773022737b
      155f5afa
    • Lewis Baker's avatar
      Add support for async stack-traces to Barrier/BarrierTask · 892c0c31
      Lewis Baker authored
      Summary:
      `BarrierTask` and `DetachedBarrierTask` coroutines now have their own `AsyncStackFrame`.
      
      This means that when a `Task` executes `co_await barrier.arriveAndWait()` that we
      can use symmetric transfer to resume the `Task` since the `BarrierTask` will
      already have an active `AsyncStackRoot` that we can reuse.
      
      This fixes the test for stack-overflow of synchronously completing coroutines that
      use `collectAll()` algorithms (which internally use `BarrierTask`).
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24435340
      
      fbshipit-source-id: 87b622e8083f6c8f2b12f6b2f4b0bccd8bd5a595
      892c0c31
    • Lewis Baker's avatar
      Add support for async stack traces to folly::coro::Task coroutines · 70852ff3
      Lewis Baker authored
      Summary:
      Initial diff that adds support for tracing through a chain of folly::coro::Task coroutines.
      
      This adds some scaffolding that handles saving/restoring the active AsyncStackFrame
      when a Task awaits some awaitable type that does not know about AsyncStackFrame
      objects but also allows awaitables to opt-in to AsyncStackFrame awareness by
      customising the new `folly::coro::co_withAsyncStack()` CPO.
      
      Currently only `Task` and `TaskWithExecutor` awaiters have customised this CPO.
      
      Also updated the awaiters for `Task` and `TaskWithExecutor` to handle being awaited
      from coroutines that are not async-stack aware - in which case they just record a
      null parent-frame for the Task.
      
      The Task's `final_suspend()` then either deactivates or pops the frame depending
      on whether there was a parent frame recorded.
      
      BUG: This change currently breaks the symmetric-transfer stack-overflow avoidance
      when awaiting synchronously-completing coroutines from an AsyncGenerator or from
      a BarrierTask (eg. inside collectAll implementations).
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24428736
      
      fbshipit-source-id: 5722e511ad10d95198ae70a5afe567d83cb06285
      70852ff3
    • Chad Austin's avatar
      convert a path to valid glob syntax when prefetching · 2c41d995
      Chad Austin authored
      Summary:
      Paths are not necessarily legal glob syntax. In particular, backslash
      is used for escaping. This caused problems on Windows, where we tried
      to pass a backslash-delimited path into `eden prefetch --no-prefetch`.
      
      Reviewed By: xavierd
      
      Differential Revision: D25072784
      
      fbshipit-source-id: 9ce8e5ccc8f28581512c39d04922889da0bc1bf6
      2c41d995
  5. 17 Nov, 2020 4 commits
  6. 16 Nov, 2020 1 commit
    • Pranjal Raihan's avatar
      Fix folly/io/async/test:async_test - AsyncSocketTest.ConnectionExpiry · ff2ecd36
      Pranjal Raihan authored
      Summary: The worker thread just needed to be killed before the `AcceptCallback`. `AsyncServerSocket` calls a method on this callback class in its dtor, so it must either be alive or removed using `removeAcceptCallback`.
      
      Differential Revision: D24972293
      
      fbshipit-source-id: 2c7b6d64c2b9b8c00d03107002fa31e565df2a01
      ff2ecd36
  7. 13 Nov, 2020 8 commits
    • Pranjal Raihan's avatar
      Add API to AsyncServerSocket that allows potentially stale connections to be dropped · 04eafeb6
      Pranjal Raihan authored
      Summary:
      Added a timestamp to `AsyncServerSocket::QueueMessage` so that `NotificationQueue` can ignore new connection messages which are deemed *expired*. Expired messages represent sockets which have *probably* timed out already.
      
      The TTL is configured per `AsyncServerSocket` instance and is applied to all future messages that are queued by it. By default, messages do not expire. This can be configured with `AsyncServerSocket::setQueueTimeout`.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24667870
      
      fbshipit-source-id: 0f9d6c235627393d964e280a0d5956676010c7aa
      04eafeb6
    • Pranjal Raihan's avatar
      Allow AtomicNotificationQueue consumers to discard dequeued tasks · d366ea8f
      Pranjal Raihan authored
      Summary:
      If `Consumer::operator()` returns `AtomicNotificationQueueTaskStatus` then this value indicates if the consumer actually consumed the task, or if it should be discarded.
      
      Enumerating all cases:
      
      `operator()` return type is `void`:
      *All tasks are considered consumed and counted towards `maxReadAtOnce`.*
      
      `operator()` returns `AtomicNotificationQueueTaskStatus::CONSUMED`:
      *Same as above.*
      
      `operator()` returns `AtomicNotificationQueueTaskStatus::DISCARD`:
      *Task does not count towards `maxReadAtOnce` and is silently discarded. The consumer is responsible for any cleanup other than calling the destructor.*
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24698221
      
      fbshipit-source-id: 0da2364d18b67468addf737b67cae573c89b7e9c
      d366ea8f
    • Abdulbaki Aydin's avatar
      Add an API to set signature algorithms in string form · 9f87fdb8
      Abdulbaki Aydin authored
      Summary:
      Add an API to enable setting signature algorithms in string form.
      An example string form: `"ECDSA+SHA256:RSA+SHA256"`.
      OpenSSL supports TLS1.3 Signature Scheme format,
      e.g.; `"ecdsa_secp256r1_sha256:rsa_pss_rsae_sha256"`.
      
      Reviewed By: mingtaoy
      
      Differential Revision: D24745481
      
      fbshipit-source-id: 018be0a242d922f36a81d051791679a20fe08893
      9f87fdb8
    • jmccl's avatar
      Fix build when liburing is installed (#1487) · cc1be4ce
      jmccl authored
      Summary:
      Fixes https://github.com/facebook/folly/issues/1486
      
      Pull Request resolved: https://github.com/facebook/folly/pull/1487
      
      Reviewed By: Orvid
      
      Differential Revision: D24851586
      
      Pulled By: yfeldblum
      
      fbshipit-source-id: ae683ba73336c562a8912ed35ef2b3318ab53519
      cc1be4ce
    • Pratik Shah's avatar
      Subprocess now does not require child processes to be reaped before destruction · 660d9244
      Pratik Shah authored
      Summary: To support cases where the child process may outlive the parent process.  Detaching the process is an option as well, but in that case the pid for the child process is not captured.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D24409176
      
      fbshipit-source-id: f74f64e3ae21baa40468c6f39ac6b4d0152db03a
      660d9244
    • Harshit Saokar's avatar
      Instrument time spent by fiber in running state and log it for SR eventbase threads · da355ecd
      Harshit Saokar authored
      Summary:
      We often run into cases where system CPU util is low but SR eventbase threads are backing up. During S213134 debugging, it took us a while to figure out that requests with http transport were expensive and causing issues on intern-rpc clients and later realized it was a known issue for http transport to be CPU intensive. We got lucky that http transport requests used different stack we could see in strobelight, but this will likely not be the case for other similar issues.
      
      For ease of debugging these issues, one idea is to log SR Eventbase cost of each request to SR scuba. **The closest proxy for this is the wall clock time spent by SR's request-dispatch task in fiber, excluding all the preemptions.** This metric does have couple of drawbacks:
      
      - It wont capture all the work a request does via runInMainContext. This should be Ok since SR tries not to do lot of work in mainContext of fiber manager.
      - If system context switching the threads a lot and if eventbase thread is preempted by OS, we will count that time as time spent in task. I am hoping that new metric will still be useful even with this caveat.
      
      Design: Allow fiberManager.AddTask() to optionally take in a bool `logRunningTime` to decide whether it can log the running time of the task. If `logRunningTime` is set, fiber will keep track of elapsed time since most recent run in  `currStartTime_` and track the running time of all previous runs together in  `prevDuration_`.
      
      I looked into using ExecutionObserver for this purpose. I did not go that route because
      1. ExecutionObserver's callbacks return fiber IDs, we want a callback to return function/task ID that we can tie back to Sr request.  See here https://fburl.com/diffusion/qh9ioeq3
      2. Caller (SR) will need to maintain the separate cache of all outstanding tasks with their running_times and manage their lifetime, which feels more complex than current approach.
      
      I have a way to turn off this change using new killswitch I added in SR.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D24823396
      
      fbshipit-source-id: 62f79b8c340cc48a22684c9cb3cdddadc260a0ab
      da355ecd
    • Pranjal Raihan's avatar
      Migrate AsyncServerSocket to use AtomicNotificationQueue · ceee59a7
      Pranjal Raihan authored
      Reviewed By: andriigrynenko
      
      Differential Revision: D24698220
      
      fbshipit-source-id: aab8c96440d60bb653f0c537af56081819129306
      ceee59a7
    • Pranjal Raihan's avatar
      Add ability to bound AtomicNotificationQueue size · dcddc5c0
      Pranjal Raihan authored
      Summary:
      Allowing a bounded `AtomicNotificationQueue` is helpful to migrate `AsyncSocketServer` over from `NotificationQueue`.
      
      `AtomicNotificationQueue::tryPutMessage` is the same as `putMessage` but also accepts a `maxSize`. If the queue size is >= `maxSize`, the message will not be queued. The user of the queue is responsible maintaining state regarding the max size of the queue.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D24698219
      
      fbshipit-source-id: 1e3d1dc0073c65cd1803629f6a4a9b015e111b08
      dcddc5c0
  8. 12 Nov, 2020 3 commits
    • Lee Howes's avatar
      Add retryingUnsafe · e5bb59b3
      Lee Howes authored
      Summary: Add retryingUnsafe as a duplicate of retrying that forces the return type to be a Future rather than a SemiFuture.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D24707194
      
      fbshipit-source-id: cb2a107f51d4d4dd47a41ce6f8a009c4d82e3138
      e5bb59b3
    • Michel Salim's avatar
      add shared library support to add_fbthrift_cpp_library · cfa6e9c6
      Michel Salim authored
      Summary:
      `add_fbthrift_cpp_library` should honor `BUILD_SHARED_LIBS`, and call
      `add_library` with the right setting (`SHARED` if enabled, `STATIC` otherwise)
      
      Reviewed By: yns88
      
      Differential Revision: D24911124
      
      fbshipit-source-id: 79df7640a758a592a3df3e9e79bb129dd57f2d47
      cfa6e9c6
    • Luca Niccolini's avatar
      add zlib as an explicit dependency for getdeps build · d4340c0a
      Luca Niccolini authored
      Summary:
      needed for QUIC/H3 interop
      
      also make it possible to run getdeps build with extra arguments (--no-tests for
      example)
      
      Reviewed By: mjoras
      
      Differential Revision: D24925777
      
      fbshipit-source-id: fbdc1aa56e398d295ef8dac0ad0bab03bd7bd803
      d4340c0a
  9. 11 Nov, 2020 1 commit
  10. 10 Nov, 2020 2 commits
    • Davide Cavalca's avatar
      proxygen: fix shared libs build · bfe27697
      Davide Cavalca authored
      Summary:
      Right now proxygen hardcodes a static build when using cmake  and
      ignores BUILD_SHARED_LIBS. Fix that, and enable PIE on the shared libs so they
      can be linked properly
      
      Closes: https://github.com/facebook/proxygen/issues/335
      
      Reviewed By: mjoras, lnicco
      
      Differential Revision: D24787944
      
      fbshipit-source-id: 7a654af7cb43227ca913a1bed67f2432703a343d
      bfe27697
    • Michel Salim's avatar
      also install executor.h · 6cd0c17d
      Michel Salim authored
      Summary: This header is needed by python/futures.h
      
      Reviewed By: yfeldblum
      
      Differential Revision: D24861734
      
      fbshipit-source-id: 913ad13e6ca155250238020538af762c2360f137
      6cd0c17d