1. 03 Dec, 2019 6 commits
    • Matt Ma's avatar
      folly/experimental/symbolizer: symbolize inlined functions · 380c720b
      Matt Ma authored
      Summary:
      The previous approach (D16760775) reserved space for inlined function info in each `SymbolizedFrame`.
      
      ```
      struct SymbolizedFrame {
        Dwarf::LocationInfo location;
        Dwarf::LocationInfo inlineLocations[Dwarf::kMaxLocationInfoPerFrame];
        ...
      ```
      
      That increased the size of `SymbolizedFrame` and `FrameArray` and lead to stack overflow in some already deep stacks.
      ```
      template <size_t N>
      struct FrameArray {
        FrameArray() {}
      
        size_t frameCount = 0;
        uintptr_t addresses[N];
        SymbolizedFrame frames[N];
      };
      ```
      
      To avoid allocate more space on stack, changed to use extra frames to store inline calls:
      - Usually the callers allocate `FrameArray<100>` frames, but the stack trace is usually smaller than 100
      - Use the unused slots to fill in inlined function info:
       -- each function gets at most `kMaxLocationInfoPerFrame` (currently 3) inlined entries
       -- when the available buffer fills up no more inlined functions are filled in.
      
      To find the inline calling stack, we need first need to find the subprogram Debug Info Entry (with tag DW_TAG_subprogram) with the given address, then recursively find all the inline subroutines (with tag DW_TAG_inlined_subroutine) in the call stack. Sadly debug info has no index we can use for jump, and a linear search over debug info entries (and their attributes) is needed during the process, which would cause performance regression.
      
      ```
      buck run mode/opt folly/experimental/symbolizer/test:dwarf_benchmark -- --benchmark
      
      ============================================================================
      folly/experimental/symbolizer/test/DwarfBenchmark.cpprelative  time/iter  iters/s
      ============================================================================
      DwarfFindAddressFast                                         4.03us  248.36K
      DwarfFindAddressFull                                         4.03us  248.18K
      DwarfFindAddressFullWithInline                             293.23us    3.41K
      ============================================================================
      ```
      
      Reviewed By: luciang
      
      Differential Revision: D17586385
      
      fbshipit-source-id: 1b84b3f3a576573ce24092b433a501a3bdf76be0
      380c720b
    • Woo Xie's avatar
      expose the underlying AsyncUDPSocket in AsyncUDPServerSocket · e4d37c6c
      Woo Xie authored
      Summary: users may need AsyncUDPSocket to send back datagram to clients.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D18764100
      
      fbshipit-source-id: 7ab7cce5711f47db92f43a68b94568729d5746d0
      e4d37c6c
    • Dan Melnic's avatar
      Workaround for something that looks like a compiler bug · 506ab28c
      Dan Melnic authored
      Summary: Workaround for something that looks like a compiler bug
      
      Reviewed By: kevin-vigor
      
      Differential Revision: D18700404
      
      fbshipit-source-id: 6d9c9a40e9e468ee910c5456bcbfdf02c1b1c59a
      506ab28c
    • Andrew Gallagher's avatar
      folly/container/detail/F14Table.h: add missing `<emmintrin.h>` header · 8a6095a5
      Andrew Gallagher authored
      Summary:
      This is the header which provides `_mm_set1_epi8`
      (https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_set1_epi8).
      
      Although this header is transitively found via other headers (which is why this
      isn't a build failure), without this explicit include IWYU's heursitics around
      template functions wrongly assume that this header should be provided by
      downstream includers of this header.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D18753275
      
      fbshipit-source-id: 7d6ee934542867f2d1e84d40397f48ea8146facc
      8a6095a5
    • Nitin Garg's avatar
      Ensure recyle is done only once in IndexedMemPool.h · 85b4ece6
      Nitin Garg authored
      Summary:
      localPush invoked in recycleIndex can end up looping more than once if it has to retry the update of head. When that happens, the onRecycle call will be made again which will call destructor on the object again if eagerRecycle is true.
      
      Fixed it by making sure only the 1st pass of the loop will call onRecycle.
      
      (Note: this ignores all push blocking failures!)
      
      Reviewed By: nbronson
      
      Differential Revision: D18752349
      
      fbshipit-source-id: 27dcb5c7840f724bbd39f4dc0176e21095a22284
      85b4ece6
    • Taylor Hopper's avatar
      Add default constructor to RecordIOReader::Iterator · c5269bb3
      Taylor Hopper authored
      Summary: For this iterator to satisfy the `input_or_output_iterator` concept in range-v3, it must also satisfy the `semiregular` concept, which requires a default constructor.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D18689298
      
      fbshipit-source-id: a38cde0041ceba1aac79128d428df05ee0f00ab4
      c5269bb3
  2. 02 Dec, 2019 1 commit
    • Ahmed Soliman's avatar
      Install wheel and cython on fbcode_builder venv environments · 19eadc88
      Ahmed Soliman authored
      Summary:
      This is an update to the fbcode_builder codebase to allow setting up the python virtualenv with python dependencies installed. I've included wheel and cython (with a pinned version to 0.28.6 which is the only version that works with thriftpy3 at the moment, due to https://github.com/cython/cython/issues/2985) as standard packages since these are required by some of our top-level dependencies (folly and thrift)
      
      As far as I know, there are no other projects that use PYTHON_VENV at the moment except LogDevice so the impact should be minimal.
      
      Reviewed By: lucaspmelo
      
      Differential Revision: D18758383
      
      fbshipit-source-id: 264941311c5e3a19dc4ef2bb78c9a1baa34dfd8c
      19eadc88
  3. 01 Dec, 2019 1 commit
  4. 30 Nov, 2019 1 commit
    • Yedidya Feldblum's avatar
      More direct overload resolution control in folly::coro::concat · 22d18f0b
      Yedidya Feldblum authored
      Summary: [Folly] More direct overload resolution control in `folly::coro::concat`. Just name the type directly rather than using SFINAE, when the SFINAE would just name the type directly in a more roundabout way.
      
      Differential Revision: D18746633
      
      fbshipit-source-id: 6da909834b511426b1cf7bbda0fd7643b5761f14
      22d18f0b
  5. 29 Nov, 2019 9 commits
    • Ahmed Soliman's avatar
      Fixing fbthrift and folly cython bindings build · b1211e4c
      Ahmed Soliman authored
      Summary:
      This diff addresses a collection of issues with our thrift-py3 build story in open source.
      - Cython 0.29 bug https://github.com/cython/cython/issues/2985 prevents us from loading the generated binary python extensions. I've tested 0.28.6 and it works perfectly.
      - Adds ssl to the extensions (since we need SSLContext in get_client, we must have access to the module)
      - No need to special case folly_pic. There is really no easy way to get the python bindings to work without using shared library libfolly.so. So, In our build (logdevice) we will build folly as a shared library, this also reduces the complexity everywhere.
      - We also need iobuf as extension for thrift to work, added that to setup.py
      - Refactored some of the CMake code that had hardcoded file names to use globbing to future-proof this as much as possible.
      - Moved `python/lang/cast.pxd` to match the python module name `folly.cast`, so now it lives in `python/cast.pxd`. This simplifies the globbing as well.
      - Moved `setup.py` to live in the `/python` directory.
      - Added `*.py` in setup.py to pickup any raw python code (including `__init__.py`)
      
      Reviewed By: yfeldblum
      
      Differential Revision: D18685009
      
      fbshipit-source-id: 749b30942a3e5e9e314b5248cc0aa90c6ac1581f
      b1211e4c
    • Ahmed Soliman's avatar
      Move optional python extensions from fbthrift to folly and add missing pyx files to thrift · 9657632b
      Ahmed Soliman authored
      Summary: This is another step of moving the python extensions for folly that are currently defined in thrift's codebase. Another step for getting us closer on unblocking LogDevice to use the python3 thrift client in open-source.
      
      Reviewed By: yfeldblum, vitaut
      
      Differential Revision: D16221352
      
      fbshipit-source-id: bc24fce073d67397f20f28f79c68512444704402
      9657632b
    • Lukas Piatkowski's avatar
      rust-shed: unify Cargo.toml definitions · f2a70ca9
      Lukas Piatkowski authored
      Reviewed By: farnz
      
      Differential Revision: D18726052
      
      fbshipit-source-id: 4f18430342ab6fd4fc82cc7d03e21f3e50e0ce25
      f2a70ca9
    • Yedidya Feldblum's avatar
      Avoid linear recursion in folly::coro::concat · 7b8d4986
      Yedidya Feldblum authored
      Summary: [Folly] Avoid linear recursion in `folly::coro::concat`.
      
      Reviewed By: kirkshoop
      
      Differential Revision: D18746601
      
      fbshipit-source-id: a383715bfc47ae8c0d4baad8799e98699c7cc14f
      7b8d4986
    • Yedidya Feldblum's avatar
      Remove remnants of replacing std::string · 30fa5801
      Yedidya Feldblum authored
      Summary: [Folly] Remove remnants of replacing `std::string` with the implementation of `folly::fbstring`.
      
      Reviewed By: luciang
      
      Differential Revision: D18738367
      
      fbshipit-source-id: b57d38100c9da38c6351145979ac6d2f677ba817
      30fa5801
    • Yedidya Feldblum's avatar
      No unnecessary SFINAE in ElfFile::at · 9b7cf53d
      Yedidya Feldblum authored
      Summary: [Folly] No unnecessary SFINAE in `ElfFile::at` - just use an explicit `static_assert` directly, rather than relying on overload resolution failure for the same goal.
      
      Reviewed By: luciang
      
      Differential Revision: D18746254
      
      fbshipit-source-id: 4bb209bf3f563f2425e4532f6553e66856cb9bbb
      9b7cf53d
    • Yedidya Feldblum's avatar
      try_call_once, a nothrow variant of call_once · 4b6ebaeb
      Yedidya Feldblum authored
      Summary: [Folly] `try_call_once`, a nothrow variant of `call_once`. Takes a `noexcept` function which returns `bool`; the function returning `false` indicates failure and that the `once_flag` is not to be marked as as set, while the function returning `true` indicates success and that the `once_flag` is to be marked as set.
      
      Reviewed By: andriigrynenko, luciang
      
      Differential Revision: D18742590
      
      fbshipit-source-id: 3b3a6fab9a328f7540cc6fee1452a0f1367b6e0f
      4b6ebaeb
    • Yedidya Feldblum's avatar
      Let test_once be noexcept · 47d6f494
      Yedidya Feldblum authored
      Summary: [Folly] Let `test_once` be `noexcept` since it is required not to fail.
      
      Reviewed By: andriigrynenko, luciang
      
      Differential Revision: D18742588
      
      fbshipit-source-id: fda78057d10fca1521ee6335f7c7758d05c43b88
      47d6f494
    • Yedidya Feldblum's avatar
      Hide some test helper functions · ea216b9b
      Yedidya Feldblum authored
      Summary: [Folly] Hide some test helper functions to permit some tests to be linked together into the same binary, if desired.
      
      Differential Revision: D18746529
      
      fbshipit-source-id: 9829eb1227cb348f65734ceb670c85508577763d
      ea216b9b
  6. 28 Nov, 2019 3 commits
    • Yedidya Feldblum's avatar
      Mark most ElfFile members as noexcept · c8c0682f
      Yedidya Feldblum authored
      Summary:
      [Folly] Mark most `ElfFile` members as `noexcept`.
      
      Since they may be used in signal handlers, they are required not to throw exceptions. Mark it so.
      
      Reviewed By: luciang
      
      Differential Revision: D18743292
      
      fbshipit-source-id: 95fb2a20511ca4cc7c0e832c1b20b8324ca7d0af
      c8c0682f
    • Lukas Piatkowski's avatar
      fbcode_builder: add cargo builder for Rust projects · f9b187fe
      Lukas Piatkowski authored
      Summary: The cargo builder will be used to verify if an opensource Rust project passes Cargo build, test and (optionally) documentation build.
      
      Reviewed By: markbt
      
      Differential Revision: D18636934
      
      fbshipit-source-id: e982e6a017eb32913e2994e7457c8add2e9d6b95
      f9b187fe
    • Taylor Hopper's avatar
      Add folly::coro::accumulate algorithm · b1f08da6
      Taylor Hopper authored
      Summary: The `accumulate` algorithm will behave similarly to `std::accumulate`, but instead returns a `Task` of the accumulated result from the values in the `AsyncGenerator`
      
      Reviewed By: kirkshoop
      
      Differential Revision: D18714284
      
      fbshipit-source-id: 66625cd1347d767467d98a421b78b6f45d70260c
      b1f08da6
  7. 27 Nov, 2019 11 commits
    • Yedidya Feldblum's avatar
      Check prev thread-id in EventBase::loopBody · 06e7b633
      Yedidya Feldblum authored
      Summary: [Folly] Check prev thread-id in `EventBase::loopBody` to enforce that it is not called concurrently.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D18722163
      
      fbshipit-source-id: 01e18758e4d517011697e025519c84e29ca96636
      06e7b633
    • Taylor Hopper's avatar
      Add folly::coro::concat algorithm · 522fc79c
      Taylor Hopper authored
      Summary: The `concat` algorithm will concatenate a series of streams known at compile time into a single stream of the same type.
      
      Reviewed By: kirkshoop
      
      Differential Revision: D18695001
      
      fbshipit-source-id: 2da7f53cbc835333ffd9551ceae1aa5db085936d
      522fc79c
    • Jorge Lopez Silva's avatar
      Update utils to print out pre-shared secret. · dc6c17ca
      Jorge Lopez Silva authored
      Summary: This hadn't been updated since we moved to openssl 1.1
      
      Reviewed By: knekritz
      
      Differential Revision: D18668447
      
      fbshipit-source-id: 26d58ef759fccf7e97c4d127b747b093077f6e59
      dc6c17ca
    • Yedidya Feldblum's avatar
      Use relaxed order for EventBase::loopThread_ · 5a370151
      Yedidya Feldblum authored
      Summary:
      [Folly] Use `relaxed` order for `EventBase::loopThread_` consistently. There is only one location which uses a different memory order.
      
      Atomic operations on this location do not guard any other locations, so non-`relaxed` memory orders are not required.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D18722614
      
      fbshipit-source-id: b73b61e4845c34c58e9f4adbeb368c815b375d38
      5a370151
    • Uladzislau Paulovich's avatar
      folly | Add explicit bool conversion operator to Function::SharedProxy · 964cde07
      Uladzislau Paulovich authored
      Summary: `folly::Function` has this explicit bool conversion operator and it makes sense to also have it for `SharedProxy`.
      
      Reviewed By: yfeldblum, ericniebler
      
      Differential Revision: D18504667
      
      fbshipit-source-id: 32fede326c7a3eebd340dea9d118714a0cdfa5c0
      964cde07
    • Lee Howes's avatar
      Add missing container overload of collectAnySemiFuture · 12049572
      Lee Howes authored
      Summary: collectAny had this overload, it was missed when collectAnySemiFuture was added.
      
      Reviewed By: mpark
      
      Differential Revision: D18689251
      
      fbshipit-source-id: 1fccd6eca9d4afde46a173842641f14aff78aada
      12049572
    • Andrii Grynenko's avatar
      Include FOLLY_FIBERS_STACK_OVERFLOW_DETECTED in the stack · a65693ba
      Andrii Grynenko authored
      Summary: Make sure that fiber stack overflow is immediatelly discoverable from the crash stack trace, not just from the error log.
      
      Reviewed By: spalamarchuk
      
      Differential Revision: D18702739
      
      fbshipit-source-id: c0a05081e67b12c94b20a929b2d10396bc1b6949
      a65693ba
    • Yedidya Feldblum's avatar
      Make Function shared proxy empty when empty · 80bef0fa
      Yedidya Feldblum authored
      Summary: [Folly] Make `Function` shared proxy empty when constructed with an empty `Function`, just like it would be when constructed with `nullptr`.
      
      Reviewed By: ericniebler, vitaut
      
      Differential Revision: D18518858
      
      fbshipit-source-id: fdad5b4cd30a1fda5465dae80054c5c74837faa9
      80bef0fa
    • Yedidya Feldblum's avatar
      Let ElfFile open functions return their errors · 38256f47
      Yedidya Feldblum authored
      Summary: [Folly] Let ElfFile open functions return their error messages v.s. taking pointers to where to place them. Overall, making these functions a bit more C++.
      
      Reviewed By: luciang
      
      Differential Revision: D18668222
      
      fbshipit-source-id: 395c20c5d0862c2b1dcc7f6f9db56b7958c5fc90
      38256f47
    • Anton Frolov's avatar
      Make it possible to compile openr through CMakeLists.txt · a8f93172
      Anton Frolov authored
      Summary:
      This diff fixes `CMakeLists.txt` to enable building `openr` tests using CMake:
      1. It adds `add_openr_test` CMake function that adds executable target, registers it as test, links it with bunch of libraries like GTest and GMock, etc...
      2. There is no `openr/tests/OpenrModuleTestBase.cpp` anywhere in the source tree, so this commit replaces it with `openr/common/Flags.cpp`.
      
      Reviewed By: jstrizich
      
      Differential Revision: D18584028
      
      fbshipit-source-id: 07d854ef98d0d2509889a08ad042a371101a2825
      a8f93172
    • Ahmed Soliman's avatar
      Fixing namespace in iobuf python utility extensions and adding it to CMake · 17c33473
      Ahmed Soliman authored
      Summary: Moving some `iobuf` helpers from the `thrift::py3` namespace into `folly` since this now lives in folly. Also fixing CMake to build iobuf in folly not in thrift.
      
      Reviewed By: vitaut
      
      Differential Revision: D16221272
      
      fbshipit-source-id: a7bdde1b60cd6d7de06bf2040506a87c3153c37f
      17c33473
  8. 22 Nov, 2019 6 commits
    • Adam Simpkins's avatar
      Update the eden manifest to drop internal FB-specific dependencies · cc3f054a
      Adam Simpkins authored
      Summary:
      Remove the Facebook-specific dependencies from the Eden manifest: these
      dependencies are now available directly in the Eden repository.
      
      Reviewed By: chadaustin
      
      Differential Revision: D18588009
      
      fbshipit-source-id: 590c74e38e4e05939e6955839af3ebb959f9251b
      cc3f054a
    • Adam Simpkins's avatar
      getdeps: remove the fb-mercurial-rust target · 58fc256b
      Adam Simpkins authored
      Summary:
      Remove the standalone fb-mercurial-rust target that was an internal-only
      dependency for the Eden build.  This build step is now done entirely in the
      Eden build.
      
      Reviewed By: fanzeyi
      
      Differential Revision: D18623943
      
      fbshipit-source-id: c62a1155ddd1c0a6b2270c472176ba25194c6145
      58fc256b
    • Robin Battey's avatar
      added "info fibers" command · 62e82322
      Robin Battey authored
      Summary:
      Re-implementation of fiber gdb integration.  All previous functionality still works as well.
      
      Commands:
      * `fiber`: switch to fiber by address, alias, or assigned short id
      * `fiber apply`: run arbitrary gdb commands for each specified fiber (or "all")
      * `fiber deactivate`: switch back to normal thread behavior
      * `fiber name`: assign or clear an alias to a particular fiber
      * `info fibers`: print out each fiber manager and fiber, with ids and current frame info
      
      Parameters:
      * `fiber info-frame-skip-words`: the "uninteresting" function matches to skip frames of when showing fiber info
      * `fiber manager-print-limit`: limit of the number of fibers displayed when printing a fiber manager
      
      Fibers are assigned ids of the form "<MANAGER_ID>.<FIBER_ID>".  When specifying fibers for `info fibers` or `fiber apply`, you can specify any number of space-separated fiber ids, and additionally you can specify a manager id by itself to refer to all fibers in that manager.
      
      When printing info with `info fibers`, every stopped fiber will be in `folly::fibers::FiberImpl::deactivate()`, which isn't particularly useful.  To give more-useful information, it will skip past all frames whose function names include the strings "folly::fibers" or "wait" and print the first non-matching frame's information instead.  The `fiber info-frame-skip-words` parameter defines the strings to match on, specified as a space-separated list of words.
      
      Due to gdb limitations, the selecting a fiber "masks" the currently selected thread, and to see the current thread again you will need to run `fiber deactivate`, same as before.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D18342802
      
      fbshipit-source-id: 5b2204bb09eca4c8f977c6c17b0348e39465dff6
      62e82322
    • Koray Polat's avatar
      Add SAI-1.4.0 to opensource build · e375f666
      Koray Polat authored
      Summary:
      - Added sai to getdeps
      - added SAI to FBOSS as a dependency
      - Included path of SAI headers to FBOSS's cmake file.
      
      Reviewed By: shri-khare
      
      Differential Revision: D18629127
      
      fbshipit-source-id: 119a6b7f2b64bd84414d9e16a903bc8df48a35e7
      e375f666
    • Lee Howes's avatar
      Add new immutable global CPU executor that is guaranteed to not be changed. · 3c383a2d
      Lee Howes authored
      Summary:
      The mutable global executor is problematic: it defaults to Inline, and it is constructed off of an error-prone weak_ptr and it returns a shared_ptr. A KeepAlive-based global immutable executor is a cleaner default.
      
      This change adds immutable global executors, and moves the default executors into separate singletons for cleaner interaction between the mutable executor and the underlying default immutable ones, including the default inline global cpu executor.
      
      Reviewed By: andriigrynenko
      
      Differential Revision: D18513433
      
      fbshipit-source-id: 0ad825c34cc7ba935f57ff81adb8cff3bf001a45
      3c383a2d
    • Dennis Zhou's avatar
      jemalloc: touch backing pages for JemallocHugePageAllocator · 7f4492d6
      Dennis Zhou authored
      Summary:
      With THP set to madvise, page faults on these pages will block until a
      huge page is found to service it. However, if memory becomes fragmented
      before these pages are touched, then we end up blocking for kcompactd to
      make a page available. As this increase pressure, oomd comes in and kills
      us :(. So, preemptively touch these pages to get them backed as early as
      possible to prevent stalling due to no available huge pages.
      
      Reviewed By: hnaz, interwq
      
      Differential Revision: D18579722
      
      fbshipit-source-id: 5f54676a128207026cb16b9cdc03a055cbb1f9b9
      7f4492d6
  9. 21 Nov, 2019 2 commits
    • Banit Agrawal's avatar
      Never decay the huge pages in JEMalloc · 9113678d
      Banit Agrawal authored
      Summary: This diff disables the purging of huge pages and hence they will never be freed and will be counted towards the RSS of the process. This was done to ensure that we don't block on madvise call to get the huge pages and increase memory pressure on the system.
      
      Reviewed By: gdankel
      
      Differential Revision: D18607196
      
      fbshipit-source-id: 79e483a715290fdfbdd1963433ec3918561a0e0e
      9113678d
    • Aaron Dierking's avatar
      Rename LockTraits.h invokers to work around Clang 7 issue · 6a47d8fb
      Aaron Dierking authored
      Summary:
      Clang 7 seems to have trouble with `FOLLY_CREATE_MEMBER_INVOKER()` on Windows targets if the invoker's name is the same as the invocable name. It's spewing `-Wmicrosoft-explicit-constructor-call` warnings and the `static_assert`s in LockTraits.h are failing. As a workaround, we can rename the LockTraits.h invokers to use an `_invoker` suffix.
      
      See https://gcc.godbolt.org/z/j4qRd9 for a demonstration of the issue.
      
      Reviewed By: vitaut
      
      Differential Revision: D18626335
      
      fbshipit-source-id: c9147daaa10b02b0692c32963f9ede8cc645e329
      6a47d8fb