Summary:
The Chatty subprocess test incorrectly assumed that we saw EOF on the last
read from the child (that is, read() returned 0). That's not the case for two
reasons: 1. the child is allowed to stall right before it closes its stdout, in
which case we get EAGAIN, and 2. Subprocess::communicate would close the fd
without calling the read callback anyway. Fix both such things.

Test Plan: ran test

Reviewed By: njormrod@fb.com

FB internal diff: D1303215

Summary: Makes a fair comparison between asox queue and notification queue.  THe updated benchmark is just noise in this diff, notificationqueue isn't fast enough (yet) to make a difference.

Test Plan:
fbconfig common/concurrency; fbmake opt
./common/concurrency/QueueBenchmark.sh

Reviewed By: afrind@fb.com

FB internal diff: D1272859

Summary: We have these helper methods, but are not using them. What sadness.

Test Plan:
fbconfig -r folly
fbmake runtests

Reviewed By: tudorb@fb.com

FB internal diff: D1299720

Summary:
- Its methods are being used; include the header

@override-unit-failures

Test Plan: - Build for iOS

Reviewed By: tudorb@fb.com

FB internal diff: D1299440

Summary:
Unless a service is overloaded, it should be able to clear
its queue frequently. When this happens, threads fall asleep until
more work is available in the queue. Waking up threads in LIFO
order gives us a lot of benefits. First, threads that were most
recently active are more likely to be mapped to the same cpu core
and thereby experience better L1 cache hit rate. Second, we can
madvise away jemalloc arenas on very idle threads. If we wake up
threads in FIFO order, we will never get a thread to remain idle
long enough for this to be worthwhile.

folly::LifoSem does just that. Benchmark in which the queue is
allowed to drain show that we get a substantial increase in
throughput by waking up threads in LIFO order.

Test Plan:
QueueBenchmark results summary:
As expected, benchmarks run faster in the case where the queue is
able to frequently drain itself, particularly in cases where the
number of threads is large. Benchmarks run slower when the
consumers cannot keep up with the producers, particularly when we
reach the queue capacity and we need to synchronize between
producers and consumers. However, in this case I think we care
less about the overhead of the queue itself and more about how
quickly we can shed the actual underlying load.

Reviewed By: davejwatson@fb.com

FB internal diff: D1298343

Summary: example results were backwards

Test Plan: unit tests

Reviewed By: brg@fb.com

FB internal diff: D1262425

Summary:
... as it comes from Elf::getSymbolName, which returns null-terminated C
strings. So there's no need to copy it into a fixed-size buffer (and have a
buffer overflow, ouch).

Test Plan: folly/experimental/symbolizer_test, see what else "arc unit" gets us

Reviewed By: tconerly@fb.com

FB internal diff: D1286348

Test Plan: unit tests

Reviewed By: lucian@fb.com

FB internal diff: D1290632

Summary:
Support addValueAggregated() for histogram to add multiple samples for
one bucket at one time.

Test Plan: Have a client to call the new API and test in production.

Reviewed By: simpkins@fb.com

FB internal diff: D1286528

Summary:
When worrying about only 2 cpp libraries, detection of "is this libstdc++ or libc++" works, but when throwing in a third option (msvc) this becomes a very brittle way of detecting features.  Also depending on how clang is used it may or may not have this header about

Changed to instead use configuration value for "do we have this header file" for including bits/c++config.h and changed the logic to be identical in all three use cases

Test Plan: fbmake runtests

Reviewed By: njormrod@fb.com

FB internal diff: D1291547

Summary:
Provide translations for gcc packed

MSVC has a very very ugly syntax for packing using pragmas which is very different from the gcc attribute
this makes it very hard to macro around, but lots of ifdefs are also painful - cleanest of the hackery is to simply have push, pop, and attribute macros for packed that define properly depending on the compiler

Test Plan: fbmake runtests

Reviewed By: njormrod@fb.com

FB internal diff: D1279966

Summary:
Allow mutable functors in Promise::fulfil.

Facebook:
Works both with EventBase and without it (thanks to wangle::Future thread-safety). CacheClientImpl lifetime is guaranteed using shared_ptr.

Test Plan: unit test

Reviewed By: stepan@fb.com

FB internal diff: D1291024

Summary:
Fix an off-by-one error causing an ASAN abort.  When calling
longestCommonPrefix() with a full mask, it would compare one byte past
the end of the address data, and would end up writing past the end of
the ba array on the stack.

Test Plan:
Built with ASAN, ran the unit tests, and verified the ASAN failure was
gone.

Reviewed By: jasmeetbagga@fb.com

FB internal diff: D1284750

Summary:
Moving our internal IP/Mac address libraries to folly/

Facebook:
We want to get rid of common/ dependencies in Mcrouter since we're going to open source it. Also looking at the original commit, seems like it's been the intention all along, so I just did it.

I tried to keep dependencies intact as much as possible. Changing projects to use this directly should be in separate diffs.

Test Plan:
Run folly/network and common/network unit tests.

Generate the list of targets with:

```
fbgs /$FILE.h | cut -f1 -d: | xargs -L1 dirname | cut -f2- -d/ | sort | uniq
```

Then fbconfig + fbmake. Will fix contbuild failures.

Revert Plan:

Reviewed By: simpkins@fb.com

FB internal diff: D1261089

Summary: - On iOS, iovec::iov_len is a size_t, which is a uint32_t; convert

Test Plan:
- fbconfig -r folly && fbmake runtests
- Builds in fbobjc

Reviewed By: simpkins@fb.com

FB internal diff: D1284931

Summary:
Headers whose includes are not sufficient cause problems when included
from other files. An easy way to prevent this is to include the header
first in its associated cpp file. In fact, we have a lint rule for this.
However, there are places where this rule is ignored. This diff fixes
many of them.

Note that a few top-level directories were excluded.

Reviewed By: robbert@fb.com

FB internal diff: D1281032

Summary: rwspinlock.h was missing the portability.h header which made the x64 detection go wonky, but only shows up when used with isolated rwspinlock use

Test Plan: fbmake runtests

Reviewed By: andrei.alexandrescu@fb.com

FB internal diff: D1284387

Blame Revision: D1282140

Summary: Another cross platform fun macro - this one detects x64 status and requires some portability.h includes (which should only define portability stuff and not actually include anything)

Test Plan: fbmake runtests

Reviewed By: delong.j@fb.com

FB internal diff: D1282140

Summary: doh
@override-unit-failures

Test Plan: compiled under android

Reviewed By: subodh@fb.com

FB internal diff: D1283177

Summary: the fbstring specific unlikely/likely defs were missing the ifdef blocks to avoid breaking things without builtin_expect

Test Plan: fbmake runtests

Reviewed By: njormrod@fb.com

FB internal diff: D1282166

Summary:
The functor allows to create ranges and sequences that advance the
iterator by more than 1 per iteration.

Test Plan: Unit test.

Reviewed By: tjackson@fb.com

FB internal diff: D1228065

Summary:
Clang chokes because it tries to instantiate both versions of `from<const int*>`, one of which calls `std::begin(const int*)`, which doesn't work. By casting the intializer list to the right type, it can pick the overload.

Clang, because it makes debugging these templates so much better.

Test Plan: `fbconfig --clang folly/gen/test && fbmake runtests_dbg`

Reviewed By: tjackson@fb.com

FB internal diff: D1280888

Summary:
change from using __thread to using FOLLY_THREAD_LOCAL macro, this will allow abstraction over gcc and msvc implementations of thread local (__thread and __declspec(thread)) which have the same semantices and will also allow drop in replacement of thread_local when compiler support for the feature is complete  This doesn't do anything about apple, however, which still has broken __thread support

This doesn't actually change any implementation for now, simply allows for correct compilation

Test Plan: fbmake runtests

Reviewed By: delong.j@fb.com

FB internal diff: D1278726

Summary: MSVC is a stricter compiler and needs additional includes explicitly declared

Test Plan: fbmake runtests

Reviewed By: njormrod@fb.com

FB internal diff: D1277004

Summary:
Adding `... | parallel(my | pipe | line) | ...` for parallelizing a portion of a generator pipeline.

```lang=cpp
auto factored = from(values)
| parallel(filter(isEven) | map(square) | sub(count))
| sum;
```

Work is divided evenly among a fixed number of threads using a `MPMCQueue`.

Test Plan: Unit tests and benchmarks testing for a variety of workloads and performance characteristics, including sub-linear (blocking) workloads, linear (mostly math) workloads, and superlinear (sleeping) workloads to simulate real-world use.

Reviewed By: lucian@fb.com

FB internal diff: D638551

Summary: A LIFO semaphore is useful for building a thread pool because we can keep cpu caches warmer by continually reusing the last used thread. This diff introduces a LIFO semaphore to folly.

Test Plan: unit tests

Reviewed By: davejwatson@fb.com

FB internal diff: D1280405

Summary: folly Uri::host() returns leading and trailing brackets for IPv6 host, for example, for URI "http:://[::1]:8080/index.html", host() method returns "[::1]". But square brackets are not part of host in fact, this diff is going to remove brackets for IPv6 host.

Test Plan: fbconfig -r folly/test && fbmake runtests_dbg

Reviewed By: chip@fb.com

FB internal diff: D1276513

Summary:
Provide translations for gcc noreturn attribute

__attribute__((noreturn)) is gcc specific, clang understands it on nix systems, but for msvc __declspec(noreturn) is the compiler specific version, which clang will imitate/use on windows.  There was already a FOLLY_NORETURN in portability.h, however because of __declspec limitations it must prefix the declaration, not postfix.  The gcc noreturn attribute does not have this limitation and will work prefixed OR postfixed, so to keep from turning code into spaghetti nonsense, the macro was moved to the front of declations where it is currently used.  This will allow it to work with the proper definitions on clang, clang on windows, gcc, and msvc

Test Plan: fbmake runtests

Reviewed By: delong.j@fb.com

FB internal diff: D1279466

Summary:
Provide translations for gcc always_inline and noinline attribute

Change to using a macro from portability.h for attributes for using always_inline and noinline for cross platform support

Test Plan: fbmake runtests

Reviewed By: delong.j@fb.com

FB internal diff: D1279652

Summary:
Using StringPiece makes it easier than std::string

Facebook: Motivation for this diff is found in D1253595

Test Plan:
Added new unittest

fbconfig folly/io/test && fbmake runtests_opt

Reviewed By: simpkins@fb.com

FB internal diff: D1276185

Summary: As part of the windows port, uint is not defined as a type and is not standard, it makes msvc choke, a simple change to unsigned int fixes the issue

Test Plan: fbmake runtests

Reviewed By: delong.j@fb.com

FB internal diff: D1278487

Test Plan: used it

Reviewed By: lucian@fb.com

FB internal diff: D1277063

@override-unit-failures

Summary:
as per discussion from https://phabricator.fb.com/D1274352

After changing the queue length to be a small value (100) I didn't see
improvement from cache alignment of atomics (too much noise). Dropping
that and just fixing the benchmark.

Test Plan: n/a

Reviewed By: ngbronson@fb.com

FB internal diff: D1276591

Summary:
The code originally contained the line

EXPECT_EQ((rcount == lineCount), r); // EOF iff at lineCount

Which asserted that, on the last iteration, EOF was being read (and thus
r was true).

This line has been causing errors in contbuild. There seem to be two
issues at play.

First, the left hand side of the EXPECT_EQ is always false. rcount is
incremented three lines down, and the (implicit) loop ends when rcount
equals lineCount. This means that, on the last iteration, rcount is
actually one less than lineCount. Hence the check is effectively
asserting that r is false.

The second issue is that r is rarely true. Empirically, a necessary but
not sufficient condition is that there be lots of other processes
running at the same time (such as when running fbconfig -r folly &&
fbmake runtests_opt). This would explain why the test passes during
development but fails in contbuild.

I am guessing that, when there are few processes running, Subprocess is
predictably ceasing to read before whichever other process yields EOF,
and that this does not always happen when there is resource contention.

As an immediate fix, I am asserting that r may only be true on the last
iteration. Could someone (cc @tudorb) who is more familiar with the
intricacies of Subprocess please check to see if this is an idiosyncrasy
of the test, or if there is actually an underlying bug.

Test Plan:
Before the change, run fbconfig -r folly && fbmake runtests_opt
repeatedly. Notice that most of the iterations yield an error from line
376 (which is consistent with contbuild error logs).

After the change, notice that the error disappears.

Reviewed By: andrewjcg@fb.com

FB internal diff: D1269124

Summary:
same as map, but runs it's argument in parallel over a number of threads.

@override-unit-failures

Test Plan: added test

Reviewed By: tjackson@fb.com

FB internal diff: D1258364

Summary: Changing the parent's WD is prone to race conditions of various sorts, and needlessly alters parent state. Python's subprocess also has this feature.

Test Plan: fbmake dbg _bin/folly/test/subprocess_test ; ../_bin/folly/test/subprocess_test

Reviewed By: agoder@fb.com

FB internal diff: D1269526

Summary:
* folly/test/CacheLocalityTest.cpp (contentionAtWidth): Work around
clang's prohibition against using flexible arrays in a lambda.

Test Plan:
fbconfig --clang --with-project-version clang:dev folly/test \
&& fbmake runtests

Reviewed By: ngbronson@fb.com

FB internal diff: D1263620

Summary: Allow specifying custom alignment.

Test Plan: fbconfig -r folly/test && fbmake opt -j32

@override-unit-failures

Reviewed By: lucian@fb.com

FB internal diff: D1264851

Summary:
D1261546 introduced regression in `sizeof(ConcurrentSkipList)`.
In case of `NodeAlloc = folly::SysAlloc`, size of an empty `NodeAlloc` struct
is 1 byte, due to the alignment requirements of the next field, it got
translated into wasting 8 bytes.

Test Plan: fbconfig -r folly/test && fbmake opt -j32

@override-unit-failures

Reviewed By: lucian@fb.com

FB internal diff: D1264730