Summary: D1615707 moved State.h to Core.h, but didn't update Makefile.am

Test Plan: visual

Reviewed By: davejwatson@fb.com

Subscribers: fugalh, njormrod

FB internal diff: D1617562

Blame Revision: D1615707

Summary:
See https://gcc.gnu.org/onlinedocs/gcc-4.8.3/gcc/Thread-Local.html
Thanks to @subodh for finding the first instance of this problem in the
proxygen codebase.

Test Plan:
unit tests, tested proxygen unit tests on my laptop and I don't
get segfaults on program exit now.

Reviewed By: subodh@fb.com

Subscribers: trunkagent, doug, njormrod, subodh

FB internal diff: D1616149

Tasks: 5217022

Summary:
spaghetti-monster

Finagle uses a nifty state machine pattern in `Promise.scala`. Each state carries its data with it, then you switch statements with a simple state `cas`, within the functions (which are the transition inputs). See https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Promise.scala#L133-L139 and https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Promise.scala#L604-L635 for example.

I was thinking we could do something similar in C++, though we don't quite have the same luxury of type cases like scala. (Although as an aside I found this really cool paper on implementing type cases in C++, it's kind of evil but very cool. http://www.stroustrup.com/OOPSLA-typeswitch-draft.pdf)

I was looking at having a union of the different state classes, e.g.

union U {
State base;
Done done;
Waiting waiting;
...
};

and then you could use the memoized `dynamic_cast` trick in that paper to make a fairly efficient type case (and fairly readable, depending on how evil you want to get with the macros). However, `dynamic_cast<Done*>(&u.base)` blissfully segfaults. I'm not sure if this is something that should work and it's a compiler bug, or whether trying to (ab)use a union this way is against some arcane C++ rules. @hannesr suggested maybe a variant type might work. We can also do memory mangling on our own if it comes to it - however those are all more advanced techniques to play with at a later optimization time.

So instead, I went for a this-is-context helper base class. The mutable context is just `this`, and you inherit from `FSM`.

The magic macros make it more succint to lay out state transitions. See the tests for examples. Maybe in the future we can get really clever and find a way to generate state machine diagrams from code using this, especially when macros are being used.

Test Plan: unit tests were written and pass

Reviewed By: davejwatson@fb.com

Subscribers: meisner, trunkagent, net-systems@, fugalh, exa, njormrod, hannesr

FB internal diff: D1613497

Summary:
codemod
`State` is such an overloaded term, and not really the best to describe this backing future/promise object. Yes, it holds the state but it's more than that and it gets in the way of calling the states of the state machines `State`s.

Test Plan: builds and tests pass

Reviewed By: davejwatson@fb.com

Subscribers: trunkagent, net-systems@, fugalh, exa, njormrod

FB internal diff: D1615707

Summary:
This forbids singleton creation/re-creation after destroyInstances() was called.
Only after reset() is called singletons can be created again.

registrationComplete() behavior is also slightly change. We disallow singleton registration after registrationComplete() is called even in Relaxed mode. Strict mode now only controlls whether singletons can be constructed before registation is complete.

Test Plan: unit test

Reviewed By: chip@fb.com

Subscribers: hphp-diffs@, ps, trunkagent, njormrod, mshneer, lins

FB internal diff: D1605136

Summary:
[Folly] Basic tests for EventHandler.

Worthwhile adding some tests just for this class since it's rather fundamental. It is already tested indirectly via any number of things that make use of EventBase under the hood, but it's worth having a place for more direct tests.

Test Plan: Unit tests in `folly/io/async/test/EventHandlerTest.cpp`.

Reviewed By: njormrod@fb.com

Subscribers: njormrod, chalfant, dougw

FB internal diff: D1606284

Test Plan: fbconfig -r folly/test && fbmake runtests

Reviewed By: mpawlowski@fb.com

Subscribers: njormrod

FB internal diff: D1604761

Blame Revision: D1420588

Summary:
D1428504 changed the `detail::State` mutex to be a recursive mutex to get around deadlock in our project, where a promise was being freed within a callback, racing with the mechanics happening while fulfilling the promise in the first place. At least, that's what seemed to be happening. I couldn't easily pull it into gdb because it's a python test.

I made my own test to repro, and it did in fact deadlock:

TEST(Future, DISABLED_promiseDestructedDuringCallback) {
auto p = folly::make_unique<Promise<void>>();
p->getFuture().then([&](Try<void>&&) {
// shouldn't deadlock.
p.reset();
});
p->setValue();
}

However, although this code fixes our project it does not fix this code, which still fails (not with a deadlock, but with a promiseAlreadyFulfilled exception). So whatever our project is doing is a bit more intricate.
I'm not convinced that even allowing this is ok - so I suspect out project is doing something bad. However, I also think it's probably bad to hold the lock while doing the callback so I am presenting this as a fix/compromise.

Test Plan:
unit tests

Reviewed By: hannesr@fb.com

Subscribers: net-systems@, fugalh, exa, njormrod, fsilva, davejwatson

FB internal diff: D1604829

Blame Revision: D1428504

Test Plan: existing tests

Reviewed By: meyering@fb.com

Subscribers: trunkagent, bmatheny, ranjeeth, njormrod, subodh

FB internal diff: D1605942

Tasks: 5183325

Test Plan: Tests under folly passed

Reviewed By: seanc@fb.com

Subscribers: trunkagent, seanc, njormrod

FB internal diff: D1600266

Tasks: 5317470

Summary:
The emulated futex (not used on Linux) has an optimization to defer
notification until after the bucket mutex has been unlocked, to avoid
lock collisions between the waiter and waker.  That code will have a
use-after-free problem if the waiter's condition_variable has a spurious
wakeup, which is allowed by the spec.  That code also doesn't do
anything about contention between multiple waiters.

This diff adds a mutex to each wait node, relieving the waiters from
having to acquire the bucket lock on wakeup.  Rather than trying to
perform a racy late notification, we just make sure to release the node
lock immediately after calling notify_one, which seems to have the
desired effect.

Test Plan:
1) existing unit tests
2) new unit tests

Reviewed By: delong.j@fb.com

Subscribers: boseant, njormrod

FB internal diff: D1602360

Summary:
This class isn't thrift specific anymore, especially now that TEventBase->EventBase.
Specific use case in folly: folly/experimental/wangle/concurrent/IOThreadPoolExecutor
EventBaseManager itself needs some work/cleanup, but that is for a later diff
For instance, we might try to push towards only allowing access to the singleton, and towards
removing overrides. i.e. only getEventBase. But that's pending an audit of how people are using it now.

Note that the ProfiledMutex protecting the event base set has been made a regular std::mutex

Test Plan:
compiled thrift/lib/cpp|cpp2, made a pass at fixing forward declarations elsewhere
will let contbuild help me iterate

Reviewed By: hans@fb.com

Subscribers: trunkagent, nli, fbcode-common-diffs@, davejwatson, hero-diffs@, zeus-diffs@, andrewcox, netego-diffs@, alandau, apollo-diffs@, antonl, laser-diffs@, ads-dsp-eng@, darshan, micha, njormrod, panin, hdoshi, scuba-diffs@, bmatheny

FB internal diff: D1590827

Tasks: 5247981

Summary:
std::atomic<bool> default constructor doesn't initialize it to
false, so the flag to end the LifoSem.multi_try_wait test was sometimes
set too early.  This diff fixes that, makes the test more deterministic,
and also fixes a couple of other benign uninitialized values (so that
they won't be used as prototypes for places where it does matter).

Test Plan: unit tests

Reviewed By: mpawlowski@fb.com

Subscribers: njormrod, lovro

FB internal diff: D1604508

Tasks: 5336837

Summary: Sometimes you just have to take a step back. :-P

Test Plan: All the unit tests including the one that had been disabled, now pass.

Reviewed By: hannesr@fb.com

Subscribers: meisner, trunkagent, net-systems@, fugalh, exa, njormrod, davejwatson

FB internal diff: D1596368

Tasks: 4920689, 4480567, 5306911

Summary:
previous change would cause reserving
as much as needed in toAppend for all arguements.
This had bad consequences for appending in a loop,
described here: https://phabricator.fb.com/D1420588#22
Not split the interfaces so that user has to decide

Test Plan: unit tests

Reviewed By: soren@fb.com

Subscribers: trunkagent, njormrod

FB internal diff: D1601614

Tasks: 5303991

Summary: Because windtunnel quirkloads needs `--json` which is enabled with gflags

Test Plan:
$ wangle-bench --json
{
"%hundredThens" : 28133155.113220215,
"no_contention" : 4842268655.11322,
"%fourThens" : 1495655.1132202148,
"%twoThens" : 882311.3632202148,
"oneThen" : 581053.5507202148,
"-" : 0,
"%withThen" : 559830.8944702148,
"%promiseAndFuture" : 250840.66009521484,
"%contention" : 8074419655.11322,
"constantFuture" : 239916.83197021484
}

Reviewed By: meisner@fb.com

Subscribers: robbert, net-systems@, fugalh, exa, njormrod, davejwatson, jsedgwick

FB internal diff: D1601364

Tasks: 5277907

Summary: This is an attempt to see the effect of lock contention in the less-common case where a promise is fulfilled at the same time the future is still being set up. Although this is expected to be off the beaten path, it will still be a good idea to keep an eye on how we improve or worsen perf, when we play with different locking strategies (which I intend to do a lot of in the near term).

Test Plan:
============================================================================
folly/wangle/test/Benchmark.cpp                 relative  time/iter  iters/s
============================================================================
constantFuture                                             249.31ns    4.01M
promiseAndFuture                                 100.88%   247.13ns    4.05M
withThen                                          43.87%   568.30ns    1.76M
----------------------------------------------------------------------------
oneThen                                                    569.62ns    1.76M
twoThens                                          63.46%   897.60ns    1.11M
fourThens                                         39.64%     1.44us  695.90K
hundredThens                                       1.91%    29.75us   33.61K
----------------------------------------------------------------------------
serial                                                       4.97ms   201.21
parallel                                          60.22%     8.25ms   121.18
============================================================================

Reviewed By: jsedgwick@fb.com

Subscribers: meisner, net-systems@, fugalh, exa, njormrod

FB internal diff: D1593010

Summary:
Inside the Linux kernel, futex() works by hashing the source
address to a fixed set of wakeup lists.  When using folly on non-Linux
systems we can emulate something similar by using std::mutex and
std::condition_variable.

Emulating futex() using higher-level APIs is less crazy than it sounds,
because the emulated futex still provides the advantages of the real
one: it allows all of the fast-paths of a new synchronization construct
to be inlined; it is space efficient, taking only 1 word and allowing
the caller to encode state into all of the word's bits; and it avoids
system calls unless a thread actually needs to be put to sleep or
woken up.  Think of this as a way of boostrapping something with the
same properties as futex() on platforms that don't expose it publically.
(Presumably these platforms have private APIs that do something similar.)

This diff moves all of the Linux-specific futex stuff into Futex.cpp,
where it is gated by #ifdef __linux__.  It also adds an emulated
implementation.  The emulated futex will be selected by default on
non-Linux platforms, or it can be used on Linux by using an Atom template
type of folly::detail::EmulatedFutexAtomic.  This means, for example,
that you can test MPMCQueue on top of the emulated API by instantiating
a MPMCQueue<ElemType, EmulatedFutexAtomic>.

As a bonus, this refactoring provides a small speed boost by removing
an unnecessary evaluation of the errno macro in the success path of
futexWait.

Test Plan:
1. existing unit tests
2. new unit tests (including tests of Futex users)
3. compile Futex.cpp on OS X (some other build failures still occur)

Reviewed By: delong.j@fb.com

Subscribers: trunkagent, njormrod, yiding, boseant, mssarang

FB internal diff: D1596118

Tasks: 4952724

Summary: We are seeing crashes which comes from the initialization of the static global variable. This particular variable is used only in a single function that was never invoked. So moving it into the scope of the function will at least solve the problem. The real issue still requires some deep investigation though.

Test Plan: unitest under folly passed

Reviewed By: subodh@fb.com

Subscribers: seanc, njormrod

FB internal diff: D1598048

Tasks: 5316441

Summary:
Malloc.h now takes care to declare mallctl properly on platforms
that aren't using weak symbols, so we should just rely on that for
MemoryIdler.cpp.  This is one step toward fixing folly build on OS X.

Test Plan: fbmake runtests

Reviewed By: je@fb.com

Subscribers: yiding, njormrod

FB internal diff: D1596930

Tasks: 4952724

Blame Revision:

Summary:
This will ensure SingletonVault is always available. It matters for singletons, not managed by folly::Singleton. Singletons in it will actually be destroyed via static SingletonVaultDestructor.

This does the same as D1591270 (which got reverted), but doesn't change destruction order.

Test Plan:
Run tests which were broken by D1591270

Reviewed By: chip@fb.com

Subscribers: njormrod

FB internal diff: D1594898

Test Plan:
unit tests

Reviewed By: ostap@fb.com

Subscribers: netego-diffs@, trunkagent, alikhtarov, marccelani, mshneer, zhuohuang, lins, njormrod

FB internal diff: D1560406

Summary: Memory corruption investigation

Test Plan: fbconfig folly/test:network_address_test; fbmake runtests

Reviewed By: davejwatson@fb.com

Subscribers: trunkagent, andrii, njormrod

FB internal diff: D1592539

Tasks: 5230657

Blame Revision: D1575098

Summary:
This reverts commit 3cf88fb680b4b9c47189d1e12699e2c24c7ac38b. It
was timing out our tests. probably static (de)initialization order fiasco related.

Test Plan: watch contbuild, git bisect

Reviewed By: afrind@fb.com

Subscribers: doug, njormrod

FB internal diff: D1593016

Summary: This will ensure SingletonVault is always available. It matters for singletons, not managed by folly::Singleton. Singletons in it will actually be destroyed via static SingletonVaultDestructor.

Test Plan: unit test

Reviewed By: chip@fb.com

Subscribers: njormrod

FB internal diff: D1591270

Summary: For great speed

Test Plan:
$ fbmake opt && _bin/folly/wangle/wangle-bench
============================================================================
folly/wangle/test/Benchmark.cpp                 relative  time/iter  iters/s
============================================================================
constantFuture                                             235.92ns    4.24M
promiseAndFuture                                 100.37%   235.04ns    4.25M
withThen                                          41.97%   562.17ns    1.78M
----------------------------------------------------------------------------
oneThen                                                    539.03ns    1.86M
twoThens                                          64.01%   842.14ns    1.19M
fourThens                                         39.27%     1.37us  728.62K
hundredThens                                       1.82%    29.63us   33.75K
============================================================================

Reviewed By: davejwatson@fb.com

Subscribers: trunkagent, net-systems@, fugalh, exa, njormrod

FB internal diff: D1587871

Summary:
I'm not thrilled with this implementation, but it mostly works. The big bummer is enforcing that Observables are held in shared_ptrs, or rather that enforcing that condition is impossible.
The protected constructor / friended dumb make_shared() pattern is clunky, and it'd be really easy for a subclasser to shoot themselves in the foot or even in the face.

It does seem like maybe Observable should be made an interface again, and all these details should live in a subclass (FanoutObservable?) where the restriction are super obvious. For instance, the langtech AudioStream object doesn't need all this crap because it overrides subscribe() without using the observer list, but it subclasses anyways.

I'm noodling another approach that (if it works) will not require the shared_ptr dancing, but will probably have some additional overhead... the observable would have to keep track of the subscriptions itself.

I like the RAII subscriptions, though perhaps subscriptions should be optional as long as it's clear that your subscription will last forever it you opt out of them.

Thoughts?

Test Plan: added unit

Reviewed By: hans@fb.com

Subscribers: rushix, hannesr, trunkagent, fugalh, mwa, jgehring, fuegen, njormrod, bmatheny

FB internal diff: D1580443

Summary:
This adds versions of stringPrintf() and stringAppendf() that accept
va_list arguments.

This also fixes the existing stringPrintf() functions to properly call
va_end() even when an exception is thrown in stringPrintfImpl().

Test Plan: Included new unit tests for stringVPrintf() and stringVAppendf().

Reviewed By: davejwatson@fb.com

Subscribers: trunkagent, doug, net-systems@, exa, njormrod

FB internal diff: D1583675

Summary: needed for thrift server. if you have some sort of stateful/functional factory and you want to hold on to it, you should be allowed to

Test Plan: compiles with forthcoming thrift diff

Reviewed By: davejwatson@fb.com

Subscribers: trunkagent, fugalh, njormrod

FB internal diff: D1584587

Summary:
I'm not 100% sure this is the best way to go about this but I don't hate it either.
I'm going to start seeing how it might fit into tserver - my guess is that some sort Cpp2WorkerFactory which also manages those objects would get plugged in as the thread factory
Haven't fleshed out how this would relate to TEventBaseManager

Test Plan: added unit, starting to play with this in Thrift2 server

Reviewed By: davejwatson@fb.com

Subscribers: alandau, bmatheny, trunkagent, fugalh, njormrod

FB internal diff: D1574660

Summary:
Could the problem be that via continues the existing chain of futures,
whereas we actually want to start a new chain?

Is there any particular reason this wasn't implemented like this originally?

Test Plan:
Ran all the unit tests. I hope to try to reproduce the thread issue and
see if this improves things.

Reviewed By: davejwatson@fb.com

Subscribers: trunkagent, net-systems@, exa, njormrod, fugalh

FB internal diff: D1500225

Tasks: 4920689

Summary:

Test Plan: fbmake runtests, OBC no longer segfaults

Reviewed By: mshneer@fb.com

Subscribers: fbcode-common-diffs@, trunkagent, mcduff, marccelani, hitesh, mshneer, njormrod, lins

FB internal diff: D1573880

Summary: I'm going to have hhvm depend on Subprocess, found this build error in open source.  I figured the intent was to set r to the return code.  I could also delete the whole thing since it's unused.

Test Plan: Unit tests

Reviewed By: tudorb@fb.com

Subscribers: trunkagent, mwilliams, doug, njormrod

FB internal diff: D1583626

Summary: Some minor bug and a failed test caused by D1578466

Test Plan:
fbconfig thrift/lib/cpp/test:HHWheelTimerTest && fbmake
runtests

Reviewed By: davejwatson@fb.com

Subscribers: trunkagent, alandau, bmatheny, njormrod

FB internal diff: D1581949

Summary: We only need exclusive lock when we add items to singletons_. Each SingletonEntry has its own mutex, so it's safe to rely on it for any modifications within individual entries.

Test Plan: Applied D1573880 and ran fbconfig -r servicerouter/client/cpp2 && fbmake runtests

Reviewed By: chip@fb.com

Subscribers: trunkagent, njormrod, hitesh, mshneer

FB internal diff: D1579877

Summary:
Because we put FBString and Malloc into libgcc, we have to be
careful which dependencies they each have. We cannot include ScopeGuard.
Reorganize the code a bit to avoid the ScopeGuard.

Test Plan: fbconfig -r folly && fbmake runtests

Reviewed By: je@fb.com

Subscribers: trunkagent, sdwilsh, njormrod

FB internal diff: D1580957

Summary:
By the birthday paradox, this will fail about once every 8,100
runs. Dropping to 100 with cut that to 1 in 860,000.

Test Plan: fbconfig -r folly && fbmake runtests

Reviewed By: sdoroshenko@fb.com

Subscribers: sdwilsh, njormrod

FB internal diff: D1581238