DynamicParser to reliably and reversibly convert JSON to structs

Summary:We have a bunch of code that manually parses `folly::dynamic`s into program structures. I can be quite hard to get this parsing to be good, user-friendly, and concise. This diff was primarily motivated by the mass of JSON-parsing done by Bistro, but this pattern recurs in other pieces of internal code that parse dynamics.

This diff **not** meant to replace using Thrift structs with Thrift's JSON serialization / deserialization. When all you have to deal with is correct, structured plain-old-data objects produced by another program -- **not** manually entered user input -- Thrift + JSON is perfect. Go use that.

However, sometimes you need to parse human-edited configuration. The input JSON might have complex semantics, and require validation beyond type-checking. The UI for editing your configs can easily enforce correct JSON syntax. Perhaps, you can use `folly/experimental/JSONSchema.h` to have your edit UI provide type correctness. Despite all this, people can still make semantic errors, and those can be impossible to detect until you interpret the config at runtime. Also, as your system evolves, sometimes you need to break semantic backwards-compatibility for the sake of moving forward ? thus making previously valid configurations invalid, and requiring them to be fixed up manually.

So, people end up needing to write manual parsers for `dynamic`s. These all have very similar recurring issues:

 - Verbose: to get an int field out of an object, typical code: (i) tests if the field is present, (ii) checks if the field is an integer, (iii) extracts the integer. Sometimes, you also want to handle exceptions, and compose helpful error messages. This makes the code far longer than its intent, and encourages people to write bad parsers.

 - Unsystematic: sometimes, we use `if (const auto* p = dyn_obj.get_ptr("key")) { ... }`, other times we use `dyn_obj.getDefault()` or `if (dyn_obj.count())`, and so on. The patterns differ subtly in meaning. Exceptions sometimes get thrown, leading to error messages that cannot be understood by the user.

 - Imperative parses: a typical parse proceeds step by step, and throws at the earliest error. This is bad because (i) errors have to be fixed one-by-one, instead of getting a full list upfront, (ii) even if 99% of the config is parseable, the imperative code has no way of recording the information it would have parsed after the first error.

 `DynamicParser` fixes all of the above, and makes your parsing so clean that you might not even bother with `JSONSchema` as your first line of defense -- type-coercing, type-enforcing, friendly-error-generating C++ ends up being more concise. Besides all the sweet syntax sugar, `DynamicParser` lets you parse **all** the valid data in your config, while recording  *all* the errors in a way that does not lose the original, buggy config. This means your code can parse a config that has errors, and still be able to meaningfully export it back to JSON. As a result, stateless clients (think REST APIs) can provide a far better user experience than just discarding the user?s input, and returning a cryptic error message.

For the details, read the docs (and see the example) in `DynamicParser.h`. Here are the principles of `DynamicParser::RECORD` mode in a nutshell:
 - Pre-populate your program struct with meaningful defaults **before** you parse.
 - Any config part that fails to parse will keep the default.
 - Any config part that parses successfully will get to update the program struct.
 - Any errors will be recorded with a helpful error message, the portion of the dynamic that caused the error, and the path through the dynamic to that portion.

 I ported Bistro to use this in D3136954. I looked at using this for JSONSchema's parsing of schemas, but it seemed like too much trouble for the gain, since it would require major surgery on the code.

Reviewed By: yfeldblum

Differential Revision: D2906819

fb-gh-sync-id: aa997b0399b17725f38712111715191ffe7f27aa
fbshipit-source-id: aa997b0399b17725f38712111715191ffe7f27aa

folly/Makefile.am

@@ -86,6 +86,8 @@ nobase_follyinclude_HEADERS = \
 	experimental/AutoTimer.h \
 	experimental/Bits.h \
 	experimental/BitVectorCoding.h \
+	experimental/DynamicParser.h \
+	experimental/DynamicParser-inl.h \
 	experimental/ExecutionObserver.h \
 	experimental/EliasFanoCoding.h \
 	experimental/EventCount.h \
@@ -448,6 +450,7 @@ libfolly_la_SOURCES = \
 	Version.cpp \
 	experimental/bser/Dump.cpp \
 	experimental/bser/Load.cpp \
+	experimental/DynamicParser.cpp \
 	experimental/fibers/Baton.cpp \
 	experimental/fibers/Fiber.cpp \
 	experimental/fibers/FiberManager.cpp \

folly/experimental/DynamicParser-inl.h

+/*
+ * Copyright 2016 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/*
+ *  Copyright (c) 2015, Facebook, Inc.
+ *  All rights reserved.
+ *
+ *  This source code is licensed under the BSD-style license found in the
+ *  LICENSE file in the root directory of this source tree. An additional grant
+ *  of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+#pragma once
+
+#include <folly/Conv.h>
+#include <boost/function_types/is_member_pointer.hpp>
+#include <boost/function_types/parameter_types.hpp>
+#include <boost/mpl/equal.hpp>
+#include <boost/mpl/pop_front.hpp>
+#include <boost/mpl/transform.hpp>
+#include <boost/mpl/vector.hpp>
+
+namespace folly {
+
+// Auto-conversion of key/value based on callback signature, documented in
+// DynamicParser.h.
+namespace detail {
+class IdentifyCallable {
+public:
+  enum class Kind { Function, MemberFunction };
+  template <typename Fn>
+  constexpr static Kind getKind() { return test<Fn>(nullptr); }
+private:
+  template <typename Fn>
+  using IsMemFn = typename boost::function_types::template is_member_pointer<
+    decltype(&Fn::operator())
+  >;
+  template <typename Fn>
+  constexpr static typename std::enable_if<IsMemFn<Fn>::value, Kind>::type
+  test(IsMemFn<Fn>*) { return IdentifyCallable::Kind::MemberFunction; }
+  template <typename>
+  constexpr static Kind test(...) { return IdentifyCallable::Kind::Function; }
+};
+
+template <IdentifyCallable::Kind, typename Fn>
+struct ArgumentTypesByKind {};
+template <typename Fn>
+struct ArgumentTypesByKind<IdentifyCallable::Kind::MemberFunction, Fn> {
+  using type = typename boost::mpl::template pop_front<
+    typename boost::function_types::template parameter_types<
+      decltype(&Fn::operator())
+    >::type
+  >::type;
+};
+template <typename Fn>
+struct ArgumentTypesByKind<IdentifyCallable::Kind::Function, Fn> {
+  using type = typename boost::function_types::template parameter_types<Fn>;
+};
+
+template <typename Fn>
+using ArgumentTypes =
+  typename ArgumentTypesByKind<IdentifyCallable::getKind<Fn>(), Fn>::type;
+
+// At present, works for lambdas or plain old functions, but can be
+// extended.  The comparison deliberately strips cv-qualifieers and
+// reference, leaving that choice up to the caller.
+template <typename Fn, typename... Args>
+constexpr bool hasArgumentTypes() {
+  using HasArgumentTypes = typename boost::mpl::template equal<
+    typename boost::mpl::template transform<
+      typename boost::mpl::template transform<
+        ArgumentTypes<Fn>,
+        typename std::template remove_reference<boost::mpl::_1>
+      >::type,
+      typename std::template remove_cv<boost::mpl::_1>
+    >::type,
+    boost::mpl::vector<Args...>
+  >::type;
+  return HasArgumentTypes::value;
+}
+template <typename... Args>
+using EnableForArgTypes =
+  typename std::enable_if<hasArgumentTypes<Args...>(), void>::type;
+
+// No arguments
+template <typename Fn> EnableForArgTypes<Fn>
+invokeForKeyValue(Fn f, const folly::dynamic&, const folly::dynamic&) {
+  f();
+}
+
+// 1 argument -- pass only the value
+//
+// folly::dynamic (no conversion)
+template <typename Fn> EnableForArgTypes<Fn, folly::dynamic>
+invokeForKeyValue(Fn fn, const folly::dynamic&, const folly::dynamic& v) {
+  fn(v);
+}
+// int64_t
+template <typename Fn> EnableForArgTypes<Fn, int64_t>
+invokeForKeyValue(Fn fn, const folly::dynamic&, const folly::dynamic& v) {
+  fn(v.asInt());
+}
+// bool
+template <typename Fn> EnableForArgTypes<Fn, bool>
+invokeForKeyValue(Fn fn, const folly::dynamic&, const folly::dynamic& v) {
+  fn(v.asBool());
+}
+// double
+template <typename Fn> EnableForArgTypes<Fn, double>
+invokeForKeyValue(Fn fn, const folly::dynamic&, const folly::dynamic& v) {
+  fn(v.asDouble());
+}
+// std::string
+template <typename Fn> EnableForArgTypes<Fn, std::string>
+invokeForKeyValue(Fn fn, const folly::dynamic&, const folly::dynamic& v) {
+  fn(v.asString().toStdString());
+}
+
+//
+// 2 arguments -- pass both the key and the value.
+//
+
+// Pass the key as folly::dynamic, without conversion
+//
+// folly::dynamic, folly::dynamic (no conversion of value, either)
+template <typename Fn> EnableForArgTypes<Fn, folly::dynamic, folly::dynamic>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k, v);
+}
+// folly::dynamic, int64_t
+template <typename Fn> EnableForArgTypes<Fn, folly::dynamic, int64_t>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k, v.asInt());
+}
+// folly::dynamic, bool
+template <typename Fn> EnableForArgTypes<Fn, folly::dynamic, bool>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k, v.asBool());
+}
+// folly::dynamic, double
+template <typename Fn> EnableForArgTypes<Fn, folly::dynamic, double>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k, v.asDouble());
+}
+// folly::dynamic, std::string
+template <typename Fn> EnableForArgTypes<Fn, folly::dynamic, std::string>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k, v.asString().toStdString());
+}
+
+// Convert the key to std::string.
+//
+// std::string, folly::dynamic (no conversion of value)
+template <typename Fn> EnableForArgTypes<Fn, std::string, folly::dynamic>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k.asString().toStdString(), v);
+}
+// std::string, int64_t
+template <typename Fn> EnableForArgTypes<Fn, std::string, int64_t>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k.asString().toStdString(), v.asInt());
+}
+// std::string, bool
+template <typename Fn> EnableForArgTypes<Fn, std::string, bool>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k.asString().toStdString(), v.asBool());
+}
+// std::string, double
+template <typename Fn> EnableForArgTypes<Fn, std::string, double>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k.asString().toStdString(), v.asDouble());
+}
+// std::string, std::string
+template <typename Fn> EnableForArgTypes<Fn, std::string, std::string>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k.asString().toStdString(), v.asString().toStdString());
+}
+
+// Convert the key to int64_t (good for arrays).
+//
+// int64_t, folly::dynamic (no conversion of value)
+template <typename Fn> EnableForArgTypes<Fn, int64_t, folly::dynamic>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k.asInt(), v);
+}
+// int64_t, int64_t
+template <typename Fn> EnableForArgTypes<Fn, int64_t, int64_t>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k.asInt(), v.asInt());
+}
+// int64_t, bool
+template <typename Fn> EnableForArgTypes<Fn, int64_t, bool>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k.asInt(), v.asBool());
+}
+// int64_t, double
+template <typename Fn> EnableForArgTypes<Fn, int64_t, double>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k.asInt(), v.asDouble());
+}
+// int64_t, std::string
+template <typename Fn> EnableForArgTypes<Fn, int64_t, std::string>
+invokeForKeyValue(Fn fn, const folly::dynamic& k, const folly::dynamic& v) {
+  fn(k.asInt(), v.asString().toStdString());
+}
+}  // namespace detail
+
+template <typename Fn>
+void DynamicParser::optional(const folly::dynamic& key, Fn fn) {
+  wrapError(&key, [&]() {
+    if (auto vp = value().get_ptr(key)) {
+      parse(key, *vp, fn);
+    }
+  });
+}
+
+//
+// Implementation of DynamicParser template & inline methods.
+//
+
+template <typename Fn>
+void DynamicParser::required(const folly::dynamic& key, Fn fn) {
+  wrapError(&key, [&]() {
+    auto vp = value().get_ptr(key);
+    if (!vp) {
+      throw std::runtime_error(folly::to<std::string>(
+        "Couldn't find key ", detail::toPseudoJson(key), " in dynamic object"
+      ));
+    }
+    parse(key, *vp, fn);
+  });
+}
+
+template <typename Fn>
+void DynamicParser::objectItems(Fn fn) {
+  wrapError(nullptr, [&]() {
+    for (const auto& kv : value().items()) {  // .items() can throw
+      parse(kv.first, kv.second, fn);
+    }
+  });
+}
+
+template <typename Fn>
+void DynamicParser::arrayItems(Fn fn) {
+  wrapError(nullptr, [&]() {
+    size_t i = 0;
+    for (const auto& v : value()) {  // Iteration can throw
+      parse(i, v, fn);  // i => dynamic cannot throw
+      ++i;
+    }
+  });
+}
+
+template <typename Fn>
+void DynamicParser::wrapError(const folly::dynamic* lookup_k, Fn fn) {
+  try {
+    fn();
+  } catch (DynamicParserLogicError& ex) {
+    // When the parser is misused, we throw all the way up to the user,
+    // instead of reporting it as if the input is invalid.
+    throw;
+  } catch (DynamicParserParseError& ex) {
+    // We are just bubbling up a parse error for OnError::THROW.
+    throw;
+  } catch (const std::exception& ex) {
+    reportError(lookup_k, ex);
+  }
+}
+
+template <typename Fn>
+void DynamicParser::parse(
+    const folly::dynamic& k, const folly::dynamic& v, Fn fn) {
+  auto guard = stack_.push(k, v);  // User code can nest parser calls.
+  wrapError(nullptr, [&]() { detail::invokeForKeyValue(fn, k, v); });
+}
+
+inline const folly::dynamic& DynamicParser::ParserStack::key() const {
+  if (!key_) {
+    throw DynamicParserLogicError("Only call key() inside parsing callbacks.");
+  }
+  return *key_;
+}
+
+inline const folly::dynamic& DynamicParser::ParserStack::value() const{
+  if (!value_) {
+    throw DynamicParserLogicError(
+      "Parsing nullptr, or parsing after releaseErrors()"
+    );
+  }
+  return *value_;
+}
+
+}  // namespace folly

folly/experimental/DynamicParser.cpp

+/*
+ * Copyright 2016 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/*
+ *  Copyright (c) 2015, Facebook, Inc.
+ *  All rights reserved.
+ *
+ *  This source code is licensed under the BSD-style license found in the
+ *  LICENSE file in the root directory of this source tree. An additional grant
+ *  of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+#include <folly/experimental/DynamicParser.h>
+#include <folly/CppAttributes.h>
+
+#include <folly/Optional.h>
+
+namespace folly {
+
+namespace {
+folly::dynamic& insertAtKey(
+    folly::dynamic* d, bool allow_non_string_keys, const folly::dynamic& key) {
+  if (key.isString()) {
+    return (*d)[key];
+  // folly::dynamic allows non-null scalars for keys.
+  } else if (key.isNumber() || key.isBool()) {
+    return allow_non_string_keys ? (*d)[key] : (*d)[key.asString()];
+  }
+  // One cause might be oddness like p.optional(dynamic::array(...), ...);
+  throw DynamicParserLogicError(
+    "Unsupported key type ", key.typeName(), " of ", detail::toPseudoJson(key)
+  );
+}
+}  // anonymous namespace
+
+void DynamicParser::reportError(
+    const folly::dynamic* lookup_k,
+    const std::exception& ex) {
+  // If descendants of this item, or other keys on it, already reported an
+  // error, the error object would already exist.
+  auto& e = stack_.errors(allowNonStringKeyErrors_);
+
+  // Save the original, unparseable value of the item causing the error.
+  //
+  // value() can throw here, but if it does, it is due to programmer error,
+  // so we don't want to report it as a parse error anyway.
+  if (auto* e_val_ptr = e.get_ptr("value")) {
+    // Failing to access distinct keys on the same value can generate
+    // multiple errors, but the value should remain the same.
+    if (*e_val_ptr != value()) {
+      throw DynamicParserLogicError(
+        "Overwriting value: ", detail::toPseudoJson(*e_val_ptr), " with ",
+        detail::toPseudoJson(value()), " for error ", ex.what()
+      );
+    }
+  } else {
+    // The e["value"].isNull() trick cannot be used because value().type()
+    // *can* be folly::dynamic::Type::NULLT, so we must hash again.
+    e["value"] = value();
+  }
+
+  // Differentiate between "parsing value" and "looking up key" errors.
+  auto& e_msg = [&]() -> folly::dynamic& {
+    if (lookup_k == nullptr) {  // {object,array}Items, or post-key-lookup
+      return e["error"];
+    }
+    // Multiple key lookups can report errors on the same collection.
+    auto& key_errors = e["key_errors"];
+    if (key_errors.isNull()) {
+      // Treat arrays as integer-keyed objects.
+      key_errors = folly::dynamic::object();
+    }
+    return insertAtKey(&key_errors, allowNonStringKeyErrors_, *lookup_k);
+  }();
+  if (!e_msg.isNull()) {
+    throw DynamicParserLogicError(
+      "Overwriting error: ", detail::toPseudoJson(e_msg), " with: ",
+      ex.what()
+    );
+  }
+  e_msg = ex.what();
+
+  switch (onError_) {
+    case OnError::RECORD:
+      break;  // Continue parsing
+    case OnError::THROW:
+      stack_.throwErrors();  // Package releaseErrors() into an exception.
+      LOG(FATAL) << "Not reached";  // silence lint false positive
+    default:
+      LOG(FATAL) << "Bad onError_: " << static_cast<int>(onError_);
+  }
+}
+
+void DynamicParser::ParserStack::Pop::operator()() noexcept {
+  stackPtr_->key_ = key_;
+  stackPtr_->value_ = value_;
+  if (stackPtr_->unmaterializedSubErrorKeys_.empty()) {
+    // There should be the current error, and the root.
+    CHECK_GE(stackPtr_->subErrors_.size(), 2)
+      << "Internal bug: out of suberrors";
+    stackPtr_->subErrors_.pop_back();
+  } else {
+    // Errors were never materialized for this subtree, so errors_ only has
+    // ancestors of the item being processed.
+    stackPtr_->unmaterializedSubErrorKeys_.pop_back();
+    CHECK(!stackPtr_->subErrors_.empty()) << "Internal bug: out of suberrors";
+  }
+}
+
+folly::ScopeGuardImpl<DynamicParser::ParserStack::Pop>
+DynamicParser::ParserStack::push(
+    const folly::dynamic& k,
+    const folly::dynamic& v) noexcept {
+  // Save the previous state of the parser.
+  folly::ScopeGuardImpl<DynamicParser::ParserStack::Pop> guard(
+    DynamicParser::ParserStack::Pop(this)
+  );
+  key_ = &k;
+  value_ = &v;
+  // We create errors_ sub-objects lazily to keep the result small.
+  unmaterializedSubErrorKeys_.emplace_back(key_);
+  return guard;
+}
+
+// `noexcept` because if the materialization loop threw, we'd end up with
+// more suberrors than we started with.
+folly::dynamic& DynamicParser::ParserStack::errors(
+    bool allow_non_string_keys) noexcept {
+  // Materialize the lazy "key + parent's type" error objects we'll need.
+  CHECK(!subErrors_.empty()) << "Internal bug: out of suberrors";
+  for (const auto& suberror_key : unmaterializedSubErrorKeys_) {
+    auto& nested = (*subErrors_.back())["nested"];
+    if (nested.isNull()) {
+      nested = folly::dynamic::object();
+    }
+    // Find, or insert a dummy entry for the current key
+    auto& my_errors =
+      insertAtKey(&nested, allow_non_string_keys, *suberror_key);
+    if (my_errors.isNull()) {
+      my_errors = folly::dynamic::object();
+    }
+    subErrors_.emplace_back(&my_errors);
+  }
+  unmaterializedSubErrorKeys_.clear();
+  return *subErrors_.back();
+}
+
+folly::dynamic DynamicParser::ParserStack::releaseErrors() {
+  if (
+    key_ || unmaterializedSubErrorKeys_.size() != 0 || subErrors_.size() != 1
+  ) {
+    throw DynamicParserLogicError(
+      "Do not releaseErrors() while parsing: ", key_ != nullptr, " / ",
+      unmaterializedSubErrorKeys_.size(), " / ", subErrors_.size()
+    );
+  }
+  return releaseErrorsImpl();
+}
+
+void DynamicParser::ParserStack::throwErrors() {
+  throw DynamicParserParseError(releaseErrorsImpl());
+}
+
+folly::dynamic DynamicParser::ParserStack::releaseErrorsImpl() {
+  if (errors_.isNull()) {
+    throw DynamicParserLogicError("Do not releaseErrors() twice");
+  }
+  auto errors = std::move(errors_);
+  errors_ = nullptr;  // Prevent a second release.
+  value_ = nullptr;  // Break attempts to parse again.
+  return errors;
+}
+
+namespace detail {
+std::string toPseudoJson(const folly::dynamic& d) {
+  std::stringstream ss;
+  ss << d;
+  return ss.str();
+}
+}  // namespace detail
+
+}  // namespace folly

folly/experimental/DynamicParser.h

+/*
+ * Copyright 2016 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/*
+ *  Copyright (c) 2015, Facebook, Inc.
+ *  All rights reserved.
+ *
+ *  This source code is licensed under the BSD-style license found in the
+ *  LICENSE file in the root directory of this source tree. An additional grant
+ *  of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+#pragma once
+
+#include <folly/dynamic.h>
+#include <folly/ScopeGuard.h>
+
+namespace folly {
+
+/**
+ * DynamicParser provides a tiny DSL for easily, correctly, and losslessly
+ * parsing a folly::dynamic into any other representation.
+ *
+ * To make this concrete, this lets you take a JSON config that potentially
+ * contains user errors, and parse __all__ of its valid parts, while
+ * automatically and __reversibly__ recording any parts that cause errors:
+ *
+ *   {"my values": {
+ *     "an int": "THIS WILL BE RECORDED AS AN ERROR, BUT WE'LL PARSE THE REST",
+ *     "a double": 3.1415,
+ *     "keys & values": {
+ *       "the sky is blue": true,
+ *       "THIS WILL ALSO BE RECORDED AS AN ERROR": "cheese",
+ *       "2+2=5": false,
+ *     }
+ *   }}
+ *
+ * To parse this JSON, you need no exception handling, it is as easy as:
+ *
+ *   folly::dynamic d = ...;  // Input
+ *   int64_t integer;  // Three outputs
+ *   double real;
+ *   std::map<std::string, bool> enabled_widgets;
+ *   DynamicParser p(DynamicParser::OnError::RECORD, &d);
+ *   p.required("my values", [&]() {
+ *     p.optional("an int", [&](int64_t v) { integer = v; });
+ *     p.required("a double", [&](double v) { real = v; });
+ *     p.optional("keys & values", [&]() {
+ *       p.objectItems([&](std::string widget, bool enabled) {
+ *         enabled_widgets.emplace(widget, enabled);
+ *       });
+ *     });
+ *   });
+ *
+ * Your code in the lambdas can throw, and this will be reported just like
+ * missing key and type conversion errors, with precise context on what part
+ * of the folly::dynamic caused the error.  No need to throw:
+ *   std::runtime_error("Value X at key Y caused a flux capacitor overload")
+ * This will do:
+ *   std::runtime_error("Flux capacitor overload")
+ *
+ * == Keys and values are auto-converted to match your callback ==
+ *
+ * DynamicParser's optional(), required(), objectItems(), and
+ * arrayItems() automatically convert the current key and value to match the
+ * signature of the provided callback.  parser.key() and parser.value() can
+ * be used to access the same data without conversion.
+ *
+ * The following types are supported -- you should generally take arguments
+ * by-value, or by-const-reference for dynamics & strings you do not copy.
+ *
+ *   Key: folly::dynamic (no conversion), std::string, int64_t
+ *   Value: folly::dynamic (no conversion), int64_t, bool, double, std::string
+ *
+ * There are 21 supported callback signatures, of three kinds:
+ *
+ *   1: No arguments -- useful if you will just call more parser methods.
+ *
+ *   5: The value alone -- the common case for optional() and required().
+ *        [&](whatever_t value) {}
+ *
+ *   15: Both the key and the value converted according to the rules above:
+ *        [&](whatever_t key, whatever_t) {}
+ *
+ * NB: The key alone should be rarely needed, but these callback styles
+ *     provide it with no conversion overhead, and only minimal verbosity:
+ *       [&](const std::string& k, const folly::dynamic&) {}
+ *       [&]() { auto k = p.key().asString().toStdString(); }
+ *
+ * == How `releaseErrors()` can make your parse lossless ==
+ *
+ * If you write parsing code by hand, you usually end up with error-handling
+ * resembling that of OnError::THROW -- the first error you hit aborts the
+ * whole parse, and you report it.
+ *
+ * OnError::RECORD offers a more user-friendly alternative for "parse,
+ * serialize, re-parse" pipelines, akin to what web-forms do.  All
+ * exception-causing parts are losslessly recorded in a parallel
+ * folly::dynamic, available via releaseErrors() at the end of the parse.
+ *
+ * Suppose we fail to look up "key1" at the root, and hit a value error in
+ * "key2": {"subkey2": ...}.  The error report will have the form:
+ *
+ *   {"nested": {
+ *     "key_errors": {"key1": "explanatory message"},
+ *     "value": <whole input>,
+ *     "nested": { "key2": { "nested": {
+ *       "subkey2": {"value": <original value>, "error": "message"}
+ *     } } }
+ *   }}
+ *
+ * Errors in array items are handled just the same, but using integer keys.
+ *
+ * The advantage of this approach is that your parsing can throw wherever,
+ * and DynamicParser isolates it, allowing the good parts to parse.
+ *
+ * Put another way, this makes it easy to implement a transformation that
+ * splits a `folly::dynamic` into a "parsed" part (which might be your
+ * struct meant for runtime use), and a matching "errors" part.  As long as
+ * your successful parses are lossless, you can always reconstruct the
+ * original input from the parse output and the recorded "errors".
+ *
+ * == Limitations ==
+ *
+ *  - The input dynamic should be an object or array. wrapError() could be
+ *    exposed to allow parsing single scalars, but this would not be a
+ *    significant usability improvement over try-catch.
+ *
+ *  - Do NOT try to parse the same part of the input dynamic twice. You
+ *    might report multiple value errors, which is currently unsupported.
+ *
+ *  - optional() does not support defaulting. This is unavoidable, since
+ *    DynamicParser does not dictate how you record parsed data.  If your
+ *    parse writes into an output struct, then it ought to be initialized at
+ *    construction time.  If your output is initialized to default values,
+ *    then you need no "default" feature.  If it is not initialized, you are
+ *    in trouble anyway.  Suppose your optional() parse hits an error.  What
+ *    does your output contain?
+ *      - Uninitialized data :(
+ *      - You rely on an optional() feature to fall back to parsing some
+ *        default dynamic.  Sadly, the default hits a parse error.  Now what?
+ *    Since there is no good way to default, DynamicParser leaves it out.
+ *
+ * == Future: un-parsed items ==
+ *
+ * DynamicParser could support erroring on un-parsed items -- the parts of
+ * the folly::dynamic, which were never asked for.  Here is an ok design:
+ *
+ * (i) At the start of parsing any value, the user may call:
+ *   parser.recursivelyForbidUnparsed();
+ *   parser.recursivelyAllowUnparsed();
+ *   parser.locallyForbidUnparsed();
+ *   parser.locallyAllowUnparsed();
+ *
+ * (ii) At the end of the parse, any unparsed items are dumped to "errors".
+ * For example, failing to parse index 1 out of ["v1", "v2", "v3"] yields:
+ *   "nested": {1: {"unparsed": "v2"}}
+ * or perhaps more verbosely:
+ *   "nested": {1: {"error": "unparsed value", "value": "v2"}}
+ *
+ * By default, unparsed items are allowed. Calling a "forbid" function after
+ * some keys have already been parsed is allowed to fail (this permits a
+ * lazy implementation, which has minimal overhead when "forbid" is not
+ * requested).
+ *
+ * == Future: multiple value errors ==
+ *
+ * The present contract is that exactly one value error is reported per
+ * location in the input (multiple key lookup errors are, of course,
+ * supported).  If the need arises, multiple value errors could easily be
+ * supported by replacing the "error" string with an "errors" array.
+ */
+
+namespace detail {
+// Why do DynamicParser error messages use folly::dynamic pseudo-JSON?
+// Firstly, the input dynamic need not correspond to valid JSON.  Secondly,
+// wrapError() uses integer-keyed objects to report arrary-indexing errors.
+std::string toPseudoJson(const folly::dynamic& d);
+}  // namespace detail
+
+/**
+ * With DynamicParser::OnError::THROW, reports the first error.
+ * It is forbidden to call releaseErrors() if you catch this.
+ */
+struct DynamicParserParseError : public std::runtime_error {
+  explicit DynamicParserParseError(folly::dynamic error)
+    : std::runtime_error(folly::to<std::string>(
+        "DynamicParserParseError: ", detail::toPseudoJson(error)
+      )),
+      error_(std::move(error)) {}
+  /**
+   * Structured just like releaseErrors(), but with only 1 error inside:
+   *   {"nested": {"key1": {"nested": {"key2": {"error": "err", "value": 5}}}}}
+   * or:
+   *   {"nested": {"key1": {"key_errors": {"key3": "err"}, "value": 7}}}
+   */
+  const folly::dynamic& error() const { return error_; }
+private:
+  folly::dynamic error_;
+};
+
+/**
+ * When DynamicParser is used incorrectly, it will throw this exception
+ * instead of reporting an error via releaseErrors().  It is unsafe to call
+ * any parser methods after catching a LogicError.
+ */
+struct DynamicParserLogicError : public std::logic_error {
+  template <typename... Args>
+  explicit DynamicParserLogicError(Args&&... args)
+    : std::logic_error(folly::to<std::string>(std::forward<Args>(args)...)) {}
+};
+
+class DynamicParser {
+public:
+  enum class OnError {
+    // After parsing, releaseErrors() reports all parse errors.
+    // Throws DynamicParserLogicError on programmer errors.
+    RECORD,
+    // Throws DynamicParserParseError on the first parse error, or
+    // DynamicParserLogicError on programmer errors.
+    THROW,
+  };
+
+  // You MUST NOT destroy `d` before the parser.
+  DynamicParser(OnError on_error, const folly::dynamic* d)
+    : onError_(on_error), stack_(d) {}  // Always access input through stack_
+
+  /**
+   * Once you finished the entire parse, returns a structured description of
+   * all parse errors (see top-of-file docblock).  May ONLY be called once.
+   * May NOT be called if the parse threw any kind of exception.  Returns an
+   * empty object for successful OnError::THROW parsers.
+   */
+  folly::dynamic releaseErrors() { return stack_.releaseErrors(); }
+
+  /**
+   * Error-wraps fn(auto-converted key & value) if d[key] is set. The
+   * top-of-file docblock explains the auto-conversion.
+   */
+  template <typename Fn>
+  void optional(const folly::dynamic& key, Fn);
+
+  // Like optional(), but reports an error if d[key] does not exist.
+  template <typename Fn>
+  void required(const folly::dynamic& key, Fn);
+
+  /**
+   * Iterate over the current object's keys and values. Report each item's
+   * errors under its own key in a matching sub-object of "errors".
+   */
+  template <typename Fn>
+  void objectItems(Fn);
+
+  /**
+   * Like objectItems() -- arrays are treated identically to objects with
+   * integer keys from 0 to size() - 1.
+   */
+  template <typename Fn>
+  void arrayItems(Fn);
+
+  /**
+   * The key currently being parsed (integer if inside an array). Throws if
+   * called outside of a parser callback.
+   */
+  inline const folly::dynamic& key() const { return stack_.key(); }
+  /**
+   * The value currently being parsed (initially, the input dynamic).
+   * Throws if parsing nullptr, or parsing after releaseErrors().
+   */
+  inline const folly::dynamic& value() const { return stack_.value(); }
+
+  /**
+   * By default, DynamicParser's "nested" object coerces all keys to
+   * strings, whether from arrayItems() or from p.optional(some_int, ...),
+   * to allow errors be serialized to JSON.  If you are parsing non-JSON
+   * dynamic objects with non-string keys, this is problematic.  When set to
+   * true, "nested" objects will report integer keys for errors coming from
+   * inside arrays, or the original key type from inside values of objects.
+   */
+  DynamicParser& setAllowNonStringKeyErrors(bool b) {
+    allowNonStringKeyErrors_ = b;
+    return *this;
+  }
+
+private:
+  /**
+   * If `fn` throws an exception, wrapError() catches it and inserts an
+   * enriched description into stack_.errors_.  If lookup_key is non-null,
+   * reports a key lookup error in "key_errors", otherwise reportse a value
+   * error in "error".
+   *
+   * Not public because that would encourage users to report multiple errors
+   * per input part, which is currently unsupported.  It does not currently
+   * seem like normal user code should need this.
+   */
+  template <typename Fn>
+  void wrapError(const folly::dynamic* lookup_key, Fn);
+
+  void reportError(const folly::dynamic* lookup_k, const std::exception& ex);
+
+  template <typename Fn>
+  void parse(const folly::dynamic& key, const folly::dynamic& value, Fn fn);
+
+  // All of the above business logic obtains the part of the folly::dynamic
+  // it is examining (and the location for reporting errors) via this class,
+  // which lets it correctly handle nesting.
+  struct ParserStack {
+    struct Pop {
+      explicit Pop(ParserStack* sp)
+        : key_(sp->key_), value_(sp->value_), stackPtr_(sp) {}
+      void operator()() noexcept;  // ScopeGuard requires noexcept
+    private:
+      const folly::dynamic* key_;
+      const folly::dynamic* value_;
+      ParserStack* stackPtr_;
+    };
+
+    explicit ParserStack(const folly::dynamic* input)
+      : value_(input),
+        errors_(folly::dynamic::object()),
+        subErrors_({&errors_}) {}
+
+    // Not copiable or movable due to numerous internal pointers
+    ParserStack(const ParserStack&) = delete;
+    ParserStack& operator=(const ParserStack&) = delete;
+    ParserStack(ParserStack&&) = delete;
+    ParserStack& operator=(ParserStack&&) = delete;
+
+    // Lets user code nest parser calls by recording current key+value and
+    // returning an RAII guard to restore the old one.  `noexcept` since it
+    // is used unwrapped.
+    folly::ScopeGuardImpl<Pop> push(
+      const folly::dynamic& k, const folly::dynamic& v
+    ) noexcept;
+
+    // Throws DynamicParserLogicError if used outside of a parsing function.
+    inline const folly::dynamic& key() const;
+    // Throws DynamicParserLogicError if used after releaseErrors().
+    inline const folly::dynamic& value() const;
+
+    // Lazily creates new "nested" sub-objects in errors_.
+    folly::dynamic& errors(bool allow_non_string_keys) noexcept;
+
+    // The user invokes this at most once after the parse is done.
+    folly::dynamic releaseErrors();
+
+    // Invoked on error when using OnError::THROW.
+    void throwErrors();
+
+  private:
+    friend struct Pop;
+
+    folly::dynamic releaseErrorsImpl();  // for releaseErrors() & throwErrors()
+
+    // Null outside of a parsing function.
+    const folly::dynamic* key_{nullptr};
+    // Null on errors: when the input was nullptr, or after releaseErrors().
+    const folly::dynamic* value_;
+
+    // An object containing some of these keys:
+    //   "key_errors" -- {"key": "description of error looking up said key"}
+    //   "error" -- why did we fail to parse this value?
+    //   "value" -- a copy of the input causing the error, and
+    //   "nested" -- {"key" or integer for arrays: <another errors_ object>}
+    //
+    // "nested" will contain identically structured objects with keys (array
+    // indices) identifying the origin of the errors.  Of course, "input"
+    // would no longer refer to the whole input, but to a part.
+    folly::dynamic errors_;
+    // We only materialize errors_ sub-objects when needed. This stores keys
+    // for unmaterialized errors, from outermost to innermost.
+    std::vector<const folly::dynamic*> unmaterializedSubErrorKeys_;
+    // Materialized errors, from outermost to innermost
+    std::vector<folly::dynamic*> subErrors_;  // Point into errors_
+  };
+
+  OnError onError_;
+  ParserStack stack_;
+  bool allowNonStringKeyErrors_{false};  // See the setter's docblock.
+};
+
+}  // namespace folly
+
+#include <folly/experimental/DynamicParser-inl.h>

folly/experimental/test/DynamicParserTest.cpp

+/*
+ * Copyright 2016 Facebook, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/*
+ *  Copyright (c) 2015, Facebook, Inc.
+ *  All rights reserved.
+ *
+ *  This source code is licensed under the BSD-style license found in the
+ *  LICENSE file in the root directory of this source tree. An additional grant
+ *  of patent rights can be found in the PATENTS file in the same directory.
+ *
+ */
+#include <folly/Optional.h>
+#include <folly/experimental/DynamicParser.h>
+#include <folly/experimental/TestUtil.h>
+#include <gtest/gtest.h>
+
+using namespace folly;
+using dynamic = folly::dynamic;
+
+// NB Auto-conversions are exercised by all the tests, there's not a great
+// reason to test all of them explicitly, since any uncaught bugs will fail
+// at compile-time.
+
+// See setAllowNonStringKeyErrors() -- most of the tests below presume that
+// all keys in releaseErrors() are coerced to string.
+void checkMaybeCoercedKeys(bool coerce, dynamic good_k, dynamic missing_k) {
+  dynamic d = dynamic::object(good_k, 7);
+  DynamicParser p(DynamicParser::OnError::RECORD, &d);
+  p.setAllowNonStringKeyErrors(!coerce);
+  auto coerce_fn = [coerce](dynamic k) -> dynamic {
+    return coerce ? k.asString() : k;
+  };
+
+  // Key and value errors have different code paths, so exercise both.
+  p.required(missing_k, [&]() {});
+  p.required(good_k, [&]() { throw std::runtime_error("failsauce"); });
+  auto errors = p.releaseErrors();
+
+  auto parse_error = errors.at("nested").at(coerce_fn(good_k));
+  EXPECT_EQ(d.at(good_k), parse_error.at("value"));
+  EXPECT_PCRE_MATCH(".*failsauce.*", parse_error.at("error").getString());
+
+  auto key_error = errors.at("key_errors").at(coerce_fn(missing_k));
+  EXPECT_PCRE_MATCH(".*Couldn't find key .* in .*", key_error.getString());
+
+  EXPECT_EQ(dynamic(dynamic::object
+    ("nested", dynamic::object(coerce_fn(good_k), parse_error))
+    ("key_errors", dynamic::object(coerce_fn(missing_k), key_error))
+    ("value", d)
+  ), errors);
+}
+
+void checkCoercedAndUncoercedKeys(dynamic good_k, dynamic missing_k) {
+  checkMaybeCoercedKeys(true, good_k, missing_k);
+  checkMaybeCoercedKeys(false, good_k, missing_k);
+}
+
+TEST(TestDynamicParser, CoercedAndUncoercedKeys) {
+  // Check that both key errors and value errors are reported via
+  checkCoercedAndUncoercedKeys("a", "b");
+  checkCoercedAndUncoercedKeys(7, 5);
+  checkCoercedAndUncoercedKeys(0.7, 0.5);
+  checkCoercedAndUncoercedKeys(true, false);
+}
+
+TEST(TestDynamicParser, OnErrorThrowSuccess) {
+  auto d = dynamic::array(dynamic::object("int", 5));
+  DynamicParser p(DynamicParser::OnError::THROW, &d);
+  folly::Optional<int64_t> i;
+  p.required(0, [&]() { p.optional("int", [&](int64_t v) { i = v; }); });
+  // With THROW, releaseErrors() isn't useful -- it's either empty or throws.
+  EXPECT_EQ(dynamic(dynamic::object()), p.releaseErrors());
+  EXPECT_EQ((int64_t)5, i);
+}
+
+TEST(TestDynamicParser, OnErrorThrowError) {
+  auto d = dynamic::array(dynamic::object("int", "fail"));
+  DynamicParser p(DynamicParser::OnError::THROW, &d);
+  try {
+    // Force the exception to bubble up through a couple levels of nesting.
+    p.required(0, [&]() { p.optional("int", [&](int64_t) {}); });
+    FAIL() << "Should have thrown";
+  } catch (const DynamicParserParseError& ex) {
+    auto error = ex.error();
+    const auto& message =
+      error.at("nested").at("0").at("nested").at("int").at("error");
+    EXPECT_PCRE_MATCH(".* conversion to integral.*", message.getString());
+    EXPECT_PCRE_MATCH(
+      "DynamicParserParseError: .* conversion to integral.*", ex.what()
+    );
+    EXPECT_EQ(dynamic(dynamic::object
+      ("nested", dynamic::object
+        ("0", dynamic::object
+          ("nested", dynamic::object
+            ("int", dynamic::object
+              ("error", message)("value", "fail")))))), error);
+    EXPECT_THROW(p.releaseErrors(), DynamicParserLogicError)
+      << "THROW releases the first error eagerly, and throws";
+  }
+}
+
+// Errors & exceptions are best tested separately, but squeezing all the
+// features into one test is good for exercising nesting.
+TEST(TestDynamicParser, AllParserFeaturesSuccess) {
+  // Input
+  auto d = dynamic::array(
+    dynamic::object("a", 7)("b", 9)("c", 13.3),
+    5,
+    dynamic::array("x", "y", 1, "z"),
+    dynamic::object("int", 7)("false", 0)("true", true)("str", "s"),
+    dynamic::object("bools", dynamic::array(false, true, 0, 1))
+  );
+  // Outputs, in the same order as the inputs.
+  std::map<std::string, double> doubles;
+  folly::Optional<int64_t> outer_int;
+  std::vector<std::string> strings;
+  folly::Optional<int64_t> inner_int;
+  folly::Optional<bool> inner_false;
+  folly::Optional<bool> inner_true;
+  folly::Optional<std::string> inner_str;
+  std::vector<bool> bools;
+  // Parse and verify some invariants
+  DynamicParser p(DynamicParser::OnError::RECORD, &d);
+  EXPECT_EQ(d, p.value());
+  p.required(0, [&](const dynamic& v) {
+    EXPECT_EQ(0, p.key().getInt());
+    EXPECT_EQ(v, p.value());
+    p.objectItems([&](const std::string& k, double v) {
+      EXPECT_EQ(k, p.key().getString());
+      EXPECT_EQ(v, p.value().asDouble());
+      doubles.emplace(k, v);
+    });
+  });
+  p.required(1, [&](int64_t k, int64_t v) {
+    EXPECT_EQ(1, k);
+    EXPECT_EQ(1, p.key().getInt());
+    EXPECT_EQ(5, p.value().getInt());
+    outer_int = v;
+  });
+  p.optional(2, [&](const dynamic& v) {
+    EXPECT_EQ(2, p.key().getInt());
+    EXPECT_EQ(v, p.value());
+    p.arrayItems([&](int64_t k, const std::string& v) {
+      EXPECT_EQ(strings.size(), k);
+      EXPECT_EQ(k, p.key().getInt());
+      EXPECT_EQ(v, p.value().asString());
+      strings.emplace_back(v);
+    });
+  });
+  p.required(3, [&](const dynamic& v) {
+    EXPECT_EQ(3, p.key().getInt());
+    EXPECT_EQ(v, p.value());
+    p.optional("int", [&](const std::string& k, int64_t v) {
+      EXPECT_EQ("int", p.key().getString());
+      EXPECT_EQ(k, p.key().getString());
+      EXPECT_EQ(v, p.value().getInt());
+      inner_int = v;
+    });
+    p.required("false", [&](const std::string& k, bool v) {
+      EXPECT_EQ("false", p.key().getString());
+      EXPECT_EQ(k, p.key().getString());
+      EXPECT_EQ(v, p.value().asBool());
+      inner_false = v;
+    });
+    p.required("true", [&](const std::string& k, bool v) {
+      EXPECT_EQ("true", p.key().getString());
+      EXPECT_EQ(k, p.key().getString());
+      EXPECT_EQ(v, p.value().getBool());
+      inner_true = v;
+    });
+    p.required("str", [&](const std::string& k, const std::string& v) {
+      EXPECT_EQ("str", p.key().getString());
+      EXPECT_EQ(k, p.key().getString());
+      EXPECT_EQ(v, p.value().getString());
+      inner_str = v;
+    });
+    p.optional("not set", [&](bool) { FAIL() << "No key 'not set'"; });
+  });
+  p.required(4, [&](const dynamic& v) {
+    EXPECT_EQ(4, p.key().getInt());
+    EXPECT_EQ(v, p.value());
+    p.optional("bools", [&](const std::string& k, const dynamic& v) {
+      EXPECT_EQ(std::string("bools"), k);
+      EXPECT_EQ(k, p.key().getString());
+      EXPECT_EQ(v, p.value());
+      p.arrayItems([&](int64_t k, bool v) {
+        EXPECT_EQ(bools.size(), k);
+        EXPECT_EQ(k, p.key().getInt());
+        EXPECT_EQ(v, p.value().asBool());
+        bools.push_back(v);
+      });
+    });
+  });
+  p.optional(5, [&](int64_t) { FAIL() << "Index 5 does not exist"; });
+  // Confirm the parse
+  EXPECT_EQ(dynamic(dynamic::object()), p.releaseErrors());
+  EXPECT_EQ((decltype(doubles){{"a", 7.}, {"b", 9.}, {"c", 13.3}}), doubles);
+  EXPECT_EQ((int64_t)5, outer_int);
+  EXPECT_EQ((decltype(strings){"x", "y", "1", "z"}), strings);
+  EXPECT_EQ((int64_t)7, inner_int);
+  EXPECT_FALSE(inner_false.value());
+  EXPECT_TRUE(inner_true.value());
+  EXPECT_EQ(std::string("s"), inner_str);
+  EXPECT_EQ(std::string("s"), inner_str);
+  EXPECT_EQ((decltype(bools){false, true, false, true}), bools);
+}
+
+// We can hit multiple key lookup errors, but only one parse error.
+template <typename Fn>
+void checkXYKeyErrorsAndParseError(
+    const dynamic& d,
+    Fn fn,
+    std::string key_re,
+    std::string parse_re) {
+  DynamicParser p(DynamicParser::OnError::RECORD, &d);
+  fn(p);
+  auto errors = p.releaseErrors();
+  auto x_key_msg = errors.at("key_errors").at("x");
+  EXPECT_PCRE_MATCH(key_re, x_key_msg.getString());
+  auto y_key_msg = errors.at("key_errors").at("y");
+  EXPECT_PCRE_MATCH(key_re, y_key_msg.getString());
+  auto parse_msg = errors.at("error");
+  EXPECT_PCRE_MATCH(parse_re, parse_msg.getString());
+  EXPECT_EQ(dynamic(dynamic::object
+    ("key_errors", dynamic::object("x", x_key_msg)("y", y_key_msg))
+    ("error", parse_msg)
+    ("value", d)), errors);
+}
+
+// Exercise key errors for optional / required, and outer parse errors for
+// arrayItems / objectItems.
+TEST(TestDynamicParser, TestKeyAndParseErrors) {
+  checkXYKeyErrorsAndParseError(
+    dynamic::object(),
+    [&](DynamicParser& p) {
+      p.required("x", [&]() {});  // key
+      p.required("y", [&]() {});  // key
+      p.arrayItems([&]() {});  // parse
+    },
+    "Couldn't find key (x|y) .*",
+    "^TypeError: .*"
+  );
+  checkXYKeyErrorsAndParseError(
+    dynamic::array(),
+    [&](DynamicParser& p) {
+      p.optional("x", [&]() {});  // key
+      p.optional("y", [&]() {});  // key
+      p.objectItems([&]() {});  // parse
+    },
+    "^TypeError: .*",
+    "^TypeError: .*"
+  );
+}
+
+// TestKeyAndParseErrors covered required/optional key errors, so only parse
+// errors remain.
+TEST(TestDynamicParser, TestRequiredOptionalParseErrors) {
+  dynamic d = dynamic::object("x", dynamic::array())("y", "z")("z", 1);
+  DynamicParser p(DynamicParser::OnError::RECORD, &d);
+  p.required("x", [&](bool) {});
+  p.required("y", [&](int64_t) {});
+  p.required("z", [&](int64_t) { throw std::runtime_error("CUSTOM"); });
+  auto errors = p.releaseErrors();
+  auto get_expected_error_fn = [&](const dynamic& k, std::string pcre) {
+    auto error = errors.at("nested").at(k);
+    EXPECT_EQ(d.at(k), error.at("value"));
+    EXPECT_PCRE_MATCH(pcre, error.at("error").getString());
+    return dynamic::object("value", d.at(k))("error", error.at("error"));
+  };
+  EXPECT_EQ(dynamic(dynamic::object("nested", dynamic::object
+    ("x", get_expected_error_fn("x", "TypeError: .* but had type `array'"))
+    ("y", get_expected_error_fn("y", ".* Invalid leading character .*"))
+    ("z", get_expected_error_fn("z", "CUSTOM")))), errors);
+}
+
+template <typename Fn>
+void checkItemParseError(
+    // real_k can differ from err_k, which is typically coerced to string
+    dynamic d, Fn fn, dynamic real_k, dynamic err_k, std::string re) {
+  DynamicParser p(DynamicParser::OnError::RECORD, &d);
+  fn(p);
+  auto errors = p.releaseErrors();
+  auto error = errors.at("nested").at(err_k);
+  EXPECT_EQ(d.at(real_k), error.at("value"));
+  EXPECT_PCRE_MATCH(re, error.at("error").getString());
+  EXPECT_EQ(dynamic(dynamic::object("nested", dynamic::object(
+    err_k, dynamic::object("value", d.at(real_k))("error", error.at("error"))
+  ))), errors);
+}
+
+// TestKeyAndParseErrors covered outer parse errors for {object,array}Items,
+// which are the only high-level API cases uncovered by
+// TestKeyAndParseErrors and TestRequiredOptionalParseErrors.
+TEST(TestDynamicParser, TestItemParseErrors) {
+  checkItemParseError(
+    dynamic::object("string", dynamic::array("not", "actually")),
+    [&](DynamicParser& p) {
+      p.objectItems([&](const std::string&, const std::string&) {});
+    },
+    "string", "string",
+    "TypeError: .* but had type `array'"
+  );
+  checkItemParseError(
+    dynamic::array("this is not a bool"),
+    [&](DynamicParser& p) { p.arrayItems([&](int64_t, bool) {}); },
+    0, "0",
+    ".* Non-whitespace: .*"
+  );
+}
+
+// The goal is to exercise the sub-error materialization logic pretty well
+TEST(TestDynamicParser, TestErrorNesting) {
+  dynamic d = dynamic::object
+    ("x", dynamic::array(
+      dynamic::object("y", dynamic::object("z", "non-object"))
+    ))
+    ("k", false);
+  DynamicParser p(DynamicParser::OnError::RECORD, &d);
+  // Start with a couple of successful nests, building up unmaterialized
+  // error objects.
+  p.required("x", [&]() {
+    p.arrayItems([&]() {
+      p.optional("y", [&]() {
+        // First, a key error
+        p.required("not a key", []() {});
+        // Nest again more to test partially materialized errors.
+        p.objectItems([&]() { p.optional("akey", []() {}); });
+        throw std::runtime_error("custom parse error");
+      });
+      // Key error inside fully materialized errors
+      p.required("also not a key", []() {});
+      throw std::runtime_error("another parse error");
+    });
+  });
+  p.required("non-key", []() {});  // Top-level key error
+  p.optional("k", [&](int64_t, bool) {});  // Non-int key for good measure
+  auto errors = p.releaseErrors();
+
+  auto& base = errors.at("nested").at("x").at("nested").at("0");
+  auto inner_key_err =
+    base.at("nested").at("y").at("key_errors").at("not a key");
+  auto innermost_key_err =
+    base.at("nested").at("y").at("nested").at("z").at("key_errors").at("akey");
+  auto outer_key_err = base.at("key_errors").at("also not a key");
+  auto root_key_err = errors.at("key_errors").at("non-key");
+  auto k_parse_err = errors.at("nested").at("k").at("error");
+
+  EXPECT_EQ(dynamic(dynamic::object
+    ("nested", dynamic::object
+        ("x", dynamic::object("nested", dynamic::object("0", dynamic::object
+          ("nested", dynamic::object("y", dynamic::object
+            ("nested", dynamic::object("z", dynamic::object
+              ("key_errors", dynamic::object("akey", innermost_key_err))
+              ("value", "non-object")
+            ))
+            ("key_errors", dynamic::object("not a key", inner_key_err))
+            ("error", "custom parse error")
+            ("value", dynamic::object("z", "non-object"))
+          ))
+          ("key_errors", dynamic::object("also not a key", outer_key_err))
+          ("error", "another parse error")
+          ("value", dynamic::object("y", dynamic::object("z", "non-object")))
+        )))
+        ("k", dynamic::object("error", k_parse_err)("value", false)))
+    ("key_errors", dynamic::object("non-key", root_key_err))
+    ("value", d)
+  ), errors);
+}
+
+TEST(TestDynamicParser, TestRecordThrowOnDoubleParseErrors) {
+  dynamic d = nullptr;
+  DynamicParser p(DynamicParser::OnError::RECORD, &d);
+  p.arrayItems([&]() {});
+  try {
+    p.objectItems([&]() {});
+    FAIL() << "Should throw on double-parsing a value with an error";
+  } catch (const DynamicParserLogicError& ex) {
+    EXPECT_PCRE_MATCH(".*Overwriting error: TypeError: .*", ex.what());
+  }
+}
+
+TEST(TestDynamicParser, TestRecordThrowOnChangingValue) {
+  dynamic d = nullptr;
+  DynamicParser p(DynamicParser::OnError::RECORD, &d);
+  p.required("x", [&]() {});  // Key error sets "value"
+  d = 5;
+  try {
+    p.objectItems([&]() {});  // Will detect the changed value
+    FAIL() << "Should throw on second error with a changing value";
+  } catch (const DynamicParserLogicError& ex) {
+    EXPECT_PCRE_MATCH(
+      // Accept 0 or null since folly used to mis-print null as 0.
+      ".*Overwriting value: (0|null) with 5 for error TypeError: .*",
+      ex.what()
+    );
+  }
+}
+
+TEST(TestDynamicParser, TestThrowOnReleaseWhileParsing) {
+  auto d = dynamic::array(1);
+  DynamicParser p(DynamicParser::OnError::RECORD, &d);
+  EXPECT_THROW(
+    p.arrayItems([&]() { p.releaseErrors(); }),
+    DynamicParserLogicError
+  );
+}
+
+TEST(TestDynamicParser, TestThrowOnReleaseTwice) {
+  dynamic d = nullptr;
+  DynamicParser p(DynamicParser::OnError::RECORD, &d);
+  p.releaseErrors();
+  EXPECT_THROW(p.releaseErrors(), DynamicParserLogicError);
+}
+
+TEST(TestDynamicParser, TestThrowOnNullValue) {
+  dynamic d = nullptr;
+  DynamicParser p(DynamicParser::OnError::RECORD, &d);
+  p.releaseErrors();
+  EXPECT_THROW(p.value(), DynamicParserLogicError);
+}
+
+TEST(TestDynamicParser, TestThrowOnKeyOutsideCallback) {
+  dynamic d = nullptr;
+  DynamicParser p(DynamicParser::OnError::RECORD, &d);
+  EXPECT_THROW(p.key(), DynamicParserLogicError);
+}