ByteEvent (socket timestamp) foundation

Summary:
Adding support for write and socket timestamps by introducing `ByteEvent` that can be delivered to observers.
`AsyncTransport::WriteFlags` has long had timestamping related flags, such as `TIMESTAMP_TX`, but the code required to act on these flags only existed in proxygen. This diff generalizes the approach so that it works for other use cases of `AsyncSocket`.

This diff is long, but much of it is unit tests designed to prevent regressions given the trickiness of socket timestamps and `ByteEvent`.

**Each `ByteEvent` contains:**
- Type (WRITE, SCHED, TX, ACK)
- Byte stream offset that the timestamp is for (relative to the raw byte stream, which means after SSL in the case of AsyncSSLSocket)
- `steady_clock` timestamp recorded by AsyncSocket when generating the `ByteEvent`
- For SCHED, TX, and ACK events, if available, hardware and software (kernel) timestamps

**How `ByteEvent` are used:**
- Support is enabled when an observer is attached with the `byteEvents` config flag set. If the socket does not support timestamps, the observer is notified through the `byteEventsUnavailable` callback. Otherwise, `byteEventsEnabled` is called
- When the application writes to a socket with `ByteEvent` support enabled and a relevant `WriteFlag`, SCHED/TX/ACK `ByteEvent` are requested from the kernel, and WRITE `ByteEvent` are generated by the socket for the *last byte* in the write.
  - If the entire write buffer cannot be written at once, then additional `ByteEvent` will also be generated for the last byte in each write.
  - This means that if the application wants to timestamp a specific byte, it must break up the write buffer before handing it to `AsyncSocket` such that the byte to timestamp is the last byte in the write buffer.
- When socket timestamps arrive from the kernel via the socket error queue, they are transformed into `ByteEvent` and passed to observers

**Caveats:**
1. Socket timestamps received from the kernel contain the byte's offset in the write stream. This counter is a `uint32_t`, and thus rolls over every ~4GB. When transforming raw timestamp into `ByteEvent`, we correct for this and transform the raw offset into an offset relative to the raw byte offset stored by `AsyncSocket` (returned via `getRawBytesWritten()`).
2. At the moment, a read callback must be installed to receive socket timestamps due to epoll's behavior. I will correct this with a patch to epoll, see https://github.com/libevent/libevent/issues/1038#issuecomment-703315425 for details
3. If a msghdr's ancillary data contains a timestamping flag (such as `SOF_TIMESTAMPING_TX_SOFTWARE`), the kernel will enqueue a socket error message containing the byte offset of the write ( `SO_EE_ORIGIN_TIMESTAMPING`) even if timestamping has not been enabled by an associated call to `setsockopt`. This creates a problem:
    1. If an application was to use a timestamp `WriteFlags` such as `TIMESTAMP_TX` without enabling timestamping, and if `AsyncSocket` transformed such `WriteFlags` to ancillary data by default, it could create a situation where epoll continues to return `EV_READ` (due to items in the socket error queue), but `AsyncSocket` would not fetch anything from the socket error queue.
    2. To prevent this scenario, `WriteFlags` related to timestamping are not translated into msghdr ancillary data unless timestamping is enabled. This required adding a boolean to `getAncillaryData` and `getAncillaryDataSize`.

Differential Revision: D24094832

fbshipit-source-id: e3bec730ddd1fc1696023d8c982ae02ab9b5fb7d

folly/io/async/AsyncSSLSocket.cpp

@@ -1861,6 +1861,18 @@ int AsyncSSLSocket::sslVerifyCallback(
   return 1;
 }
 
+void AsyncSSLSocket::enableByteEvents() {
+  if (getSSLVersion() == SSL3_VERSION || getSSLVersion() == TLS1_VERSION) {
+    // Socket timestamping can cause us to split up TLS records in a way that
+    // breaks some old Android (<= 3.0) clients.
+    return failByteEvents(AsyncSocketException(
+        AsyncSocketException::NOT_SUPPORTED,
+        withAddr("failed to enable byte events: "
+                 "not supported for SSLv3 or TLSv1")));
+  }
+  AsyncSocket::enableByteEvents();
+}
+
 void AsyncSSLSocket::enableClientHelloParsing() {
   parseClientHello_ = true;
   clientHelloInfo_ = std::make_unique<ssl::ClientHelloInfo>();

folly/io/async/AsyncSSLSocket.h

@@ -382,6 +382,18 @@ class AsyncSSLSocket : public AsyncSocket {
   void setEorTracking(bool track) override;
   size_t getRawBytesWritten() const override;
   size_t getRawBytesReceived() const override;
+
+  // End of methods inherited from AsyncTransport
+
+  /**
+   * Enable ByteEvents for this socket.
+   *
+   * ByteEvents cannot be enabled if TLS 1.0 or earlier is in use, as these
+   * client implementations often have trouble handling cases where a TLS
+   * record is split across multiple packets.
+   */
+  void enableByteEvents() override;
+
   void enableClientHelloParsing();
 
   /**

folly/io/async/AsyncSocket.cpp

@@ -40,6 +40,8 @@
 #include <folly/portability/Unistd.h>
 
 #if defined(__linux__)
+#include <linux/if_packet.h>
+#include <linux/net_tstamp.h>
 #include <linux/sockios.h>
 #include <sys/ioctl.h>
 #endif
@@ -76,6 +78,43 @@ static AsyncSocketException const& getSocketShutdownForWritesEx() {
   return ex;
 }
 
+namespace {
+#ifdef FOLLY_HAVE_SO_TIMESTAMPING
+const sock_extended_err* FOLLY_NULLABLE
+cmsgToSockExtendedErr(const cmsghdr& cmsg) {
+  if ((cmsg.cmsg_level == SOL_IP && cmsg.cmsg_type == IP_RECVERR) ||
+      (cmsg.cmsg_level == SOL_IPV6 && cmsg.cmsg_type == IPV6_RECVERR) ||
+      (cmsg.cmsg_level == SOL_PACKET &&
+       cmsg.cmsg_type == PACKET_TX_TIMESTAMP)) {
+    return reinterpret_cast<const sock_extended_err*>(CMSG_DATA(&cmsg));
+  }
+  (void)cmsg;
+  return nullptr;
+}
+
+const sock_extended_err* FOLLY_NULLABLE
+cmsgToSockExtendedErrTimestamping(const cmsghdr& cmsg) {
+  const auto serr = cmsgToSockExtendedErr(cmsg);
+  if (serr && serr->ee_errno == ENOMSG &&
+      serr->ee_origin == SO_EE_ORIGIN_TIMESTAMPING) {
+    return serr;
+  }
+  (void)cmsg;
+  return nullptr;
+}
+
+const scm_timestamping* FOLLY_NULLABLE
+cmsgToScmTimestamping(const cmsghdr& cmsg) {
+  if (cmsg.cmsg_level == SOL_SOCKET && cmsg.cmsg_type == SCM_TIMESTAMPING) {
+    return reinterpret_cast<const struct scm_timestamping*>(CMSG_DATA(&cmsg));
+  }
+  (void)cmsg;
+  return nullptr;
+}
+
+#endif // FOLLY_HAVE_SO_TIMESTAMPING
+} // namespace
+
 // TODO: It might help performance to provide a version of BytesWriteRequest
 // that users could derive from, so we can avoid the extra allocation for each
 // call to write()/writev().
@@ -275,6 +314,183 @@ int AsyncSocket::SendMsgParamsCallback::getDefaultFlags(
   return msg_flags;
 }
 
+void AsyncSocket::SendMsgParamsCallback::getAncillaryData(
+    folly::WriteFlags flags,
+    void* data,
+    const bool byteEventsEnabled) noexcept {
+  auto ancillaryDataSize = getAncillaryDataSize(flags, byteEventsEnabled);
+  if (!ancillaryDataSize) {
+    return;
+  }
+#ifdef FOLLY_HAVE_MSG_ERRQUEUE
+  CHECK_NOTNULL(data);
+  // this function only handles ancillary data for timestamping
+  //
+  // if getAncillaryDataSize() is overridden and returning a size different
+  // than what we expect, then this function needs to be overridden too, in
+  // order to avoid conflict with how cmsg / msg are written
+  CHECK_EQ(CMSG_LEN(sizeof(uint32_t)), ancillaryDataSize);
+
+  uint32_t sofFlags = 0;
+  if (byteEventsEnabled && isSet(flags, WriteFlags::TIMESTAMP_TX)) {
+    sofFlags = sofFlags | SOF_TIMESTAMPING_TX_SOFTWARE;
+  }
+  if (byteEventsEnabled && isSet(flags, WriteFlags::TIMESTAMP_ACK)) {
+    sofFlags = sofFlags | SOF_TIMESTAMPING_TX_ACK;
+  }
+  if (byteEventsEnabled && isSet(flags, WriteFlags::TIMESTAMP_SCHED)) {
+    sofFlags = sofFlags | SOF_TIMESTAMPING_TX_SCHED;
+  }
+
+  msghdr msg;
+  msg.msg_control = data;
+  msg.msg_controllen = ancillaryDataSize;
+  struct cmsghdr* cmsg = CMSG_FIRSTHDR(&msg);
+  CHECK_NOTNULL(cmsg);
+  cmsg->cmsg_level = SOL_SOCKET;
+  cmsg->cmsg_type = SO_TIMESTAMPING;
+  cmsg->cmsg_len = CMSG_LEN(sizeof(uint32_t));
+  memcpy(CMSG_DATA(cmsg), &sofFlags, sizeof(sofFlags));
+#endif
+  return;
+}
+
+uint32_t AsyncSocket::SendMsgParamsCallback::getAncillaryDataSize(
+    folly::WriteFlags flags, const bool byteEventsEnabled) noexcept {
+#ifdef FOLLY_HAVE_MSG_ERRQUEUE
+  if (WriteFlags::NONE != (flags & kWriteFlagsForTimestamping) &&
+      byteEventsEnabled) {
+    return CMSG_LEN(sizeof(uint32_t));
+  }
+#endif
+  return 0;
+}
+
+folly::Optional<AsyncSocket::ByteEvent>
+AsyncSocket::ByteEventHelper::processCmsg(
+    const cmsghdr& cmsg, const size_t rawBytesWritten) {
+#ifdef FOLLY_HAVE_SO_TIMESTAMPING
+  if (!byteEventsEnabled || maybeEx.has_value()) {
+    return folly::none;
+  }
+  if (!maybeTsState_.has_value()) {
+    maybeTsState_ = TimestampState();
+  }
+  auto& state = maybeTsState_.value();
+  if (auto serrTs = cmsgToSockExtendedErrTimestamping(cmsg)) {
+    if (state.serrReceived) {
+      // already have this part of the message pending
+      throw Exception("already have serr event");
+    }
+    state.serrReceived = true;
+    state.typeRaw = serrTs->ee_info;
+    state.byteOffsetKernel = serrTs->ee_data;
+  } else if (auto scmTs = cmsgToScmTimestamping(cmsg)) {
+    if (state.scmTsReceived) {
+      throw Exception("already have scmTs event");
+    }
+    state.scmTsReceived = true;
+
+    auto timespecToDuration =
+        [](const timespec& ts) -> folly::Optional<std::chrono::nanoseconds> {
+      std::chrono::nanoseconds duration = std::chrono::seconds(ts.tv_sec) +
+          std::chrono::nanoseconds(ts.tv_nsec);
+      if (duration == duration.zero()) {
+        return folly::none;
+      }
+      return duration;
+    };
+    // ts[0] -> software timestamp
+    // ts[1] -> hardware timestamp transformed to userspace time (deprecated)
+    // ts[2] -> hardware timestamp
+    state.maybeSoftwareTs = timespecToDuration(scmTs->ts[0]);
+    state.maybeHardwareTs = timespecToDuration(scmTs->ts[2]);
+  }
+
+  // if we have both components needed for a complete timestamp, build it
+  if (state.serrReceived && state.scmTsReceived) {
+    // cleanup state so that we're ready for next timestamp
+    TimestampState completeState = state;
+    maybeTsState_ = folly::none;
+
+    // map the type
+    folly::Optional<ByteEvent::Type> tsType;
+    switch (completeState.typeRaw) {
+      case SCM_TSTAMP_SND: {
+        tsType = ByteEvent::Type::TX;
+        break;
+      }
+      case SCM_TSTAMP_ACK: {
+        tsType = ByteEvent::Type::ACK;
+        break;
+      }
+      case SCM_TSTAMP_SCHED: {
+        tsType = ByteEvent::Type::SCHED;
+        break;
+      }
+      default:
+        break; // unknown, maybe something new
+    }
+    if (!tsType) {
+      // it's a timestamp, but not one that we're set up to handle
+      // we've cleared our state, loop back around
+      return folly::none;
+    }
+
+    // Calculate the byte offset.
+    //
+    // See documentation for SOF_TIMESTAMPING_OPT_ID for details.
+    //
+    // In summary, two things we have to consider:
+    //
+    //   (1) The byte stream offset is relative:
+    //       Socket timestamps include the byte stream offset for which the
+    //       timestamp applies. There may have been bytes transferred before the
+    //       fd was controlled by AsyncSocket. As a result, we don't know the
+    //       socket byte stream offset when we enable timestamping.
+    //
+    //       To get around this, we set SOF_TIMESTAMPING_OPT_ID when we enable
+    //       timestamping via setsockopt. This flag causes the kernel to reset
+    //       the offset it uses for timestamps to 0. This allows us to determine
+    //       an offset relative to the number of bytes that had been written to
+    //       the socket since timestamps were enabled.
+    //
+    //       Note that offsets begin at zero; if only a single byte is written
+    //       after timestamping is enabled, the offset included in the kernel
+    //       cmsg will be 0.
+    //
+    //   (2) The byte stream offset is a uint32_t:
+    //       Because the kernel uses a uint32_t to store and communicate the
+    //       byte stream offset, the offset will wrap every ~4GB. When we get a
+    //       timestamp, we need to figure out which byte it is for. We assume
+    //       that there will never be more than ~4GB of bytes sent between us
+    //       requesting timestamping for a byte and receiving the timestamp;
+    //       this is a realistic assumption given CWND and TCP buffer sizes. We
+    //       then calculate assuming that the counter has not wrapped since we
+    //       sent the byte that we are getting the timestamp for. If the counter
+    //       has wrapped, we detect it, and go back one position.
+    const uint64_t bytesPerOffsetWrap =
+        static_cast<uint64_t>(std::numeric_limits<uint32_t>::max()) + 1;
+    size_t byteOffset = rawBytesWritten -
+        (rawBytesWritten % bytesPerOffsetWrap) +
+        completeState.byteOffsetKernel + rawBytesWrittenWhenByteEventsEnabled;
+    if (byteOffset > rawBytesWritten) {
+      // kernel's uint32_t var wrapped around; go back one wrap
+      CHECK_GE(byteOffset, bytesPerOffsetWrap);
+      byteOffset = byteOffset - bytesPerOffsetWrap;
+    }
+
+    ByteEvent event = {};
+    event.type = tsType.value();
+    event.offset = byteOffset;
+    event.maybeSoftwareTs = state.maybeSoftwareTs;
+    event.maybeHardwareTs = state.maybeHardwareTs;
+    return event;
+  }
+#endif // FOLLY_HAVE_SO_TIMESTAMPING
+  return folly::none;
+}
+
 namespace {
 AsyncSocket::SendMsgParamsCallback defaultSendMsgParamsCallback;
 
@@ -362,6 +578,7 @@ AsyncSocket::AsyncSocket(AsyncSocket* oldAsyncSocket)
           oldAsyncSocket->getZeroCopyBufId()) {
   appBytesWritten_ = oldAsyncSocket->appBytesWritten_;
   rawBytesWritten_ = oldAsyncSocket->rawBytesWritten_;
+  byteEventHelper_ = std::move(oldAsyncSocket->byteEventHelper_);
   preReceivedData_ = std::move(oldAsyncSocket->preReceivedData_);
 
   // inform lifecycle observers to give them an opportunity to unsubscribe from
@@ -1076,6 +1293,118 @@ void AsyncSocket::processZeroCopyMsg(const cmsghdr& cmsg) {
 #endif
 }
 
+void AsyncSocket::enableByteEvents() {
+  if (!byteEventHelper_) {
+    byteEventHelper_ = std::make_unique<ByteEventHelper>();
+  }
+
+  if (byteEventHelper_->byteEventsEnabled ||
+      byteEventHelper_->maybeEx.has_value()) {
+    return;
+  }
+
+  try {
+#ifdef FOLLY_HAVE_SO_TIMESTAMPING
+    // make sure we have a connected IP socket that supports error queues
+    // (Unix sockets do not support error queues)
+    if (NetworkSocket() == fd_ || !good()) {
+      throw AsyncSocketException(
+          AsyncSocketException::INVALID_STATE,
+          withAddr("failed to enable byte events: "
+                   "socket is not open or not in a good state"));
+    }
+    folly::SocketAddress addr = {};
+    try {
+      // explicitly fetch local address (instead of using cache)
+      // to ensure socket is currently healthy
+      addr.setFromLocalAddress(fd_);
+    } catch (const std::system_error&) {
+      throw AsyncSocketException(
+          AsyncSocketException::INVALID_STATE,
+          withAddr("failed to enable byte events: "
+                   "socket is not open or not in a good state"));
+    }
+    const auto family = addr.getFamily();
+    if (family != AF_INET && family != AF_INET6) {
+      throw AsyncSocketException(
+          AsyncSocketException::NOT_SUPPORTED,
+          withAddr("failed to enable byte events: socket type not supported"));
+    }
+
+    // check if timestamping is already enabled on the socket by another source
+    {
+      uint32_t flags = 0;
+      socklen_t len = sizeof(flags);
+      const auto ret =
+          getSockOptVirtual(SOL_SOCKET, SO_TIMESTAMPING, &flags, &len);
+      int getSockOptErrno = errno;
+      if (0 != ret) {
+        throw AsyncSocketException(
+            AsyncSocketException::INTERNAL_ERROR,
+            withAddr("failed to enable byte events: "
+                     "timestamps may not be supported for this socket type "
+                     "or socket be closed"),
+            getSockOptErrno);
+      }
+      if (0 != flags) {
+        throw AsyncSocketException(
+            AsyncSocketException::INTERNAL_ERROR,
+            withAddr("failed to enable byte events: "
+                     "timestamps may have already been enabled"),
+            getSockOptErrno);
+      }
+    }
+
+    // enable control messages for software and hardware timestamps
+    // WriteFlags will determine which messages are generated
+    //
+    // SOF_TIMESTAMPING_OPT_ID: see discussion in ByteEventHelper::processCmsg
+    // SOF_TIMESTAMPING_OPT_TSONLY: only get timestamps, not original packet
+    // SOF_TIMESTAMPING_SOFTWARE: get software timestamps if generated
+    // SOF_TIMESTAMPING_RAW_HARDWARE: get hardware timestamps if generated
+    // SOF_TIMESTAMPING_OPT_TX_SWHW: get both sw + hw timestamps if generated
+    // SOF_TIMESTAMPING_OPT_MULTIMSG: older version SOF_TIMESTAMPING_OPT_TX_SWHW
+    const uint32_t flags =
+        (SOF_TIMESTAMPING_OPT_ID | SOF_TIMESTAMPING_OPT_TSONLY |
+         SOF_TIMESTAMPING_SOFTWARE | SOF_TIMESTAMPING_RAW_HARDWARE
+#if defined(SOF_TIMESTAMPING_OPT_TX_SWHW)
+         | SOF_TIMESTAMPING_OPT_TX_SWHW
+#elif defined(SOF_TIMESTAMPING_OPT_MULTIMSG)
+         | SOF_TIMESTAMPING_OPT_MULTIMSG
+#endif
+        );
+    socklen_t len = sizeof(flags);
+    const auto ret =
+        setSockOptVirtual(SOL_SOCKET, SO_TIMESTAMPING, &flags, len);
+    int setSockOptErrno = errno;
+    if (ret == 0) {
+      byteEventHelper_->byteEventsEnabled = true;
+      byteEventHelper_->rawBytesWrittenWhenByteEventsEnabled =
+          getRawBytesWritten();
+      for (const auto& observer : lifecycleObservers_) {
+        if (observer->getConfig().byteEvents) {
+          observer->byteEventsEnabled(this);
+        }
+      }
+      return;
+    }
+
+    // failed
+    throw AsyncSocketException(
+        AsyncSocketException::INTERNAL_ERROR,
+        withAddr("failed to enable byte events: setsockopt failed"),
+        setSockOptErrno);
+#endif // FOLLY_HAVE_SO_TIMESTAMPING
+    // unsupported by platform
+    throw AsyncSocketException(
+        AsyncSocketException::NOT_SUPPORTED,
+        withAddr("failed to enable byte events: platform not supported"));
+
+  } catch (const AsyncSocketException& ex) {
+    failByteEvents(ex);
+  }
+}
+
 void AsyncSocket::write(
     WriteCallback* callback, const void* buf, size_t bytes, WriteFlags flags) {
   iovec op;
@@ -2011,9 +2340,11 @@ size_t AsyncSocket::handleErrMessages() noexcept {
   // supporting per-socket error queues.
   VLOG(5) << "AsyncSocket::handleErrMessages() this=" << this << ", fd=" << fd_
           << ", state=" << state_;
-  if (errMessageCallback_ == nullptr && idZeroCopyBufPtrMap_.empty()) {
+  if (errMessageCallback_ == nullptr && idZeroCopyBufPtrMap_.empty() &&
+      (!byteEventHelper_ || !byteEventHelper_->byteEventsEnabled)) {
     VLOG(7) << "AsyncSocket::handleErrMessages(): "
-            << "no callback installed - exiting.";
+            << "no err message callback installed and "
+            << "ByteEvents not enabled - exiting.";
     return 0;
   }
 
@@ -2061,11 +2392,56 @@ size_t AsyncSocket::handleErrMessages() noexcept {
       ++num;
       if (isZeroCopyMsg(*cmsg)) {
         processZeroCopyMsg(*cmsg);
-      } else {
-        if (errMessageCallback_) {
-          errMessageCallback_->errMessage(*cmsg);
+        continue;
+      }
+
+      // try to process it as a ByteEvent and forward to observers
+      //
+      // observers cannot throw and thus we expect only exceptions from
+      // ByteEventHelper, but we guard against other cases for safety
+      if (byteEventHelper_) {
+        try {
+          if (const auto maybeByteEvent =
+                  byteEventHelper_->processCmsg(*cmsg, getRawBytesWritten())) {
+            const auto& byteEvent = maybeByteEvent.value();
+            for (const auto& observer : lifecycleObservers_) {
+              if (observer->getConfig().byteEvents) {
+                observer->byteEvent(this, byteEvent);
+              }
+            }
+          }
+        } catch (const ByteEventHelper::Exception& behEx) {
+          // rewrap the ByteEventHelper::Exception with extra information
+          AsyncSocketException ex(
+              AsyncSocketException::INTERNAL_ERROR,
+              withAddr(
+                  string("AsyncSocket::handleErrMessages(), "
+                         "internal exception during ByteEvent processing: ") +
+                  behEx.what()));
+          failByteEvents(ex);
+        } catch (const std::exception& ex) {
+          AsyncSocketException tex(
+              AsyncSocketException::UNKNOWN,
+              string("AsyncSocket::handleErrMessages(), "
+                     "unhandled exception during ByteEvent processing, "
+                     "threw exception: ") +
+                  ex.what());
+          failByteEvents(tex);
+        } catch (...) {
+          AsyncSocketException tex(
+              AsyncSocketException::UNKNOWN,
+              string("AsyncSocket::handleErrMessages(), "
+                     "unhandled exception during ByteEvent processing, "
+                     "threw non-exception type"));
+          failByteEvents(tex);
         }
       }
+
+      // even if it is a timestamp, hand it off to the errMessageCallback,
+      // the application may want it as well.
+      if (errMessageCallback_) {
+        errMessageCallback_->errMessage(*cmsg);
+      }
     }
   }
   return num;
@@ -2092,6 +2468,16 @@ void AsyncSocket::addLifecycleObserver(
   }
   lifecycleObservers_.push_back(observer);
   observer->observerAttach(this);
+  if (observer->getConfig().byteEvents) {
+    if (byteEventHelper_ && byteEventHelper_->maybeEx.has_value()) {
+      observer->byteEventsUnavailable(this, *byteEventHelper_->maybeEx);
+    } else if (byteEventHelper_ && byteEventHelper_->byteEventsEnabled) {
+      observer->byteEventsEnabled(this);
+    } else if (state_ == StateEnum::ESTABLISHED) {
+      enableByteEvents(); // try to enable now
+    }
+    // do nothing right now; wait until we're connected
+  }
 }
 
 bool AsyncSocket::removeLifecycleObserver(
@@ -2547,6 +2933,10 @@ ssize_t AsyncSocket::tfoSendMsg(
 
 AsyncSocket::WriteResult AsyncSocket::sendSocketMessage(
     const iovec* vec, size_t count, WriteFlags flags) {
+  const bool byteEventsEnabled =
+      (byteEventHelper_ && byteEventHelper_->byteEventsEnabled &&
+       !byteEventHelper_->maybeEx.has_value());
+
   struct msghdr msg = {};
   msg.msg_name = nullptr;
   msg.msg_namelen = 0;
@@ -2554,18 +2944,34 @@ AsyncSocket::WriteResult AsyncSocket::sendSocketMessage(
   msg.msg_iovlen = std::min<size_t>(count, kIovMax);
   msg.msg_flags = 0; // ignored, must forward flags via sendmsg parameter
   msg.msg_control = nullptr;
-  msg.msg_controllen = sendMsgParamCallback_->getAncillaryDataSize(flags);
+  msg.msg_controllen =
+      sendMsgParamCallback_->getAncillaryDataSize(flags, byteEventsEnabled);
   CHECK_GE(
       AsyncSocket::SendMsgParamsCallback::maxAncillaryDataSize,
       msg.msg_controllen);
 
   if (msg.msg_controllen != 0) {
     msg.msg_control = reinterpret_cast<char*>(alloca(msg.msg_controllen));
-    sendMsgParamCallback_->getAncillaryData(flags, msg.msg_control);
+    sendMsgParamCallback_->getAncillaryData(
+        flags, msg.msg_control, byteEventsEnabled);
   }
   int msg_flags = sendMsgParamCallback_->getFlags(flags, zeroCopyEnabled_);
 
+  const auto prewriteRawBytesWritten = getRawBytesWritten();
   auto writeResult = sendSocketMessage(fd_, &msg, msg_flags);
+  if (writeResult.writeReturn > 0 && byteEventsEnabled &&
+      isSet(flags, WriteFlags::TIMESTAMP_WRITE)) {
+    CHECK_GT(getRawBytesWritten(), prewriteRawBytesWritten); // sanity check
+    ByteEvent byteEvent = {};
+    byteEvent.type = ByteEvent::Type::WRITE;
+    byteEvent.offset = getRawBytesWritten() - 1;
+    byteEvent.maybeWriteFlags = flags;
+    for (const auto& observer : lifecycleObservers_) {
+      if (observer->getConfig().byteEvents) {
+        observer->byteEvent(this, byteEvent);
+      }
+    }
+  }
 
   if (writeResult.writeReturn < 0 && zeroCopyEnabled_ && errno == ENOBUFS) {
     // workaround for running with zerocopy enabled but without a big enough
@@ -2919,6 +3325,17 @@ void AsyncSocket::failAllWrites(const AsyncSocketException& ex) {
   totalAppBytesScheduledForWrite_ = appBytesWritten_;
 }
 
+void AsyncSocket::failByteEvents(const AsyncSocketException& ex) {
+  CHECK(byteEventHelper_) << "failByteEvents called without ByteEventHelper";
+  byteEventHelper_->maybeEx = ex;
+  // inform any observers that want ByteEvents
+  for (const auto& observer : lifecycleObservers_) {
+    if (observer->getConfig().byteEvents) {
+      observer->byteEventsUnavailable(this, ex);
+    }
+  }
+}
+
 void AsyncSocket::invalidState(ConnectCallback* callback) {
   VLOG(5) << "AsyncSocket(this=" << this << ", fd=" << fd_
           << "): connect() called in invalid state " << state_;
@@ -2988,8 +3405,13 @@ void AsyncSocket::invokeConnectSuccess() {
   VLOG(5) << "AsyncSocket(this=" << this << ", fd=" << fd_
           << "): connect success invoked";
   connectEndTime_ = std::chrono::steady_clock::now();
-  for (const auto& cb : lifecycleObservers_) {
-    cb->connect(this);
+  bool enableByteEventsForObserver = false;
+  for (const auto& observer : lifecycleObservers_) {
+    observer->connect(this);
+    enableByteEventsForObserver |= ((observer->getConfig().byteEvents) ? 1 : 0);
+  }
+  if (enableByteEventsForObserver) {
+    enableByteEvents();
   }
   if (connectCallback_) {
     ConnectCallback* callback = connectCallback_;

folly/io/async/AsyncSocket.h

@@ -215,30 +215,41 @@ class AsyncSocket : public AsyncTransport {
     }
 
     /**
-     * getAncillaryData() will be invoked to initialize ancillary data
-     * buffer referred by "msg_control" field of msghdr structure passed to
-     * ::sendmsg() system call based on the flags set in the passed
-     * folly::WriteFlags enum. Some flags in folly::WriteFlags are not relevant
-     * during this process. The function assumes that the size of buffer
-     * is not smaller than the value returned by getAncillaryDataSize() method
-     * for the same combination of flags.
+     * getAncillaryData() will be invoked to initialize ancillary data buffer
+     * referred by "msg_control" field of msghdr structure passed to ::sendmsg()
+     * system call based on the flags set in the passed folly::WriteFlags enum.
+     *
+     * Some flags in folly::WriteFlags are not relevant during this process;
+     * the default implementation only handles timestamping flags.
+     *
+     * The function requires that the size of buffer passed is equal to the
+     * value returned by getAncillaryDataSize() method for the same combination
+     * of flags.
      *
      * @param flags     Write flags requested for the given write operation
      * @param data      Pointer to ancillary data buffer to initialize.
+     * @param byteEventsEnabled      If byte events are enabled for this socket.
+     *                               When enabled, flags relevant to socket
+     *                               timestamps (e.g., TIMESTAMP_TX) should be
+     *                               included in ancillary (msg_control) data.
      */
     virtual void getAncillaryData(
-        folly::WriteFlags /*flags*/, void* /*data*/) noexcept {}
+        folly::WriteFlags flags,
+        void* data,
+        const bool byteEventsEnabled = false) noexcept;
 
     /**
      * getAncillaryDataSize() will be invoked to retrieve the size of
      * ancillary data buffer which should be passed to ::sendmsg() system call
      *
      * @param flags     Write flags requested for the given write operation
+     * @param byteEventsEnabled      If byte events are enabled for this socket.
+     *                               When enabled, flags relevant to socket
+     *                               timestamps (e.g., TIMESTAMP_TX) should be
+     *                               included in ancillary (msg_control) data.
      */
     virtual uint32_t getAncillaryDataSize(
-        folly::WriteFlags /*flags*/) noexcept {
-      return 0;
-    }
+        folly::WriteFlags flags, const bool byteEventsEnabled = false) noexcept;
 
     static const size_t maxAncillaryDataSize{0x5000};
 
@@ -269,6 +280,72 @@ class AsyncSocket : public AsyncTransport {
     int getDefaultFlags(folly::WriteFlags flags, bool zeroCopyEnabled) noexcept;
   };
 
+  /**
+   * Container with state and processing logic for ByteEvents.
+   */
+  struct ByteEventHelper {
+    bool byteEventsEnabled{false};
+    size_t rawBytesWrittenWhenByteEventsEnabled{0};
+    folly::Optional<AsyncSocketException> maybeEx;
+
+    /**
+     * Process a Cmsg and return a ByteEvent if available.
+     *
+     * The kernel will pass two cmsg for each timestamp:
+     *   1. ScmTimestamping: Software / Hardware Timestamps.
+     *   2. SockExtendedErrTimestamping: Byte offset associated with timestamp.
+     *
+     * These messages will be passed back-to-back; processCmsg() can handle them
+     * in any order (1 then 2, or 2 then 1), as long the order is consistent
+     * across timestamps.
+     *
+     * processCmsg() gracefully ignores Cmsg unrelated to socket timestamps, but
+     * will throw if it receives a sequence of Cmsg that are not compliant with
+     * its expectations.
+     *
+     * @return If the helper has received all components required to generate a
+     *         ByteEvent (e.g., ScmTimestamping and SockExtendedErrTimestamping
+     *         messages), it returns a ByteEvent and clears its local state.
+     *         Otherwise, returns an empty optional.
+     *
+     *         If the helper has previously thrown a ByteEventHelper::Exception,
+     *         it will not process further Cmsg and will continiously return an
+     *         empty optional.
+     *
+     * @throw  If the helper receives a sequence of Cmsg that violate its
+     *         expectations (e.g., multiple ScmTimestamping messages in a row
+     *         without corresponding SockExtendedErrTimestamping messages), it
+     *         throws a ByteEventHelper::Exception. Subsequent calls will return
+     *         an empty optional.
+     */
+    folly::Optional<ByteEvent> processCmsg(
+        const cmsghdr& cmsg, const size_t rawBytesWritten);
+
+    /**
+     * Exception class thrown by processCmsg.
+     *
+     * ByteEventHelper does not know the socket address and thus cannot
+     * construct a AsyncSocketException. Instead, ByteEventHelper throws a
+     * custom Exception and AsyncSocket rewraps it as an AsyncSocketException.
+     */
+    class Exception : public std::runtime_error {
+      using std::runtime_error::runtime_error;
+    };
+
+   private:
+    // state, reinitialized each time a complete timestamp is processed
+    struct TimestampState {
+      bool serrReceived{false};
+      uint32_t typeRaw{0};
+      uint32_t byteOffsetKernel{0};
+
+      bool scmTsReceived{false};
+      folly::Optional<std::chrono::nanoseconds> maybeSoftwareTs;
+      folly::Optional<std::chrono::nanoseconds> maybeHardwareTs;
+    };
+    folly::Optional<TimestampState> maybeTsState_;
+  };
+
   explicit AsyncSocket();
   /**
    * Create a new unconnected AsyncSocket.
@@ -668,6 +745,8 @@ class AsyncSocket : public AsyncTransport {
   }
   size_t getRawBytesBuffered() const override { return getAppBytesBuffered(); }
 
+  // End of methods inherited from AsyncTransport
+
   std::chrono::nanoseconds getConnectTime() const {
     return connectEndTime_ - connectStartTime_;
   }
@@ -1026,7 +1105,7 @@ class AsyncSocket : public AsyncTransport {
     uint32_t totalBytesWritten_{0}; ///< total bytes written
   };
 
-  class LifecycleObserver : public AsyncTransport::LifecycleObserver {
+  class LifecycleObserver : virtual public AsyncTransport::LifecycleObserver {
    public:
     using AsyncTransport::LifecycleObserver::LifecycleObserver;
 
@@ -1367,6 +1446,7 @@ class AsyncSocket : public AsyncTransport {
       const AsyncSocketException& ex);
   void failWrite(const char* fn, const AsyncSocketException& ex);
   void failAllWrites(const AsyncSocketException& ex);
+  void failByteEvents(const AsyncSocketException& ex);
   virtual void invokeConnectErr(const AsyncSocketException& ex);
   virtual void invokeConnectSuccess();
   void invalidState(ConnectCallback* callback);
@@ -1397,6 +1477,27 @@ class AsyncSocket : public AsyncTransport {
   bool containsZeroCopyBuf(folly::IOBuf* ptr);
   void releaseZeroCopyBuf(uint32_t id);
 
+  /**
+   * Attempt to enable Observer ByteEvents for this socket.
+   *
+   * Once enabled, ByteEvents rename enabled for the socket's life.
+   *
+   * ByteEvents are delivered to Observers; when an observer is added:
+   *    - If this function has already been called, byteEventsEnabled() or
+   *      byteEventsUnavailable() will be called, depending on ByteEvent state.
+   *    - Else if the socket is connected, this function is called immediately.
+   *    - Else if the socket has not yet connected, this function will be called
+   *      after the socket has connected (ByteEvents cannot be set up earlier).
+   *
+   * If ByteEvents are successfully enabled, byteEventsEnabled() will be called
+   * on each Observer that has requested ByteEvents. If unable to enable, or if
+   * ByteEvents become unavailable (e.g., due to close), byteEventsUnavailable()
+   * will be called on each Observer that has requested ByteEvents.
+   *
+   * This function does need to be explicitly called under other circumstances.
+   */
+  virtual void enableByteEvents();
+
   AsyncWriter::ZeroCopyEnableFunc zeroCopyEnableFunc_;
 
   // a folly::IOBuf can be used in multiple partial requests
@@ -1477,6 +1578,10 @@ class AsyncSocket : public AsyncTransport {
   bool noTSocks_{false};
   // Whether to track EOR or not.
   bool trackEor_{false};
+
+  // ByteEvent state
+  std::unique_ptr<ByteEventHelper> byteEventHelper_;
+
   bool zeroCopyEnabled_{false};
   bool zeroCopyVal_{false};
   // zerocopy re-enable logic

folly/io/async/AsyncTransport.h

@@ -16,8 +16,10 @@
 
 #pragma once
 
+#include <chrono>
 #include <memory>
 
+#include <folly/Optional.h>
 #include <folly/io/IOBuf.h>
 #include <folly/io/async/AsyncSocketBase.h>
 #include <folly/io/async/AsyncTransportCertificate.h>
@@ -81,6 +83,10 @@ enum class WriteFlags : uint32_t {
    * Request timestamp when entire buffer has entered packet scheduler.
    */
   TIMESTAMP_SCHED = 0x40,
+  /*
+   * Request timestamp when entire buffer has been written to system socket.
+   */
+  TIMESTAMP_WRITE = 0x80,
 };
 
 /*
@@ -136,6 +142,12 @@ constexpr bool isSet(WriteFlags a, WriteFlags b) {
   return (a & b) == b;
 }
 
+/**
+ * Write flags that are related to timestamping.
+ */
+constexpr WriteFlags kWriteFlagsForTimestamping = WriteFlags::TIMESTAMP_SCHED |
+    WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+
 class AsyncReader {
  public:
   class ReadCallback {
@@ -743,10 +755,97 @@ class AsyncTransport : public DelayedDestruction,
     }
   }
 
+  /**
+   * Structure used to communicate ByteEvents, such as TX and ACK timestamps.
+   */
+  struct ByteEvent {
+    enum Type : uint8_t { WRITE = 1, SCHED = 2, TX = 3, ACK = 4 };
+    // type
+    Type type;
+
+    // offset of corresponding byte in raw byte stream
+    uint64_t offset{0};
+
+    // transport timestamp, as recorded by AsyncTransport implementation
+    std::chrono::steady_clock::time_point ts = {
+        std::chrono::steady_clock::now()};
+
+    // kernel software timestamp; for Linux this is CLOCK_REALTIME
+    // see https://www.kernel.org/doc/Documentation/networking/timestamping.txt
+    folly::Optional<std::chrono::nanoseconds> maybeSoftwareTs;
+
+    // hardware timestamp; see kernel documentation
+    // see https://www.kernel.org/doc/Documentation/networking/timestamping.txt
+    folly::Optional<std::chrono::nanoseconds> maybeHardwareTs;
+
+    // for WRITE ByteEvents, additional WriteFlags passed
+    folly::Optional<WriteFlags> maybeWriteFlags;
+
+    /**
+     * For WRITE events, returns if SCHED timestamp requested.
+     */
+    bool schedTimestampRequested() const {
+      CHECK_EQ(Type::WRITE, type);
+      CHECK(maybeWriteFlags.has_value());
+      return isSet(*maybeWriteFlags, WriteFlags::TIMESTAMP_SCHED);
+    }
+
+    /**
+     * For WRITE events, returns if TX timestamp requested.
+     */
+    bool txTimestampRequested() const {
+      CHECK_EQ(Type::WRITE, type);
+      CHECK(maybeWriteFlags.has_value());
+      return isSet(*maybeWriteFlags, WriteFlags::TIMESTAMP_TX);
+    }
+
+    /**
+     * For WRITE events, returns if ACK timestamp requested.
+     */
+    bool ackTimestampRequested() const {
+      CHECK_EQ(Type::WRITE, type);
+      CHECK(maybeWriteFlags.has_value());
+      return isSet(*maybeWriteFlags, WriteFlags::TIMESTAMP_ACK);
+    }
+  };
+
+  /**
+   * Observer of transport events.
+   */
   class LifecycleObserver {
    public:
+    /**
+     * Observer configuration.
+     *
+     * Specifies events observer wants to receive. Cannot be changed post
+     * initialization because the transport may turn on / off instrumentation
+     * when observers are added / removed, based on the observer configuration.
+     */
+    struct Config {
+      // enables full support for ByteEvents
+      bool byteEvents{false};
+    };
+
+    /**
+     * Constructor for observer, uses default config (instrumentation disabled).
+     */
+    LifecycleObserver() : LifecycleObserver(Config()) {}
+
+    /**
+     * Constructor for observer.
+     *
+     * @param config      Config, defaults to auxilary instrumentaton disabled.
+     */
+    explicit LifecycleObserver(const Config& observerConfig)
+        : observerConfig_(observerConfig) {}
+
     virtual ~LifecycleObserver() = default;
 
+    /**
+     * Returns observers configuration.
+     */
+    const Config& getConfig() { return observerConfig_; }
+
     /**
      * observerAttach() will be invoked when an observer is added.
      *
@@ -794,26 +893,75 @@ class AsyncTransport : public DelayedDestruction,
     virtual void connect(AsyncTransport* /* transport */) noexcept = 0;
 
     /**
-     * Called when the socket has been attached to a new EVB
-     * and is called from within the new EVB's thread
+     * Invoked when the transport is being attached to an EventBase.
+     *
+     * Called from within the EventBase thread being attached.
      *
-     * @param socket    The socket on which the new EVB was attached.
-     * @param evb       The new event base that is being attached.
+     * @param transport   Transport with EventBase change.
+     * @param evb         The EventBase being attached.
      */
-    virtual void evbAttach(AsyncTransport* /* socket */, EventBase* /* evb */) {
-      // do nothing
-    }
+    virtual void evbAttach(
+        AsyncTransport* /* transport */, EventBase* /* evb */) {}
 
     /**
-     * Called when the socket is detached from an EVB and
-     * is called from the existing EVB's thread.
+     * Invoked when the transport is being detached from an EventBase.
+     *
+     * Called from within the EventBase thread being detached.
      *
-     * @param socket    The socket from which the EVB was detached.
-     * @param evb       The existing evb that is being detached.
+     * @param transport   Transport with EventBase change.
+     * @param evb         The EventBase that is being detached.
      */
-    virtual void evbDetach(AsyncTransport* /* socket */, EventBase* /* evb */) {
-      // do nothing
-    }
+    virtual void evbDetach(
+        AsyncTransport* /* transport */, EventBase* /* evb */) {}
+
+    /**
+     * Invoked each time a ByteEvent is available.
+     *
+     * Multiple ByteEvent may be generated for the same byte offset and event.
+     * For instance, kernel software and hardware TX timestamps for the same
+     * are delivered in separate CMsg, and thus will result in separate
+     * ByteEvent.
+     *
+     * @param transport   Transport that ByteEvent is available for.
+     * @param event       ByteEvent (WRITE, SCHED, TX, ACK).
+     */
+    virtual void byteEvent(
+        AsyncTransport* /* transport */,
+        const ByteEvent& /* event */) noexcept {}
+
+    /**
+     * Invoked if ByteEvents are enabled.
+     *
+     * Only called if the observer's configuration requested ByteEvents. May
+     * be invoked multiple times if ByteEvent configuration changes (i.e., if
+     * ByteEvents are enabled without hardware timestamps, and then enabled
+     * with them).
+     *
+     * @param transport    Transport that ByteEvents are enabled for.
+     */
+    virtual void byteEventsEnabled(AsyncTransport* /* transport */) noexcept {}
+
+    /**
+     * Invoked if ByteEvents could not be enabled, or if an error occurred that
+     * will prevent further delivery of ByteEvents.
+     *
+     * An observer may be waiting to receive a ByteEvent, such as an ACK event
+     * confirming delivery of the last byte of a payload, before closing the
+     * transport. If the transport has become unhealthy then this ByteEvent may
+     * never occur, yet the handler may be unaware that the transport is
+     * unhealthy if reads have been shutdown and no writes are occurring; this
+     * observer signal breaks this 'deadlock'.
+     *
+     * @param transport   Transport that ByteEvents are now unavailable for.
+     * @param ex          Details on why ByteEvents are now unavailable.
+     */
+    virtual void byteEventsUnavailable(
+        AsyncTransport* /* transport */,
+        const AsyncSocketException& /* ex */) noexcept {}
+
+   protected:
+    // observer configuration; cannot be changed post instantiation
+    const Config observerConfig_;
   };
 
   /**

folly/io/async/test/AsyncSSLSocketTest.cpp

@@ -35,8 +35,10 @@
 #include <folly/io/async/EventBaseThread.h>
 #include <folly/io/async/ScopedEventBaseThread.h>
 #include <folly/io/async/test/BlockingSocket.h>
+#include <folly/io/async/test/MockAsyncTransportObserver.h>
 #include <folly/net/NetOps.h>
 #include <folly/net/NetworkSocket.h>
+#include <folly/net/test/MockNetOpsDispatcher.h>
 #include <folly/portability/GMock.h>
 #include <folly/portability/GTest.h>
 #include <folly/portability/OpenSSL.h>
@@ -45,6 +47,8 @@
 
 #ifdef __linux__
 #include <dlfcn.h>
+#include <linux/errqueue.h>
+#include <linux/net_tstamp.h>
 #endif
 
 #if FOLLY_OPENSSL_IS_110
@@ -2980,107 +2984,481 @@ TEST(AsyncSSLSocketTest, SendMsgParamsCallback) {
   cerr << "SendMsgParamsCallback test completed" << endl;
 }
 
-#ifdef FOLLY_HAVE_MSG_ERRQUEUE
-/**
- * Test connecting to, writing to, reading from, and closing the
- * connection to the SSL server with ancillary data from the application.
- */
-TEST(AsyncSSLSocketTest, SendMsgDataCallback) {
-  // This test requires Linux kernel v4.6 or later
-  struct utsname s_uname;
-  memset(&s_uname, 0, sizeof(s_uname));
-  ASSERT_EQ(uname(&s_uname), 0);
-  int major, minor;
-  folly::StringPiece extra;
-  if (folly::split<false>(
-          '.', std::string(s_uname.release) + ".", major, minor, extra)) {
-    if (major < 4 || (major == 4 && minor < 6)) {
-      LOG(INFO) << "Kernel version: 4.6 and newer required for this test ("
-                << "kernel ver. " << s_uname.release << " detected).";
-      return;
+#ifdef FOLLY_HAVE_SO_TIMESTAMPING
+
+class AsyncSSLSocketByteEventTest : public ::testing::Test {
+ protected:
+  using MockDispatcher = ::testing::NiceMock<netops::test::MockDispatcher>;
+  using TestObserver = test::MockAsyncTransportObserverForByteEvents;
+  using ByteEventType = AsyncTransport::ByteEvent::Type;
+
+  /**
+   * Components of a client connection to TestServer.
+   *
+   * Includes EventBase, client's AsyncSocket.
+   */
+  class ClientConn {
+   public:
+    explicit ClientConn(
+        std::shared_ptr<TestSSLServer> server,
+        std::shared_ptr<AsyncSSLSocket> socket = nullptr)
+        : server_(std::move(server)), socket_(std::move(socket)) {
+      if (!socket_) {
+        socket_ = AsyncSSLSocket::newSocket(getSslContext(), &getEventBase());
+      }
+      socket_->setOverrideNetOpsDispatcher(netOpsDispatcher_);
+      netOpsDispatcher_->forwardToDefaultImpl();
+    }
+
+    ~ClientConn() {
+      if (socket_) {
+        socket_->close();
+      }
+    }
+
+    void connect() {
+      CHECK_NOTNULL(socket_.get());
+      CHECK_NOTNULL(socket_->getEventBase());
+      socket_->connect(&connCb_, server_->getAddress(), 30);
+      socket_->getEventBase()->loop();
+      ASSERT_EQ(connCb_.state, ConnCallback::State::SUCCESS);
+      setReadCb();
+    }
+
+    void setReadCb() {
+      // Due to how libevent works, we currently need to be subscribed to
+      // EV_READ events in order to get error messages.
+      //
+      // TODO(bschlinker): Resolve this with libevent modification.
+      // See https://github.com/libevent/libevent/issues/1038 for details.
+      socket_->setReadCB(&readCb_);
+      readCb_.setSocket(socket_);
+    }
+
+    std::shared_ptr<NiceMock<TestObserver>> attachObserver(
+        bool enableByteEvents) {
+      auto observer = AsyncSSLSocketByteEventTest::attachObserver(
+          socket_.get(), enableByteEvents);
+      observers.push_back(observer);
+      return observer;
+    }
+
+    /**
+     * Write to client socket, echo at server, and wait for echo at client.
+     *
+     * Waiting for echo at client ensures that we have given opportunity for
+     * timestamps to be generated by the kernel.
+     */
+    void writeAndReflect(
+        const std::vector<uint8_t>& wbuf, const WriteFlags writeFlags) {
+      CHECK_NOTNULL(socket_.get());
+      CHECK_NOTNULL(socket_->getEventBase());
+
+      // write to the client socket
+      WriteCallbackBase wcb;
+      socket_->write(&wcb, wbuf.data(), wbuf.size(), writeFlags);
+      while (wcb.state == STATE_WAITING) {
+        socket_->getEventBase()->loopOnce();
+      }
+      ASSERT_EQ(wcb.state, STATE_SUCCEEDED);
+
+      // TestSSLServer reads and reflects for us
+
+      // read reflection at client
+      while (wbuf.size() != readCb_.dataRead()) {
+        socket_->getEventBase()->loopOnce();
+      }
+      readCb_.verifyData(wbuf.data(), wbuf.size());
+      readCb_.clearData();
     }
+
+    std::shared_ptr<AsyncSSLSocket> getRawSocket() { return socket_; }
+
+    std::shared_ptr<SSLContext> getSslContext() {
+      static std::shared_ptr<SSLContext> sslContext = initSslContext();
+      return sslContext;
+    }
+
+    EventBase& getEventBase() {
+      static EventBase evb; // use same EventBase for all client sockets
+      return evb;
+    }
+
+    void netOpsExpectTimestampingSetSockOpt() {
+      // must whitelist other calls
+      EXPECT_CALL(*netOpsDispatcher_, setsockopt(_, _, _, _, _))
+          .Times(AnyNumber());
+      EXPECT_CALL(
+          *netOpsDispatcher_, setsockopt(_, SOL_SOCKET, SO_TIMESTAMPING, _, _))
+          .Times(1);
+    }
+
+    void netOpsExpectNoTimestampingSetSockOpt() {
+      // must whitelist other calls
+      EXPECT_CALL(*netOpsDispatcher_, setsockopt(_, _, _, _, _))
+          .Times(AnyNumber());
+      EXPECT_CALL(
+          *netOpsDispatcher_, setsockopt(_, SOL_SOCKET, SO_TIMESTAMPING, _, _))
+          .Times(0);
+    }
+
+    void netOpsExpectWriteWithFlags(WriteFlags writeFlags) {
+      EXPECT_CALL(*netOpsDispatcher_, sendmsg(_, _, _))
+          .WillOnce(Invoke(
+              [this, writeFlags](
+                  NetworkSocket socket, const msghdr* message, int flags) {
+                EXPECT_EQ(writeFlags, getMsgWriteFlags(*message));
+                return netOpsDispatcher_->netops::Dispatcher::sendmsg(
+                    socket, message, flags);
+              }));
+    }
+
+    void netOpsVerifyAndClearExpectations() {
+      Mock::VerifyAndClearExpectations(netOpsDispatcher_.get());
+    }
+
+    /**
+     * Static utilities.
+     */
+    static std::shared_ptr<SSLContext> initSslContext() {
+      auto sslContext = std::make_shared<SSLContext>();
+      sslContext->ciphers("ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH");
+      return sslContext;
+    }
+
+   private:
+    // server
+    std::shared_ptr<TestSSLServer> server_;
+
+    // managed observers
+    std::vector<std::shared_ptr<TestObserver>> observers;
+
+    // socket components
+    ConnCallback connCb_;
+    ReadCallback readCb_;
+    std::shared_ptr<MockDispatcher> netOpsDispatcher_{
+        std::make_shared<MockDispatcher>()};
+    std::shared_ptr<AsyncSSLSocket> socket_;
+  };
+
+  ClientConn getClientConn() { return ClientConn(server_); }
+
+  void SetUp() override {
+    serverWriteCb_ = std::make_unique<WriteCallbackBase>();
+    serverReadCb_ = std::make_unique<ReadCallback>(serverWriteCb_.get());
+    serverHandshakeCb_ =
+        std::make_unique<HandshakeCallback>(serverReadCb_.get());
+    serverAcceptCb_ =
+        std::make_unique<SSLServerAcceptCallback>(serverHandshakeCb_.get());
+    server_ = std::make_shared<TestSSLServer>(serverAcceptCb_.get());
   }
 
-  // Start listening on a local port
-  SendMsgAncillaryDataCallback msgCallback;
-  WriteCheckTimestampCallback writeCallback(&msgCallback);
-  ReadCallback readCallback(&writeCallback);
-  HandshakeCallback handshakeCallback(&readCallback);
-  SSLServerAcceptCallback acceptCallback(&handshakeCallback);
-  TestSSLServer server(&acceptCallback);
+  /**
+   * Static utility functions.
+   */
 
-  // Set up SSL context.
-  auto sslContext = std::make_shared<SSLContext>();
-  sslContext->ciphers("ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH");
+  static std::shared_ptr<NiceMock<TestObserver>> attachObserver(
+      AsyncSocket* socket, bool enableByteEvents) {
+    AsyncTransport::LifecycleObserver::Config config = {};
+    config.byteEvents = enableByteEvents;
+    return std::make_shared<NiceMock<TestObserver>>(socket, config);
+  }
 
-  // connect
-  auto socket =
-      std::make_shared<BlockingSocket>(server.getAddress(), sslContext);
-  socket->open();
+  static WriteFlags getMsgWriteFlags(const struct msghdr& msg) {
+    const struct cmsghdr* cmsg = CMSG_FIRSTHDR(&msg);
+    if (!cmsg || cmsg->cmsg_level != SOL_SOCKET ||
+        cmsg->cmsg_type != SO_TIMESTAMPING ||
+        cmsg->cmsg_len != CMSG_LEN(sizeof(uint32_t))) {
+      return WriteFlags::NONE;
+    }
 
-  // we'll pass the EOR and TIMESTAMP_TX flags with the write back
-  // EOR tracking must be enabled for WriteFlags be passed
-  const auto writeFlags =
-      folly::WriteFlags::EOR | folly::WriteFlags::TIMESTAMP_TX;
-  readCallback.setWriteFlags(writeFlags);
-  msgCallback.setEorTracking(true);
-
-  // Init ancillary data buffer to trigger timestamp notification
-  //
-  // We generate the same ancillary data regardless of the specific WriteFlags,
-  // we verify that the WriteFlags are observed as expected below.
-  union {
-    uint8_t ctrl_data[CMSG_LEN(sizeof(uint32_t))];
-    struct cmsghdr cmsg;
-  } u;
-  u.cmsg.cmsg_level = SOL_SOCKET;
-  u.cmsg.cmsg_type = SO_TIMESTAMPING;
-  u.cmsg.cmsg_len = CMSG_LEN(sizeof(uint32_t));
-  uint32_t flags = SOF_TIMESTAMPING_TX_SCHED | SOF_TIMESTAMPING_TX_SOFTWARE |
-      SOF_TIMESTAMPING_TX_ACK;
-  memcpy(CMSG_DATA(&u.cmsg), &flags, sizeof(uint32_t));
-  std::vector<char> ctrl(CMSG_LEN(sizeof(uint32_t)));
-  memcpy(ctrl.data(), u.ctrl_data, CMSG_LEN(sizeof(uint32_t)));
-  msgCallback.resetData(std::move(ctrl));
-
-  // write(), including flags
-  std::vector<uint8_t> buf(128, 'a');
-  socket->write(buf.data(), buf.size(), writeFlags);
+    const uint32_t* sofFlags =
+        (reinterpret_cast<const uint32_t*>(CMSG_DATA(cmsg)));
+    WriteFlags flags = WriteFlags::NONE;
+    if (*sofFlags & SOF_TIMESTAMPING_TX_SCHED) {
+      flags = flags | WriteFlags::TIMESTAMP_SCHED;
+    }
+    if (*sofFlags & SOF_TIMESTAMPING_TX_SOFTWARE) {
+      flags = flags | WriteFlags::TIMESTAMP_TX;
+    }
+    if (*sofFlags & SOF_TIMESTAMPING_TX_ACK) {
+      flags = flags | WriteFlags::TIMESTAMP_ACK;
+    }
 
-  // read()
-  std::vector<uint8_t> readbuf(buf.size());
-  uint32_t bytesRead = socket->readAll(readbuf.data(), readbuf.size());
-  EXPECT_EQ(bytesRead, buf.size());
-  EXPECT_TRUE(std::equal(buf.begin(), buf.end(), readbuf.begin()));
-
-  // should receive three timestamps (schedule, TX/SND, ACK)
-  // may take some time for all to arrive, so loop to wait
-  //
-  // socket error queue does not have the equivalent of an EOF, so we must
-  // loop on it unless we want to use libevent for this test...
-  const std::vector<int32_t> timestampsExpected = {
-      SCM_TSTAMP_SCHED, SCM_TSTAMP_SND, SCM_TSTAMP_ACK};
-  std::vector<int32_t> timestampsReceived;
-  while (timestampsExpected.size() != timestampsReceived.size()) {
-    const auto timestamps = writeCallback.getTimestampNotifications();
-    timestampsReceived.insert(
-        timestampsReceived.end(), timestamps.begin(), timestamps.end());
+    return flags;
   }
-  EXPECT_THAT(timestampsReceived, ElementsAreArray(timestampsExpected));
 
-  // check the observed write flags
-  EXPECT_EQ(
-      static_cast<std::underlying_type<folly::WriteFlags>::type>(
-          msgCallback.getObservedWriteFlags()),
-      static_cast<std::underlying_type<folly::WriteFlags>::type>(writeFlags));
+  static WriteFlags dropWriteFromFlags(WriteFlags writeFlags) {
+    return writeFlags & ~WriteFlags::TIMESTAMP_WRITE;
+  }
 
-  // close()
-  socket->close();
-  cerr << "SendMsgDataCallback test completed" << endl;
+  // server components
+  std::unique_ptr<WriteCallbackBase> serverWriteCb_;
+  std::unique_ptr<ReadCallback> serverReadCb_;
+  std::unique_ptr<HandshakeCallback> serverHandshakeCb_;
+  std::unique_ptr<SSLServerAcceptCallback> serverAcceptCb_;
+  std::shared_ptr<TestSSLServer> server_;
+};
+
+TEST_F(AsyncSSLSocketByteEventTest, ObserverAttachedBeforeConnect) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(1, 'a');
+
+  auto clientConn = getClientConn();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  clientConn.netOpsExpectTimestampingSetSockOpt();
+  clientConn.connect();
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  {
+    clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+    clientConn.writeAndReflect(wbuf, flags);
+    clientConn.netOpsVerifyAndClearExpectations();
+
+    // may have more than four new ByteEvents if write split further by kernel
+    EXPECT_THAT(observer->byteEvents, SizeIs(Ge(4)));
+
+    // due to SSL overhead, offset will not be 0
+    auto offsetExpected = clientConn.getRawSocket()->getRawBytesWritten() - 1;
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+
+  // write again to check offsets
+  {
+    const auto startNumByteEvents = observer->byteEvents.size();
+    clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+    clientConn.writeAndReflect(wbuf, flags);
+    clientConn.netOpsVerifyAndClearExpectations();
+
+    // may have more than four new ByteEvents if write split further by kernel
+    EXPECT_THAT(observer->byteEvents, SizeIs(Ge(startNumByteEvents + 4)));
+
+    // due to SSL overhead, offset will not be 1
+    auto offsetExpected = clientConn.getRawSocket()->getRawBytesWritten() - 1;
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
 }
-#endif // FOLLY_HAVE_MSG_ERRQUEUE
 
-#endif
+TEST_F(AsyncSSLSocketByteEventTest, ObserverAttachedAfterConnect) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(1, 'a');
+
+  auto clientConn = getClientConn();
+  clientConn.netOpsExpectNoTimestampingSetSockOpt();
+  clientConn.connect();
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  clientConn.netOpsExpectTimestampingSetSockOpt();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  {
+    clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+    clientConn.writeAndReflect(wbuf, flags);
+    clientConn.netOpsVerifyAndClearExpectations();
+
+    // may have more than four new ByteEvents if write split further by kernel
+    EXPECT_THAT(observer->byteEvents, SizeIs(Ge(4)));
+
+    // due to SSL overhead, offset will not be 0
+    auto offsetExpected = clientConn.getRawSocket()->getRawBytesWritten() - 1;
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+
+  // write again to check offsets
+  {
+    const auto startNumByteEvents = observer->byteEvents.size();
+    clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+    clientConn.writeAndReflect(wbuf, flags);
+    clientConn.netOpsVerifyAndClearExpectations();
+
+    // may have more than four new ByteEvents if write split further by kernel
+    EXPECT_THAT(observer->byteEvents, SizeIs(Ge(startNumByteEvents + 4)));
+
+    // due to SSL overhead, offset will not be 1
+    auto offsetExpected = clientConn.getRawSocket()->getRawBytesWritten() - 1;
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+}
+
+TEST_F(AsyncSSLSocketByteEventTest, MultiByteWrites) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+
+  auto clientConn = getClientConn();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  clientConn.netOpsExpectTimestampingSetSockOpt();
+  clientConn.connect();
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  // write 20 bytes
+  {
+    std::vector<uint8_t> wbuf(20, 'a'); // 20 bytes
+
+    clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+    clientConn.writeAndReflect(wbuf, flags);
+    clientConn.netOpsVerifyAndClearExpectations();
+
+    // may have more than four new ByteEvents if write split further by kernel
+    EXPECT_THAT(observer->byteEvents, SizeIs(Ge(4)));
+
+    // due to SSL overhead, offset will not be 0
+    auto offsetExpected = clientConn.getRawSocket()->getRawBytesWritten() - 1;
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+
+  // write 40 bytes
+  {
+    std::vector<uint8_t> wbuf(20, 'a'); // 20 bytes
+
+    const auto startNumByteEvents = observer->byteEvents.size();
+    clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+    clientConn.writeAndReflect(wbuf, flags);
+    clientConn.netOpsVerifyAndClearExpectations();
+
+    // may have more than four new ByteEvents if write split further by kernel
+    EXPECT_THAT(observer->byteEvents, SizeIs(Ge(startNumByteEvents + 4)));
+
+    // due to SSL overhead, offset will not be 1
+    auto offsetExpected = clientConn.getRawSocket()->getRawBytesWritten() - 1;
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+}
+
+TEST_F(AsyncSSLSocketByteEventTest, MultiByteWritesEnableSecondWrite) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+
+  auto clientConn = getClientConn();
+  clientConn.netOpsExpectNoTimestampingSetSockOpt();
+  clientConn.connect();
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  // write 20 bytes with no ByteEvents / observer
+  {
+    std::vector<uint8_t> wbuf(20, 'a'); // 20 bytes
+    clientConn.netOpsExpectWriteWithFlags(WriteFlags::NONE);
+    clientConn.writeAndReflect(wbuf, flags);
+    clientConn.netOpsVerifyAndClearExpectations();
+  }
+
+  // enable observer
+  clientConn.netOpsExpectTimestampingSetSockOpt();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  // write 40 bytes
+  {
+    std::vector<uint8_t> wbuf(20, 'a'); // 20 bytes
+
+    const auto startNumByteEvents = observer->byteEvents.size();
+    clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+    clientConn.writeAndReflect(wbuf, flags);
+    clientConn.netOpsVerifyAndClearExpectations();
+
+    // may have more than four new ByteEvents if write split further by kernel
+    EXPECT_THAT(observer->byteEvents, SizeIs(Ge(startNumByteEvents + 4)));
+
+    // due to SSL overhead, offset will not be 1
+    auto offsetExpected = clientConn.getRawSocket()->getRawBytesWritten() - 1;
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        offsetExpected,
+        observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+}
+
+#endif // FOLLY_HAVE_SO_TIMESTAMPING
+
+#endif // __linux__
 
 TEST(AsyncSSLSocketTest, TestSNIClientHelloBehavior) {
   EventBase eventBase;

folly/io/async/test/AsyncSSLSocketTest.h

@@ -74,12 +74,16 @@ class SendMsgParamsCallbackBase
     return oldCallback_->getFlags(flags, false /*zeroCopyEnabled*/);
   }
 
-  void getAncillaryData(folly::WriteFlags flags, void* data) noexcept override {
-    oldCallback_->getAncillaryData(flags, data);
+  void getAncillaryData(
+      folly::WriteFlags flags,
+      void* data,
+      const bool byteEventsEnabled) noexcept override {
+    oldCallback_->getAncillaryData(flags, data, byteEventsEnabled);
   }
 
-  uint32_t getAncillaryDataSize(folly::WriteFlags flags) noexcept override {
-    return oldCallback_->getAncillaryDataSize(flags);
+  uint32_t getAncillaryDataSize(
+      folly::WriteFlags flags, const bool byteEventsEnabled) noexcept override {
+    return oldCallback_->getAncillaryDataSize(flags, byteEventsEnabled);
   }
 
   std::shared_ptr<AsyncSSLSocket> socket_;
@@ -119,7 +123,10 @@ class SendMsgAncillaryDataCallback : public SendMsgParamsCallbackBase {
    */
   folly::WriteFlags getObservedWriteFlags() { return observedWriteFlags_; }
 
-  void getAncillaryData(folly::WriteFlags flags, void* data) noexcept override {
+  void getAncillaryData(
+      folly::WriteFlags flags,
+      void* data,
+      const bool byteEventsEnabled) noexcept override {
     // getAncillaryData is called through a long chain of functions after send
     // record the observed write flags so we can compare later
     observedWriteFlags_ = flags;
@@ -128,16 +135,17 @@ class SendMsgAncillaryDataCallback : public SendMsgParamsCallbackBase {
       std::cerr << "getAncillaryData: copying data" << std::endl;
       memcpy(data, ancillaryData_.data(), ancillaryData_.size());
     } else {
-      oldCallback_->getAncillaryData(flags, data);
+      oldCallback_->getAncillaryData(flags, data, byteEventsEnabled);
     }
   }
 
-  uint32_t getAncillaryDataSize(folly::WriteFlags flags) noexcept override {
+  uint32_t getAncillaryDataSize(
+      folly::WriteFlags flags, const bool byteEventsEnabled) noexcept override {
     if (ancillaryData_.size()) {
       std::cerr << "getAncillaryDataSize: returning size" << std::endl;
       return ancillaryData_.size();
     } else {
-      return oldCallback_->getAncillaryDataSize(flags);
+      return oldCallback_->getAncillaryDataSize(flags, byteEventsEnabled);
     }
   }
 
@@ -201,114 +209,6 @@ class ExpectWriteErrorCallback : public WriteCallbackBase {
   }
 };
 
-#ifdef FOLLY_HAVE_MSG_ERRQUEUE
-/* copied from include/uapi/linux/net_tstamp.h */
-/* SO_TIMESTAMPING gets an integer bit field comprised of these values */
-enum SOF_TIMESTAMPING {
-  SOF_TIMESTAMPING_TX_SOFTWARE = (1 << 1),
-  SOF_TIMESTAMPING_SOFTWARE = (1 << 4),
-  SOF_TIMESTAMPING_OPT_ID = (1 << 7),
-  SOF_TIMESTAMPING_TX_SCHED = (1 << 8),
-  SOF_TIMESTAMPING_TX_ACK = (1 << 9),
-  SOF_TIMESTAMPING_OPT_TSONLY = (1 << 11),
-};
-
-class WriteCheckTimestampCallback : public WriteCallbackBase {
- public:
-  explicit WriteCheckTimestampCallback(SendMsgParamsCallbackBase* mcb = nullptr)
-      : WriteCallbackBase(mcb) {}
-
-  ~WriteCheckTimestampCallback() override { EXPECT_EQ(STATE_SUCCEEDED, state); }
-
-  void setSocket(const std::shared_ptr<AsyncSSLSocket>& socket) override {
-    WriteCallbackBase::setSocket(socket);
-
-    EXPECT_NE(socket_->getNetworkSocket(), NetworkSocket());
-    int flags = SOF_TIMESTAMPING_OPT_ID | SOF_TIMESTAMPING_OPT_TSONLY |
-        SOF_TIMESTAMPING_SOFTWARE;
-    SocketOptionKey tstampingOpt = {SOL_SOCKET, SO_TIMESTAMPING};
-    int ret = tstampingOpt.apply(socket_->getNetworkSocket(), flags);
-    EXPECT_EQ(ret, 0);
-  }
-
-  std::vector<int32_t> getTimestampNotifications() noexcept {
-    auto fd = socket_->getNetworkSocket();
-    std::vector<char> ctrl(1024, 0);
-    unsigned char data;
-    struct msghdr msg;
-    iovec entry;
-
-    memset(&msg, 0, sizeof(msg));
-    entry.iov_base = &data;
-    entry.iov_len = sizeof(data);
-    msg.msg_iov = &entry;
-    msg.msg_iovlen = 1;
-    msg.msg_control = ctrl.data();
-    msg.msg_controllen = ctrl.size();
-
-    std::vector<int32_t> timestampsFound;
-
-    folly::Optional<int32_t> timestampType;
-    bool gotTimestamp = false;
-    bool gotByteSeq = false;
-    int ret;
-    while (true) {
-      ret = netops::recvmsg(fd, &msg, MSG_ERRQUEUE);
-      if (ret < 0) {
-        if (errno != EAGAIN) {
-          auto errnoCopy = errno;
-          std::cerr << "::recvmsg exited with code " << ret
-                    << ", errno: " << errnoCopy << std::endl;
-          AsyncSocketException ex(
-              AsyncSocketException::INTERNAL_ERROR,
-              "recvmsg() failed",
-              errnoCopy);
-          exception = ex;
-        }
-        return timestampsFound;
-      }
-
-      for (struct cmsghdr* cmsg = CMSG_FIRSTHDR(&msg);
-           cmsg != nullptr && cmsg->cmsg_len != 0;
-           cmsg = CMSG_NXTHDR(&msg, cmsg)) {
-        if (cmsg->cmsg_level == SOL_SOCKET &&
-            cmsg->cmsg_type == SCM_TIMESTAMPING) {
-          CHECK(!gotTimestamp); // shouldn't already be set
-          gotTimestamp = true;
-        }
-
-        if ((cmsg->cmsg_level == SOL_IP && cmsg->cmsg_type == IP_RECVERR) ||
-            (cmsg->cmsg_level == SOL_IPV6 && cmsg->cmsg_type == IPV6_RECVERR)) {
-          const struct cmsghdr& cmsgh = *cmsg;
-          const auto serr = reinterpret_cast<const struct sock_extended_err*>(
-              CMSG_DATA(&cmsgh));
-          if (serr->ee_errno != ENOMSG ||
-              serr->ee_origin != SO_EE_ORIGIN_TIMESTAMPING) {
-            // not a timestamp
-            continue;
-          }
-
-          CHECK(!timestampType); // shouldn't already be set
-          CHECK(!gotByteSeq); // shouldn't already be set
-          gotByteSeq = true;
-          timestampType = serr->ee_info;
-        }
-
-        // check if we have both a timestamp and byte sequence
-        if (gotTimestamp && gotByteSeq) {
-          timestampsFound.push_back(*timestampType);
-          timestampType = folly::none;
-          gotTimestamp = false;
-          gotByteSeq = false;
-        }
-      } // for(...)
-    } // while(true)
-
-    return timestampsFound;
-  }
-};
-#endif // FOLLY_HAVE_MSG_ERRQUEUE
-
 class ReadCallbackBase : public AsyncTransport::ReadCallback {
  public:
   explicit ReadCallbackBase(WriteCallbackBase* wcb)
@@ -352,8 +252,13 @@ class ReadCallbackBase : public AsyncTransport::ReadCallback {
  */
 class ReadCallback : public ReadCallbackBase {
  public:
-  explicit ReadCallback(WriteCallbackBase* wcb)
-      : ReadCallbackBase(wcb), buffers(), writeFlags(folly::WriteFlags::NONE) {}
+  explicit ReadCallback(WriteCallbackBase* wcb, bool reflect = true)
+      : ReadCallbackBase(wcb),
+        buffers(),
+        writeFlags(folly::WriteFlags::NONE),
+        reflect(reflect) {}
+
+  explicit ReadCallback() : ReadCallback(nullptr, false) {}
 
   ~ReadCallback() override {
     for (std::vector<Buffer>::iterator it = buffers.begin();
@@ -382,13 +287,46 @@ class ReadCallback : public ReadCallbackBase {
     }
 
     // Write back the same data.
-    socket_->write(wcb_, currentBuffer.buffer, len, writeFlags);
+    if (reflect) {
+      socket_->write(wcb_, currentBuffer.buffer, len, writeFlags);
+    }
 
     buffers.push_back(currentBuffer);
     currentBuffer.reset();
     state = STATE_SUCCEEDED;
   }
 
+  void verifyData(const char* expected, size_t expectedLen) const {
+    verifyData((const unsigned char*)expected, expectedLen);
+  }
+
+  void verifyData(const unsigned char* expected, size_t expectedLen) const {
+    size_t offset = 0;
+    for (size_t idx = 0; idx < buffers.size(); ++idx) {
+      const auto& buf = buffers[idx];
+      size_t cmpLen = std::min(buf.length, expectedLen - offset);
+      CHECK_EQ(memcmp(buf.buffer, expected + offset, cmpLen), 0);
+      CHECK_EQ(cmpLen, buf.length);
+      offset += cmpLen;
+    }
+    CHECK_EQ(offset, expectedLen);
+  }
+
+  void clearData() {
+    for (auto& buffer : buffers) {
+      buffer.free();
+    }
+    buffers.clear();
+  }
+
+  size_t dataRead() const {
+    size_t ret = 0;
+    for (const auto& buf : buffers) {
+      ret += buf.length;
+    }
+    return ret;
+  }
+
   /**
    * These flags will be used when writing the read data back to the socket.
    */
@@ -420,6 +358,7 @@ class ReadCallback : public ReadCallbackBase {
   std::vector<Buffer> buffers;
   Buffer currentBuffer;
   folly::WriteFlags writeFlags;
+  bool reflect; // whether read bytes will be written back to the transport
 };
 
 class ReadErrorCallback : public ReadCallbackBase {

folly/io/async/test/AsyncSocketTest.h

@@ -145,6 +145,10 @@ class ReadCallback : public folly::AsyncTransport::ReadCallback {
   }
 
   void verifyData(const char* expected, size_t expectedLen) const {
+    verifyData((const unsigned char*)expected, expectedLen);
+  }
+
+  void verifyData(const unsigned char* expected, size_t expectedLen) const {
     size_t offset = 0;
     for (size_t idx = 0; idx < buffers.size(); ++idx) {
       const auto& buf = buffers[idx];
@@ -156,6 +160,13 @@ class ReadCallback : public folly::AsyncTransport::ReadCallback {
     CHECK_EQ(offset, expectedLen);
   }
 
+  void clearData() {
+    for (auto& buffer : buffers) {
+      buffer.free();
+    }
+    buffers.clear();
+  }
+
   size_t dataRead() const {
     size_t ret = 0;
     for (const auto& buf : buffers) {
@@ -261,7 +272,10 @@ class TestSendMsgParamsCallback
     return flags_;
   }
 
-  void getAncillaryData(folly::WriteFlags flags, void* data) noexcept override {
+  void getAncillaryData(
+      folly::WriteFlags flags,
+      void* data,
+      const bool /* byteEventsEnabled */) noexcept override {
     queriedData_ = true;
     if (writeFlags_ == folly::WriteFlags::NONE) {
       writeFlags_ = flags;
@@ -272,7 +286,9 @@ class TestSendMsgParamsCallback
     memcpy(data, data_, dataSize_);
   }
 
-  uint32_t getAncillaryDataSize(folly::WriteFlags flags) noexcept override {
+  uint32_t getAncillaryDataSize(
+      folly::WriteFlags flags,
+      const bool /* byteEventsEnabled */) noexcept override {
     if (writeFlags_ == folly::WriteFlags::NONE) {
       writeFlags_ = flags;
     } else {

folly/io/async/test/AsyncSocketTest2.cpp

@@ -34,7 +34,10 @@
 #include <folly/io/async/EventBase.h>
 #include <folly/io/async/ScopedEventBaseThread.h>
 #include <folly/io/async/test/AsyncSocketTest.h>
+#include <folly/io/async/test/MockAsyncSocketObserver.h>
+#include <folly/io/async/test/MockAsyncTransportObserver.h>
 #include <folly/io/async/test/Util.h>
+#include <folly/net/test/MockNetOpsDispatcher.h>
 #include <folly/portability/GMock.h>
 #include <folly/portability/GTest.h>
 #include <folly/portability/Sockets.h>
@@ -42,6 +45,11 @@
 #include <folly/synchronization/Baton.h>
 #include <folly/test/SocketAddressTestHelper.h>
 
+#if defined(__linux__)
+#include <linux/errqueue.h>
+#include <linux/net_tstamp.h>
+#endif
+
 using std::min;
 using std::string;
 using std::unique_ptr;
@@ -3056,19 +3064,6 @@ TEST(AsyncSocket, BytesWrittenWithMove) {
 }
 
 #ifdef FOLLY_HAVE_MSG_ERRQUEUE
-/* copied from include/uapi/linux/net_tstamp.h */
-/* SO_TIMESTAMPING gets an integer bit field comprised of these values */
-enum SOF_TIMESTAMPING {
-  SOF_TIMESTAMPING_TX_SOFTWARE = (1 << 1),
-  SOF_TIMESTAMPING_RX_HARDWARE = (1 << 2),
-  SOF_TIMESTAMPING_RX_SOFTWARE = (1 << 3),
-  SOF_TIMESTAMPING_SOFTWARE = (1 << 4),
-  SOF_TIMESTAMPING_OPT_ID = (1 << 7),
-  SOF_TIMESTAMPING_TX_SCHED = (1 << 8),
-  SOF_TIMESTAMPING_OPT_CMSG = (1 << 10),
-  SOF_TIMESTAMPING_OPT_TSONLY = (1 << 11),
-};
-
 struct AsyncSocketErrMessageCallbackTestParams {
   folly::Optional<int> resetCallbackAfter;
   folly::Optional<int> closeSocketAfter;
@@ -3284,33 +3279,1923 @@ TEST_P(AsyncSocketErrMessageCallbackTest, ErrMessageCallback) {
   ASSERT_EQ(errMsgCB.gotByteSeq_, testParams.gotByteSeqExpected);
   ASSERT_EQ(errMsgCB.gotTimestamp_, testParams.gotTimestampExpected);
 }
+
 #endif // FOLLY_HAVE_MSG_ERRQUEUE
 
-class MockAsyncSocketLifecycleObserver : public AsyncSocket::LifecycleObserver {
+#ifdef FOLLY_HAVE_SO_TIMESTAMPING
+
+class AsyncSocketByteEventTest : public ::testing::Test {
+ protected:
+  using MockDispatcher = ::testing::NiceMock<netops::test::MockDispatcher>;
+  using TestObserver = MockAsyncTransportObserverForByteEvents;
+  using ByteEventType = AsyncTransport::ByteEvent::Type;
+
+  /**
+   * Components of a client connection to TestServer.
+   *
+   * Includes EventBase, client's AsyncSocket, and corresponding server socket.
+   */
+  class ClientConn {
+   public:
+    explicit ClientConn(
+        std::shared_ptr<TestServer> server,
+        std::shared_ptr<AsyncSocket> socket = nullptr,
+        std::shared_ptr<BlockingSocket> acceptedSocket = nullptr)
+        : server_(std::move(server)),
+          socket_(std::move(socket)),
+          acceptedSocket_(std::move(acceptedSocket)) {
+      if (!socket_) {
+        socket_ = AsyncSocket::newSocket(&getEventBase());
+      } else {
+        setReadCb();
+      }
+      socket_->setOverrideNetOpsDispatcher(netOpsDispatcher_);
+      netOpsDispatcher_->forwardToDefaultImpl();
+    }
+
+    void connect() {
+      CHECK_NOTNULL(socket_.get());
+      CHECK_NOTNULL(socket_->getEventBase());
+      socket_->connect(&connCb_, server_->getAddress(), 30);
+      socket_->getEventBase()->loopOnce();
+      ASSERT_EQ(connCb_.state, STATE_SUCCEEDED);
+      setReadCb();
+
+      // accept the socket at the server
+      acceptedSocket_ = server_->accept();
+    }
+
+    void setReadCb() {
+      // Due to how libevent works, we currently need to be subscribed to
+      // EV_READ events in order to get error messages.
+      //
+      // TODO(bschlinker): Resolve this with libevent modification.
+      // See https://github.com/libevent/libevent/issues/1038 for details.
+      socket_->setReadCB(&readCb_);
+    }
+
+    std::shared_ptr<NiceMock<TestObserver>> attachObserver(
+        bool enableByteEvents) {
+      auto observer = AsyncSocketByteEventTest::attachObserver(
+          socket_.get(), enableByteEvents);
+      observers_.push_back(observer);
+      return observer;
+    }
+
+    /**
+     * Write to client socket and read at server.
+     */
+    void write(const std::vector<uint8_t>& wbuf, const WriteFlags writeFlags) {
+      CHECK_NOTNULL(socket_.get());
+      CHECK_NOTNULL(socket_->getEventBase());
+
+      // read buffer for server
+      std::vector<uint8_t> rbuf(wbuf.size(), 0);
+      uint64_t rbufReadBytes = 0;
+
+      // write to the client socket, incrementally read at the server
+      WriteCallback wcb;
+      socket_->write(&wcb, wbuf.data(), wbuf.size(), writeFlags);
+      while (wcb.state == STATE_WAITING) {
+        socket_->getEventBase()->loopOnce();
+        rbufReadBytes += acceptedSocket_->read(
+            rbuf.data() + rbufReadBytes, rbuf.size() - rbufReadBytes);
+      }
+      ASSERT_EQ(wcb.state, STATE_SUCCEEDED);
+
+      // finish reading
+      rbufReadBytes += acceptedSocket_->readAll(
+          rbuf.data() + rbufReadBytes, rbuf.size() - rbufReadBytes);
+      ASSERT_EQ(rbufReadBytes, wbuf.size());
+      ASSERT_TRUE(std::equal(wbuf.begin(), wbuf.end(), rbuf.begin()));
+    }
+
+    /**
+     * Write to client socket, echo at server, and wait for echo at client.
+     *
+     * Waiting for echo at client ensures that we have given opportunity for
+     * timestamps to be generated by the kernel.
+     */
+    void writeAndReflect(
+        const std::vector<uint8_t>& wbuf, const WriteFlags writeFlags) {
+      write(wbuf, writeFlags);
+
+      // reflect
+      acceptedSocket_->write(wbuf.data(), wbuf.size());
+      while (wbuf.size() != readCb_.dataRead()) {
+        socket_->getEventBase()->loopOnce();
+      }
+      readCb_.verifyData(wbuf.data(), wbuf.size());
+      readCb_.clearData();
+    }
+
+    std::shared_ptr<AsyncSocket> getRawSocket() { return socket_; }
+
+    std::shared_ptr<BlockingSocket> getAcceptedSocket() {
+      return acceptedSocket_;
+    }
+
+    EventBase& getEventBase() {
+      static EventBase evb; // use same EventBase for all client sockets
+      return evb;
+    }
+
+    void netOpsExpectTimestampingSetSockOpt() {
+      // must whitelist other calls
+      EXPECT_CALL(*netOpsDispatcher_, setsockopt(_, _, _, _, _))
+          .Times(AnyNumber());
+      EXPECT_CALL(
+          *netOpsDispatcher_, setsockopt(_, SOL_SOCKET, SO_TIMESTAMPING, _, _))
+          .Times(1);
+    }
+
+    void netOpsExpectNoTimestampingSetSockOpt() {
+      // must whitelist other calls
+      EXPECT_CALL(*netOpsDispatcher_, setsockopt(_, _, _, _, _))
+          .Times(AnyNumber());
+      EXPECT_CALL(
+          *netOpsDispatcher_, setsockopt(_, SOL_SOCKET, SO_TIMESTAMPING, _, _))
+          .Times(0);
+    }
+
+    void netOpsExpectWriteWithFlags(WriteFlags writeFlags) {
+      EXPECT_CALL(*netOpsDispatcher_, sendmsg(_, _, _))
+          .WillOnce(Invoke(
+              [this, writeFlags](
+                  NetworkSocket socket, const msghdr* message, int flags) {
+                EXPECT_EQ(writeFlags, getMsgWriteFlags(*message));
+                return netOpsDispatcher_->netops::Dispatcher::sendmsg(
+                    socket, message, flags);
+              }));
+    }
+
+    void netOpsVerifyAndClearExpectations() {
+      Mock::VerifyAndClearExpectations(netOpsDispatcher_.get());
+    }
+
+   private:
+    // server
+    std::shared_ptr<TestServer> server_;
+
+    // managed observers
+    std::vector<std::shared_ptr<TestObserver>> observers_;
+
+    // socket components
+    ConnCallback connCb_;
+    ReadCallback readCb_;
+    std::shared_ptr<MockDispatcher> netOpsDispatcher_{
+        std::make_shared<MockDispatcher>()};
+    std::shared_ptr<AsyncSocket> socket_;
+
+    // accepted socket at server
+    std::shared_ptr<BlockingSocket> acceptedSocket_;
+  };
+
+  ClientConn getClientConn() { return ClientConn(server_); }
+
+  /**
+   * Static utility functions.
+   */
+
+  static std::shared_ptr<NiceMock<TestObserver>> attachObserver(
+      AsyncSocket* socket, bool enableByteEvents) {
+    AsyncTransport::LifecycleObserver::Config config = {};
+    config.byteEvents = enableByteEvents;
+    return std::make_shared<NiceMock<TestObserver>>(socket, config);
+  }
+
+  static WriteFlags getMsgWriteFlags(const struct msghdr& msg) {
+    const struct cmsghdr* cmsg = CMSG_FIRSTHDR(&msg);
+    if (!cmsg || cmsg->cmsg_level != SOL_SOCKET ||
+        cmsg->cmsg_type != SO_TIMESTAMPING ||
+        cmsg->cmsg_len != CMSG_LEN(sizeof(uint32_t))) {
+      return WriteFlags::NONE;
+    }
+
+    const uint32_t* sofFlags =
+        (reinterpret_cast<const uint32_t*>(CMSG_DATA(cmsg)));
+    WriteFlags flags = WriteFlags::NONE;
+    if (*sofFlags & SOF_TIMESTAMPING_TX_SCHED) {
+      flags = flags | WriteFlags::TIMESTAMP_SCHED;
+    }
+    if (*sofFlags & SOF_TIMESTAMPING_TX_SOFTWARE) {
+      flags = flags | WriteFlags::TIMESTAMP_TX;
+    }
+    if (*sofFlags & SOF_TIMESTAMPING_TX_ACK) {
+      flags = flags | WriteFlags::TIMESTAMP_ACK;
+    }
+
+    return flags;
+  }
+
+  static WriteFlags dropWriteFromFlags(WriteFlags writeFlags) {
+    return writeFlags & ~WriteFlags::TIMESTAMP_WRITE;
+  }
+
+  // server
+  std::shared_ptr<TestServer> server_{std::make_shared<TestServer>()};
+};
+
+TEST_F(AsyncSocketByteEventTest, GetMsgWriteFlags) {
+  auto ancillaryDataSize = CMSG_LEN(sizeof(uint32_t));
+  auto ancillaryData = reinterpret_cast<char*>(alloca(ancillaryDataSize));
+
+  auto getMsg = [&ancillaryDataSize, &ancillaryData](uint32_t sofFlags) {
+    struct msghdr msg = {};
+    msg.msg_name = nullptr;
+    msg.msg_namelen = 0;
+    msg.msg_iov = nullptr;
+    msg.msg_iovlen = 0;
+    msg.msg_flags = 0;
+    msg.msg_controllen = 0;
+    msg.msg_control = nullptr;
+    if (sofFlags) {
+      msg.msg_controllen = ancillaryDataSize;
+      msg.msg_control = ancillaryData;
+      struct cmsghdr* cmsg = CMSG_FIRSTHDR(&msg);
+      CHECK_NOTNULL(cmsg);
+      cmsg->cmsg_level = SOL_SOCKET;
+      cmsg->cmsg_type = SO_TIMESTAMPING;
+      cmsg->cmsg_len = CMSG_LEN(sizeof(uint32_t));
+      memcpy(CMSG_DATA(cmsg), &sofFlags, sizeof(sofFlags));
+    }
+    return msg;
+  };
+
+  // SCHED
+  {
+    auto msg = getMsg(SOF_TIMESTAMPING_TX_SCHED);
+    EXPECT_EQ(WriteFlags::TIMESTAMP_SCHED, getMsgWriteFlags(msg));
+  }
+
+  // TX
+  {
+    auto msg = getMsg(SOF_TIMESTAMPING_TX_SOFTWARE);
+    EXPECT_EQ(WriteFlags::TIMESTAMP_TX, getMsgWriteFlags(msg));
+  }
+
+  // ACK
+  {
+    auto msg = getMsg(SOF_TIMESTAMPING_TX_ACK);
+    EXPECT_EQ(WriteFlags::TIMESTAMP_ACK, getMsgWriteFlags(msg));
+  }
+
+  // SCHED + TX + ACK
+  {
+    auto msg = getMsg(
+        SOF_TIMESTAMPING_TX_SCHED | SOF_TIMESTAMPING_TX_SOFTWARE |
+        SOF_TIMESTAMPING_TX_ACK);
+    EXPECT_EQ(
+        WriteFlags::TIMESTAMP_SCHED | WriteFlags::TIMESTAMP_TX |
+            WriteFlags::TIMESTAMP_ACK,
+        getMsgWriteFlags(msg));
+  }
+}
+
+TEST_F(AsyncSocketByteEventTest, ObserverAttachedBeforeConnect) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(1, 'a');
+
+  auto clientConn = getClientConn();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  clientConn.netOpsExpectTimestampingSetSockOpt();
+  clientConn.connect();
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer->byteEvents, SizeIs(4));
+  EXPECT_EQ(0, observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+  EXPECT_EQ(0, observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+  EXPECT_EQ(0, observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+  EXPECT_EQ(0, observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+
+  // write again to check offsets
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer->byteEvents, SizeIs(8));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+}
+
+TEST_F(AsyncSocketByteEventTest, ObserverAttachedAfterConnect) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(1, 'a');
+
+  auto clientConn = getClientConn();
+  clientConn.netOpsExpectNoTimestampingSetSockOpt();
+  clientConn.connect();
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  clientConn.netOpsExpectTimestampingSetSockOpt();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer->byteEvents, SizeIs(4));
+  EXPECT_EQ(0, observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+  EXPECT_EQ(0, observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+  EXPECT_EQ(0, observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+  EXPECT_EQ(0, observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+
+  // write again to check offsets
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer->byteEvents, SizeIs(8));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+}
+
+TEST_F(
+    AsyncSocketByteEventTest, ObserverAttachedBeforeConnectByteEventsDisabled) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(1, 'a');
+
+  auto clientConn = getClientConn();
+  auto observer = clientConn.attachObserver(false /* enableByteEvents */);
+  clientConn.netOpsExpectNoTimestampingSetSockOpt();
+
+  clientConn.connect(); // connect after observer attached
+  EXPECT_EQ(0, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  clientConn.netOpsExpectWriteWithFlags(WriteFlags::NONE); // events disabled
+  clientConn.writeAndReflect(wbuf, flags);
+  EXPECT_THAT(observer->byteEvents, IsEmpty());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  // now enable ByteEvents with another observer, then write again
+  clientConn.netOpsExpectTimestampingSetSockOpt();
+  auto observer2 = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(0, observer->byteEventsEnabledCalled); // observer 1 doesn't want
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  EXPECT_EQ(1, observer2->byteEventsEnabledCalled); // should be set
+  EXPECT_EQ(0, observer2->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  EXPECT_NE(WriteFlags::NONE, flags);
+  EXPECT_NE(WriteFlags::NONE, dropWriteFromFlags(flags));
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  // expect no ByteEvents for first observer, four for the second
+  EXPECT_THAT(observer->byteEvents, IsEmpty());
+  EXPECT_THAT(observer2->byteEvents, SizeIs(4));
+  EXPECT_EQ(1, observer2->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+  EXPECT_EQ(1, observer2->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+  EXPECT_EQ(1, observer2->maxOffsetForByteEventReceived(ByteEventType::TX));
+  EXPECT_EQ(1, observer2->maxOffsetForByteEventReceived(ByteEventType::ACK));
+}
+
+TEST_F(
+    AsyncSocketByteEventTest, ObserverAttachedAfterConnectByteEventsDisabled) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(1, 'a');
+
+  auto clientConn = getClientConn();
+  clientConn.netOpsExpectNoTimestampingSetSockOpt();
+
+  clientConn.connect(); // connect before observer attached
+
+  auto observer = clientConn.attachObserver(false /* enableByteEvents */);
+  EXPECT_EQ(0, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  clientConn.netOpsExpectWriteWithFlags(WriteFlags::NONE); // events disabled
+  clientConn.writeAndReflect(wbuf, flags);
+  EXPECT_THAT(observer->byteEvents, IsEmpty());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  // now enable ByteEvents with another observer, then write again
+  clientConn.netOpsExpectTimestampingSetSockOpt();
+  auto observer2 = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(0, observer->byteEventsEnabledCalled); // observer 1 doesn't want
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  EXPECT_EQ(1, observer2->byteEventsEnabledCalled); // should be set
+  EXPECT_EQ(0, observer2->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  EXPECT_NE(WriteFlags::NONE, flags);
+  EXPECT_NE(WriteFlags::NONE, dropWriteFromFlags(flags));
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  // expect no ByteEvents for first observer, four for the second
+  EXPECT_THAT(observer->byteEvents, IsEmpty());
+  EXPECT_THAT(observer2->byteEvents, SizeIs(4));
+  EXPECT_EQ(1, observer2->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+  EXPECT_EQ(1, observer2->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+  EXPECT_EQ(1, observer2->maxOffsetForByteEventReceived(ByteEventType::TX));
+  EXPECT_EQ(1, observer2->maxOffsetForByteEventReceived(ByteEventType::ACK));
+}
+
+TEST_F(AsyncSocketByteEventTest, ObserverAttachedAfterWrite) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(1, 'a');
+
+  auto clientConn = getClientConn();
+  clientConn.netOpsExpectNoTimestampingSetSockOpt();
+  clientConn.connect(); // connect before observer attached
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  clientConn.netOpsExpectWriteWithFlags(WriteFlags::NONE); // events disabled
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  clientConn.netOpsExpectTimestampingSetSockOpt();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  EXPECT_THAT(observer->byteEvents, SizeIs(4));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+}
+
+TEST_F(AsyncSocketByteEventTest, ObserverAttachedAfterClose) {
+  auto clientConn = getClientConn();
+  clientConn.connect();
+  clientConn.getRawSocket()->close();
+  EXPECT_TRUE(clientConn.getRawSocket()->isClosedBySelf());
+
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(0, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+}
+
+TEST_F(AsyncSocketByteEventTest, MultipleObserverAttached) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(50, 'a');
+
+  // attach observer 1 before connect
+  auto clientConn = getClientConn();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  clientConn.netOpsExpectTimestampingSetSockOpt();
+  clientConn.connect();
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  // attach observer 2 after connect
+  auto observer2 = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer2->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer2->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+
+  // write
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  // check observer1
+  EXPECT_THAT(observer->byteEvents, SizeIs(4));
+  EXPECT_EQ(49, observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+  EXPECT_EQ(49, observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+  EXPECT_EQ(49, observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+  EXPECT_EQ(49, observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+
+  // check observer2
+  EXPECT_THAT(observer2->byteEvents, SizeIs(4));
+  EXPECT_EQ(49, observer2->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+  EXPECT_EQ(49, observer2->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+  EXPECT_EQ(49, observer2->maxOffsetForByteEventReceived(ByteEventType::TX));
+  EXPECT_EQ(49, observer2->maxOffsetForByteEventReceived(ByteEventType::ACK));
+}
+
+/**
+ * Test when kernel offset (uint32_t) wraps around.
+ */
+TEST_F(AsyncSocketByteEventTest, KernelOffsetWrap) {
+  auto clientConn = getClientConn();
+  clientConn.connect();
+  clientConn.netOpsExpectTimestampingSetSockOpt();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  const uint64_t wbufSize = 3000000;
+  const std::vector<uint8_t> wbuf(wbufSize, 'a');
+
+  // part 1: write close to the wrap point with no ByteEvents to speed things up
+  const uint64_t bytesToWritePt1 =
+      static_cast<uint64_t>(std::numeric_limits<uint32_t>::max()) -
+      (wbufSize * 5);
+  while (clientConn.getRawSocket()->getRawBytesWritten() < bytesToWritePt1) {
+    clientConn.write(wbuf, WriteFlags::NONE); // no reflect needed
+  }
+
+  // part 2: write over the wrap point with ByteEvents
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const uint64_t bytesToWritePt2 =
+      static_cast<uint64_t>(std::numeric_limits<uint32_t>::max()) +
+      (wbufSize * 5);
+  while (clientConn.getRawSocket()->getRawBytesWritten() < bytesToWritePt2) {
+    clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+    clientConn.writeAndReflect(wbuf, flags);
+    clientConn.netOpsVerifyAndClearExpectations();
+    const uint64_t expectedOffset =
+        clientConn.getRawSocket()->getRawBytesWritten() - 1;
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+
+  // part 3: one more write outside of a loop with extra checks
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+  const auto expectedOffset =
+      clientConn.getRawSocket()->getRawBytesWritten() - 1;
+  EXPECT_LT(std::numeric_limits<uint32_t>::max(), expectedOffset);
+  EXPECT_EQ(
+      expectedOffset,
+      observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+  EXPECT_EQ(
+      expectedOffset,
+      observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+  EXPECT_EQ(
+      expectedOffset,
+      observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+  EXPECT_EQ(
+      expectedOffset,
+      observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+}
+
+/**
+ * Ensure that ErrMessageCallback still works when ByteEvents enabled.
+ */
+TEST_F(AsyncSocketByteEventTest, ErrMessageCallbackStillTriggered) {
+  auto clientConn = getClientConn();
+  clientConn.connect();
+  clientConn.netOpsExpectTimestampingSetSockOpt();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  TestErrMessageCallback errMsgCB;
+  clientConn.getRawSocket()->setErrMessageCB(&errMsgCB);
+
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+
+  std::vector<uint8_t> wbuf(1, 'a');
+  EXPECT_NE(WriteFlags::NONE, flags);
+  EXPECT_NE(WriteFlags::NONE, dropWriteFromFlags(flags));
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  // observer should get events
+  EXPECT_THAT(observer->byteEvents, SizeIs(4));
+  EXPECT_EQ(0, observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+  EXPECT_EQ(0, observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+  EXPECT_EQ(0, observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+  EXPECT_EQ(0, observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+
+  // err message callbach should get events, too
+  EXPECT_EQ(3, errMsgCB.gotByteSeq_);
+  EXPECT_EQ(3, errMsgCB.gotTimestamp_);
+
+  // write again, more events for both
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer->byteEvents, SizeIs(8));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+  EXPECT_EQ(1, observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  EXPECT_EQ(6, errMsgCB.gotByteSeq_);
+  EXPECT_EQ(6, errMsgCB.gotTimestamp_);
+}
+
+/**
+ * Ensure that ByteEvents disabled for unix sockets (not supported).
+ */
+TEST_F(AsyncSocketByteEventTest, FailUnixSocket) {
+  std::shared_ptr<NiceMock<TestObserver>> observer;
+  auto netOpsDispatcher = std::make_shared<MockDispatcher>();
+
+  NetworkSocket fd[2];
+  EXPECT_EQ(netops::socketpair(AF_UNIX, SOCK_STREAM, 0, fd), 0);
+  ASSERT_NE(fd[0], NetworkSocket());
+  ASSERT_NE(fd[1], NetworkSocket());
+  SCOPE_EXIT { netops::close(fd[1]); };
+
+  EXPECT_EQ(netops::set_socket_non_blocking(fd[0]), 0);
+  EXPECT_EQ(netops::set_socket_non_blocking(fd[1]), 0);
+
+  auto clientSocketRaw = AsyncSocket::newSocket(nullptr, fd[0]);
+  auto clientBlockingSocket = BlockingSocket(std::move(clientSocketRaw));
+  clientBlockingSocket.getSocket()->setOverrideNetOpsDispatcher(
+      netOpsDispatcher);
+
+  // make sure no SO_TIMESTAMPING setsockopt on observer attach
+  EXPECT_CALL(*netOpsDispatcher, setsockopt(_, _, _, _, _)).Times(AnyNumber());
+  EXPECT_CALL(
+      *netOpsDispatcher, setsockopt(_, SOL_SOCKET, SO_TIMESTAMPING, _, _))
+      .Times(0); // no calls
+  observer = attachObserver(
+      clientBlockingSocket.getSocket(), true /* enableByteEvents */);
+  EXPECT_EQ(0, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(1, observer->byteEventsUnavailableCalled);
+  EXPECT_TRUE(observer->byteEventsUnavailableCalledEx.has_value());
+  Mock::VerifyAndClearExpectations(netOpsDispatcher.get());
+
+  // do a write, we should see it has no timestamp flags
+  const std::vector<uint8_t> wbuf(1, 'a');
+  EXPECT_CALL(*netOpsDispatcher, sendmsg(_, _, _))
+      .WillOnce(WithArgs<1>(Invoke([](const msghdr* message) {
+        EXPECT_EQ(WriteFlags::NONE, getMsgWriteFlags(*message));
+        return 1;
+      })));
+  clientBlockingSocket.write(
+      wbuf.data(),
+      wbuf.size(),
+      WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+          WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK);
+  Mock::VerifyAndClearExpectations(netOpsDispatcher.get());
+}
+
+/**
+ * If socket timestamps already enabled, do not enable ByteEvents.
+ */
+TEST_F(AsyncSocketByteEventTest, FailTimestampsAlreadyEnabled) {
+  auto clientConn = getClientConn();
+  clientConn.connect();
+
+  // enable timestamps via setsockopt
+  const uint32_t flags = SOF_TIMESTAMPING_OPT_ID | SOF_TIMESTAMPING_OPT_TSONLY |
+      SOF_TIMESTAMPING_SOFTWARE | SOF_TIMESTAMPING_RAW_HARDWARE |
+      SOF_TIMESTAMPING_OPT_TX_SWHW;
+  const auto ret = clientConn.getRawSocket()->setSockOpt(
+      SOL_SOCKET, SO_TIMESTAMPING, &flags);
+  EXPECT_EQ(0, ret);
+
+  clientConn.netOpsExpectNoTimestampingSetSockOpt();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(0, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(1, observer->byteEventsUnavailableCalled); // fail
+  EXPECT_TRUE(observer->byteEventsUnavailableCalledEx.has_value());
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  std::vector<uint8_t> wbuf(1, 'a');
+  clientConn.netOpsExpectWriteWithFlags(WriteFlags::NONE);
+  clientConn.writeAndReflect(
+      wbuf,
+      WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+          WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK);
+  clientConn.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer->byteEvents, IsEmpty());
+}
+
+/**
+ * Verify that ByteEvent information is properly copied during socket moves.
+ */
+
+TEST_F(AsyncSocketByteEventTest, MoveByteEventsEnabled) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(50, 'a');
+
+  auto clientConn = getClientConn();
+  clientConn.connect();
+
+  // observer with ByteEvents enabled
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+
+  // move the socket immediately and add an observer with ByteEvents enabled
+  auto clientConn2 = ClientConn(
+      server_,
+      AsyncSocket::UniquePtr(new AsyncSocket(clientConn.getRawSocket().get())),
+      clientConn.getAcceptedSocket());
+  auto observer2 = clientConn2.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer2->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer2->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+
+  // write following move, make sure the offsets are correct
+  clientConn2.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn2.writeAndReflect(wbuf, flags);
+  clientConn2.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer2->byteEvents, SizeIs(Ge(4)));
+  {
+    const auto expectedOffset = 49;
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+
+  // write again
+  clientConn2.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn2.writeAndReflect(wbuf, flags);
+  clientConn2.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer2->byteEvents, SizeIs(Ge(8)));
+  {
+    const auto expectedOffset = 99;
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+}
+
+TEST_F(AsyncSocketByteEventTest, WriteThenMoveByteEventsEnabled) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(50, 'a');
+
+  auto clientConn = getClientConn();
+  clientConn.connect();
+
+  // observer with ByteEvents enabled
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+
+  // write
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer->byteEvents, SizeIs(Ge(4)));
+  {
+    const auto expectedOffset = 49;
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+
+  // now move the socket and add an observer with ByteEvents enabled
+  auto clientConn2 = ClientConn(
+      server_,
+      AsyncSocket::UniquePtr(
+          new AsyncSocket(std::move(clientConn.getRawSocket().get()))),
+      clientConn.getAcceptedSocket());
+  auto observer2 = clientConn2.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer2->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer2->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+
+  // write following move, make sure the offsets are correct
+  clientConn2.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn2.writeAndReflect(wbuf, flags);
+  clientConn2.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer2->byteEvents, SizeIs(Ge(4)));
+  {
+    const auto expectedOffset = 99;
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+
+  // write again
+  clientConn2.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn2.writeAndReflect(wbuf, flags);
+  clientConn2.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer2->byteEvents, SizeIs(Ge(8)));
+  {
+    const auto expectedOffset = 149;
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+}
+
+TEST_F(AsyncSocketByteEventTest, MoveThenEnableByteEvents) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(50, 'a');
+
+  auto clientConn = getClientConn();
+  clientConn.connect();
+
+  // observer with ByteEvents disabled
+  auto observer = clientConn.attachObserver(false /* enableByteEvents */);
+  EXPECT_EQ(0, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+
+  // move the socket immediately and add an observer with ByteEvents enabled
+  auto clientConn2 = ClientConn(
+      server_,
+      AsyncSocket::UniquePtr(new AsyncSocket(clientConn.getRawSocket().get())),
+      clientConn.getAcceptedSocket());
+  auto observer2 = clientConn2.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer2->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer2->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+
+  // write following move, make sure the offsets are correct
+  clientConn2.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn2.writeAndReflect(wbuf, flags);
+  clientConn2.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer2->byteEvents, SizeIs(Ge(4)));
+  {
+    const auto expectedOffset = 49;
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+
+  // write again
+  clientConn2.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn2.writeAndReflect(wbuf, flags);
+  clientConn2.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer2->byteEvents, SizeIs(Ge(8)));
+  {
+    const auto expectedOffset = 99;
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+}
+
+TEST_F(AsyncSocketByteEventTest, WriteThenMoveThenEnableByteEvents) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(50, 'a');
+
+  auto clientConn = getClientConn();
+  clientConn.connect();
+
+  // observer with ByteEvents disabled
+  auto observer = clientConn.attachObserver(false /* enableByteEvents */);
+  EXPECT_EQ(0, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+
+  // write, ByteEvents disabled
+  clientConn.netOpsExpectWriteWithFlags(WriteFlags::NONE); // events diabled
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+
+  // now move the socket and add an observer with ByteEvents enabled
+  auto clientConn2 = ClientConn(
+      server_,
+      AsyncSocket::UniquePtr(new AsyncSocket(clientConn.getRawSocket().get())),
+      clientConn.getAcceptedSocket());
+  auto observer2 = clientConn2.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer2->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer2->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+
+  // write following move, make sure the offsets are correct
+  clientConn2.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn2.writeAndReflect(wbuf, flags);
+  clientConn2.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer2->byteEvents, SizeIs(Ge(4)));
+  {
+    const auto expectedOffset = 99;
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+
+  // write again
+  clientConn2.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn2.writeAndReflect(wbuf, flags);
+  clientConn2.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer2->byteEvents, SizeIs(Ge(8)));
+  {
+    const auto expectedOffset = 149;
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        expectedOffset,
+        observer2->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+}
+
+TEST_F(AsyncSocketByteEventTest, NoObserverMoveThenEnableByteEvents) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(50, 'a');
+
+  auto clientConn = getClientConn();
+  clientConn.connect();
+
+  // no observer
+
+  // move the socket immediately and add an observer with ByteEvents enabled
+  auto clientConn2 = ClientConn(
+      server_,
+      AsyncSocket::UniquePtr(new AsyncSocket(clientConn.getRawSocket().get())),
+      clientConn.getAcceptedSocket());
+  auto observer = clientConn2.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+
+  // write following move, make sure the offsets are correct
+  clientConn2.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn2.writeAndReflect(wbuf, flags);
+  clientConn2.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer->byteEvents, SizeIs(Ge(4)));
+  {
+    const auto expectedOffset = 49;
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+
+  // write again
+  clientConn2.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn2.writeAndReflect(wbuf, flags);
+  clientConn2.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer->byteEvents, SizeIs(Ge(8)));
+  {
+    const auto expectedOffset = 99;
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::WRITE));
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::SCHED));
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::TX));
+    EXPECT_EQ(
+        expectedOffset,
+        observer->maxOffsetForByteEventReceived(ByteEventType::ACK));
+  }
+}
+
+/**
+ * Inspect ByteEvent fields, including xTimestampRequested in WRITE events.
+ */
+TEST_F(AsyncSocketByteEventTest, CheckByteEventDetails) {
+  const auto flags = WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+      WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK;
+  const std::vector<uint8_t> wbuf(1, 'a');
+
+  auto clientConn = getClientConn();
+  clientConn.connect();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+
+  EXPECT_NE(WriteFlags::NONE, dropWriteFromFlags(flags));
+  clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+  clientConn.writeAndReflect(wbuf, flags);
+  clientConn.netOpsVerifyAndClearExpectations();
+  EXPECT_THAT(observer->byteEvents, SizeIs(Eq(4)));
+  const auto expectedOffset = wbuf.size() - 1;
+
+  // check WRITE
+  {
+    auto maybeByteEvent = observer->getByteEventReceivedWithOffset(
+        expectedOffset, ByteEventType::WRITE);
+    ASSERT_TRUE(maybeByteEvent.has_value());
+    auto& byteEvent = maybeByteEvent.value();
+
+    EXPECT_EQ(ByteEventType::WRITE, byteEvent.type);
+    EXPECT_EQ(expectedOffset, byteEvent.offset);
+    EXPECT_GE(std::chrono::steady_clock::now(), byteEvent.ts);
+    EXPECT_LT(
+        std::chrono::steady_clock::now() - std::chrono::seconds(60),
+        byteEvent.ts);
+
+    EXPECT_EQ(flags, byteEvent.maybeWriteFlags);
+    EXPECT_TRUE(byteEvent.schedTimestampRequested());
+    EXPECT_TRUE(byteEvent.txTimestampRequested());
+    EXPECT_TRUE(byteEvent.ackTimestampRequested());
+
+    EXPECT_FALSE(byteEvent.maybeSoftwareTs.has_value());
+    EXPECT_FALSE(byteEvent.maybeHardwareTs.has_value());
+  }
+
+  // check SCHED, TX, ACK
+  for (const auto& byteEventType :
+       {ByteEventType::SCHED, ByteEventType::TX, ByteEventType::ACK}) {
+    auto maybeByteEvent =
+        observer->getByteEventReceivedWithOffset(expectedOffset, byteEventType);
+    ASSERT_TRUE(maybeByteEvent.has_value());
+    auto& byteEvent = maybeByteEvent.value();
+
+    EXPECT_EQ(byteEventType, byteEvent.type);
+    EXPECT_EQ(expectedOffset, byteEvent.offset);
+    EXPECT_GE(std::chrono::steady_clock::now(), byteEvent.ts);
+    EXPECT_LT(
+        std::chrono::steady_clock::now() - std::chrono::seconds(60),
+        byteEvent.ts);
+
+    EXPECT_FALSE(byteEvent.maybeWriteFlags.has_value());
+    EXPECT_DEATH(byteEvent.schedTimestampRequested(), ".*");
+    EXPECT_DEATH(byteEvent.txTimestampRequested(), ".*");
+    EXPECT_DEATH(byteEvent.ackTimestampRequested(), ".*");
+
+    EXPECT_TRUE(byteEvent.maybeSoftwareTs.has_value());
+    EXPECT_FALSE(byteEvent.maybeHardwareTs.has_value());
+  }
+}
+
+struct AsyncSocketByteEventDetailsTestParams {
+  struct WriteParams {
+    WriteParams(uint64_t bufferSize, WriteFlags writeFlags)
+        : bufferSize(bufferSize), writeFlags(writeFlags) {}
+    uint64_t bufferSize{0};
+    WriteFlags writeFlags{WriteFlags::NONE};
+  };
+
+  std::vector<WriteParams> writesWithParams;
+};
+
+class AsyncSocketByteEventDetailsTest
+    : public AsyncSocketByteEventTest,
+      public testing::WithParamInterface<
+          AsyncSocketByteEventDetailsTestParams> {
  public:
-  GMOCK_METHOD1_(, noexcept, , observerAttach, void(AsyncTransport*));
-  GMOCK_METHOD1_(, noexcept, , observerDetach, void(AsyncTransport*));
-  GMOCK_METHOD1_(, noexcept, , destroy, void(AsyncTransport*));
-  GMOCK_METHOD1_(, noexcept, , close, void(AsyncTransport*));
-  GMOCK_METHOD1_(, noexcept, , connect, void(AsyncTransport*));
-  GMOCK_METHOD1_(, noexcept, , fdDetach, void(AsyncSocket*));
-  GMOCK_METHOD2_(, noexcept, , move, void(AsyncSocket*, AsyncSocket*));
-  GMOCK_METHOD2_(, noexcept, , evbAttach, void(AsyncTransport*, EventBase*));
-  GMOCK_METHOD2_(, noexcept, , evbDetach, void(AsyncTransport*, EventBase*));
+  static std::vector<AsyncSocketByteEventDetailsTestParams> getTestingValues() {
+    const std::array<WriteFlags, 9> writeFlagCombinations{
+        // SCHED
+        WriteFlags::TIMESTAMP_SCHED,
+        // TX
+        WriteFlags::TIMESTAMP_TX,
+        // ACK
+        WriteFlags::TIMESTAMP_ACK,
+        // SCHED + TX + ACK
+        WriteFlags::TIMESTAMP_SCHED | WriteFlags::TIMESTAMP_TX |
+            WriteFlags::TIMESTAMP_ACK,
+        // WRITE
+        WriteFlags::TIMESTAMP_WRITE,
+        // WRITE + SCHED
+        WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED,
+        // WRITE + TX
+        WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_TX,
+        // WRITE + ACK
+        WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_ACK,
+        // WRITE + SCHED + TX + ACK
+        WriteFlags::TIMESTAMP_WRITE | WriteFlags::TIMESTAMP_SCHED |
+            WriteFlags::TIMESTAMP_TX | WriteFlags::TIMESTAMP_ACK,
+    };
+
+    std::vector<AsyncSocketByteEventDetailsTestParams> vals;
+    for (const auto& writeFlags : writeFlagCombinations) {
+      // write 1 byte
+      {
+        AsyncSocketByteEventDetailsTestParams params;
+        params.writesWithParams.emplace_back(1, writeFlags);
+        vals.push_back(params);
+      }
+
+      // write 1 byte twice
+      {
+        AsyncSocketByteEventDetailsTestParams params;
+        params.writesWithParams.emplace_back(1, writeFlags);
+        params.writesWithParams.emplace_back(1, writeFlags);
+        vals.push_back(params);
+      }
+
+      // write 10 bytes
+      {
+        AsyncSocketByteEventDetailsTestParams params;
+        params.writesWithParams.emplace_back(10, writeFlags);
+        vals.push_back(params);
+      }
+
+      // write 10 bytes twice
+      {
+        AsyncSocketByteEventDetailsTestParams params;
+        params.writesWithParams.emplace_back(10, writeFlags);
+        params.writesWithParams.emplace_back(10, writeFlags);
+        vals.push_back(params);
+      }
+    }
+
+    return vals;
+  }
 };
 
-class MockAsyncTransportLifecycleObserver
-    : public AsyncTransport::LifecycleObserver {
+INSTANTIATE_TEST_CASE_P(
+    ByteEventDetailsTest,
+    AsyncSocketByteEventDetailsTest,
+    ::testing::ValuesIn(AsyncSocketByteEventDetailsTest::getTestingValues()));
+
+/**
+ * Inspect ByteEvent fields, including xTimestampRequested in WRITE events.
+ */
+TEST_P(AsyncSocketByteEventDetailsTest, CheckByteEventDetails) {
+  auto params = GetParam();
+
+  auto clientConn = getClientConn();
+  clientConn.connect();
+  auto observer = clientConn.attachObserver(true /* enableByteEvents */);
+  EXPECT_EQ(1, observer->byteEventsEnabledCalled);
+  EXPECT_EQ(0, observer->byteEventsUnavailableCalled);
+  EXPECT_FALSE(observer->byteEventsUnavailableCalledEx.has_value());
+
+  uint64_t expectedNumByteEvents = 0;
+  for (const auto& writeParams : params.writesWithParams) {
+    const std::vector<uint8_t> wbuf(writeParams.bufferSize, 'a');
+    const auto flags = writeParams.writeFlags;
+    clientConn.netOpsExpectWriteWithFlags(dropWriteFromFlags(flags));
+    clientConn.writeAndReflect(wbuf, flags);
+    clientConn.netOpsVerifyAndClearExpectations();
+    const auto expectedOffset =
+        clientConn.getRawSocket()->getRawBytesWritten() - 1;
+
+    // check WRITE
+    if ((flags & WriteFlags::TIMESTAMP_WRITE) != WriteFlags::NONE) {
+      expectedNumByteEvents++;
+
+      auto maybeByteEvent = observer->getByteEventReceivedWithOffset(
+          expectedOffset, ByteEventType::WRITE);
+      ASSERT_TRUE(maybeByteEvent.has_value());
+      auto& byteEvent = maybeByteEvent.value();
+
+      EXPECT_EQ(ByteEventType::WRITE, byteEvent.type);
+      EXPECT_EQ(expectedOffset, byteEvent.offset);
+      EXPECT_GE(std::chrono::steady_clock::now(), byteEvent.ts);
+      EXPECT_LT(
+          std::chrono::steady_clock::now() - std::chrono::seconds(60),
+          byteEvent.ts);
+
+      EXPECT_EQ(flags, byteEvent.maybeWriteFlags);
+      EXPECT_EQ(
+          isSet(flags, WriteFlags::TIMESTAMP_SCHED),
+          byteEvent.schedTimestampRequested());
+      EXPECT_EQ(
+          isSet(flags, WriteFlags::TIMESTAMP_TX),
+          byteEvent.txTimestampRequested());
+      EXPECT_EQ(
+          isSet(flags, WriteFlags::TIMESTAMP_ACK),
+          byteEvent.ackTimestampRequested());
+
+      EXPECT_FALSE(byteEvent.maybeSoftwareTs.has_value());
+      EXPECT_FALSE(byteEvent.maybeHardwareTs.has_value());
+    }
+
+    // check SCHED, TX, ACK
+    for (const auto& byteEventType :
+         {ByteEventType::SCHED, ByteEventType::TX, ByteEventType::ACK}) {
+      auto maybeByteEvent = observer->getByteEventReceivedWithOffset(
+          expectedOffset, byteEventType);
+      switch (byteEventType) {
+        case ByteEventType::WRITE:
+          FAIL();
+        case ByteEventType::SCHED:
+          if ((flags & WriteFlags::TIMESTAMP_SCHED) == WriteFlags::NONE) {
+            EXPECT_FALSE(maybeByteEvent.has_value());
+            continue;
+          }
+          break;
+        case ByteEventType::TX:
+          if ((flags & WriteFlags::TIMESTAMP_TX) == WriteFlags::NONE) {
+            EXPECT_FALSE(maybeByteEvent.has_value());
+            continue;
+          }
+          break;
+        case ByteEventType::ACK:
+          if ((flags & WriteFlags::TIMESTAMP_ACK) == WriteFlags::NONE) {
+            EXPECT_FALSE(maybeByteEvent.has_value());
+            continue;
+          }
+          break;
+      }
+
+      expectedNumByteEvents++;
+      ASSERT_TRUE(maybeByteEvent.has_value());
+      auto& byteEvent = maybeByteEvent.value();
+
+      EXPECT_EQ(byteEventType, byteEvent.type);
+      EXPECT_EQ(expectedOffset, byteEvent.offset);
+      EXPECT_GE(std::chrono::steady_clock::now(), byteEvent.ts);
+      EXPECT_LT(
+          std::chrono::steady_clock::now() - std::chrono::seconds(60),
+          byteEvent.ts);
+      EXPECT_FALSE(byteEvent.maybeWriteFlags.has_value());
+      // don't check xTimestampRequested fields to save time with CHECK_DEATH
+      // already checked in AsyncSocketByteEventTest::CheckByteEventDetails
+
+      EXPECT_TRUE(byteEvent.maybeSoftwareTs.has_value());
+      EXPECT_FALSE(byteEvent.maybeHardwareTs.has_value());
+    }
+  }
+
+  // should have at least expectedNumByteEvents
+  // may be more if writes were split up by kernel
+  EXPECT_THAT(observer->byteEvents, SizeIs(Ge(expectedNumByteEvents)));
+}
+
+class AsyncSocketByteEventHelperTest : public ::testing::Test {
+ protected:
+  using ByteEventType = AsyncTransport::ByteEvent::Type;
+
+  /**
+   * Wrapper around a vector containing cmsg header + data.
+   */
+  class WrappedCMsg {
+   public:
+    explicit WrappedCMsg(std::vector<char>&& data) : data_(std::move(data)) {}
+
+    operator const struct cmsghdr &() {
+      return *reinterpret_cast<struct cmsghdr*>(data_.data());
+    }
+
+   protected:
+    std::vector<char> data_;
+  };
+
+  /**
+   * Wrapper around a vector containing cmsg header + data.
+   */
+  class WrappedSockExtendedErrTsCMsg : public WrappedCMsg {
+   public:
+    using WrappedCMsg::WrappedCMsg;
+
+    // ts[0] -> software timestamp
+    // ts[1] -> hardware timestamp transformed to userspace time (deprecated)
+    // ts[2] -> hardware timestamp
+
+    void setSoftwareTimestamp(
+        const std::chrono::seconds seconds,
+        const std::chrono::nanoseconds nanoseconds) {
+      struct cmsghdr* cmsg{reinterpret_cast<cmsghdr*>(data_.data())};
+      struct scm_timestamping* tss{
+          reinterpret_cast<struct scm_timestamping*>(CMSG_DATA(cmsg))};
+      tss->ts[0].tv_sec = seconds.count();
+      tss->ts[0].tv_nsec = nanoseconds.count();
+    }
+
+    void setHardwareTimestamp(
+        const std::chrono::seconds seconds,
+        const std::chrono::nanoseconds nanoseconds) {
+      struct cmsghdr* cmsg{reinterpret_cast<cmsghdr*>(data_.data())};
+      struct scm_timestamping* tss{
+          reinterpret_cast<struct scm_timestamping*>(CMSG_DATA(cmsg))};
+      tss->ts[2].tv_sec = seconds.count();
+      tss->ts[2].tv_nsec = nanoseconds.count();
+    }
+  };
+
+  static std::vector<char> cmsgData(int level, int type, size_t len) {
+    std::vector<char> data(CMSG_LEN(len), 0);
+    struct cmsghdr* cmsg{reinterpret_cast<cmsghdr*>(data.data())};
+    cmsg->cmsg_level = level;
+    cmsg->cmsg_type = type;
+    cmsg->cmsg_len = CMSG_LEN(len);
+    return data;
+  }
+
+  static WrappedSockExtendedErrTsCMsg cmsgForSockExtendedErrTimestamping() {
+    return WrappedSockExtendedErrTsCMsg(
+        cmsgData(SOL_SOCKET, SO_TIMESTAMPING, sizeof(struct scm_timestamping)));
+  }
+
+  static WrappedCMsg cmsgForScmTimestamping(
+      const uint32_t type, const uint32_t kernelByteOffset) {
+    auto data = cmsgData(SOL_IP, IP_RECVERR, sizeof(struct sock_extended_err));
+    struct cmsghdr* cmsg{reinterpret_cast<cmsghdr*>(data.data())};
+    struct sock_extended_err* serr{
+        reinterpret_cast<struct sock_extended_err*>(CMSG_DATA(cmsg))};
+    serr->ee_errno = ENOMSG;
+    serr->ee_origin = SO_EE_ORIGIN_TIMESTAMPING;
+    serr->ee_info = type;
+    serr->ee_data = kernelByteOffset;
+    return WrappedCMsg(std::move(data));
+  }
+};
+
+TEST_F(AsyncSocketByteEventHelperTest, ByteOffsetThenTs) {
+  auto scmTs = cmsgForScmTimestamping(SCM_TSTAMP_SND, 0);
+  const auto softwareTsSec = std::chrono::seconds(59);
+  const auto softwareTsNs = std::chrono::nanoseconds(11);
+  auto serrTs = cmsgForSockExtendedErrTimestamping();
+  serrTs.setSoftwareTimestamp(softwareTsSec, softwareTsNs);
+
+  AsyncSocket::ByteEventHelper helper = {};
+  helper.byteEventsEnabled = true;
+  helper.rawBytesWrittenWhenByteEventsEnabled = 0;
+
+  EXPECT_FALSE(helper.processCmsg(scmTs, 1 /* rawBytesWritten */));
+  EXPECT_TRUE(helper.processCmsg(serrTs, 1 /* rawBytesWritten */));
+}
+
+TEST_F(AsyncSocketByteEventHelperTest, TsThenByteOffset) {
+  auto scmTs = cmsgForScmTimestamping(SCM_TSTAMP_SND, 0);
+  const auto softwareTsSec = std::chrono::seconds(59);
+  const auto softwareTsNs = std::chrono::nanoseconds(11);
+  auto serrTs = cmsgForSockExtendedErrTimestamping();
+  serrTs.setSoftwareTimestamp(softwareTsSec, softwareTsNs);
+
+  AsyncSocket::ByteEventHelper helper = {};
+  helper.byteEventsEnabled = true;
+  helper.rawBytesWrittenWhenByteEventsEnabled = 0;
+
+  EXPECT_FALSE(helper.processCmsg(serrTs, 1 /* rawBytesWritten */));
+  EXPECT_TRUE(helper.processCmsg(scmTs, 1 /* rawBytesWritten */));
+}
+
+TEST_F(AsyncSocketByteEventHelperTest, ByteEventsDisabled) {
+  auto scmTs = cmsgForScmTimestamping(SCM_TSTAMP_SND, 0);
+  const auto softwareTsSec = std::chrono::seconds(59);
+  const auto softwareTsNs = std::chrono::nanoseconds(11);
+  auto serrTs = cmsgForSockExtendedErrTimestamping();
+  serrTs.setSoftwareTimestamp(softwareTsSec, softwareTsNs);
+
+  AsyncSocket::ByteEventHelper helper = {};
+  helper.byteEventsEnabled = false;
+  helper.rawBytesWrittenWhenByteEventsEnabled = 0;
+
+  // fails because disabled
+  EXPECT_FALSE(helper.processCmsg(scmTs, 1 /* rawBytesWritten */));
+  EXPECT_FALSE(helper.processCmsg(serrTs, 1 /* rawBytesWritten */));
+
+  // enable, try again to prove this works
+  helper.byteEventsEnabled = true;
+  EXPECT_FALSE(helper.processCmsg(scmTs, 1 /* rawBytesWritten */));
+  EXPECT_TRUE(helper.processCmsg(serrTs, 1 /* rawBytesWritten */));
+}
+
+TEST_F(AsyncSocketByteEventHelperTest, IgnoreUnsupportedEvent) {
+  auto scmType = SCM_TSTAMP_ACK + 10; // imaginary new type of SCM event
+  auto scmTs = cmsgForScmTimestamping(scmType, 0);
+  const auto softwareTsSec = std::chrono::seconds(59);
+  const auto softwareTsNs = std::chrono::nanoseconds(11);
+  auto serrTs = cmsgForSockExtendedErrTimestamping();
+  serrTs.setSoftwareTimestamp(softwareTsSec, softwareTsNs);
+
+  AsyncSocket::ByteEventHelper helper = {};
+  helper.byteEventsEnabled = true;
+  helper.rawBytesWrittenWhenByteEventsEnabled = 0;
+
+  // unsupported event is eaten
+  EXPECT_FALSE(helper.processCmsg(scmTs, 1 /* rawBytesWritten */));
+  EXPECT_FALSE(helper.processCmsg(serrTs, 1 /* rawBytesWritten */));
+
+  // change type, try again to prove this works
+  scmTs = cmsgForScmTimestamping(SCM_TSTAMP_ACK, 0);
+  EXPECT_FALSE(helper.processCmsg(scmTs, 1 /* rawBytesWritten */));
+  EXPECT_TRUE(helper.processCmsg(serrTs, 1 /* rawBytesWritten */));
+}
+
+TEST_F(AsyncSocketByteEventHelperTest, ErrorDoubleScmCmsg) {
+  auto scmTs = cmsgForScmTimestamping(SCM_TSTAMP_SND, 0);
+
+  AsyncSocket::ByteEventHelper helper = {};
+  helper.byteEventsEnabled = true;
+  helper.rawBytesWrittenWhenByteEventsEnabled = 0;
+  EXPECT_FALSE(helper.processCmsg(scmTs, 1 /* rawBytesWritten */));
+  EXPECT_THROW(
+      helper.processCmsg(scmTs, 1 /* rawBytesWritten */),
+      AsyncSocket::ByteEventHelper::Exception);
+}
+
+TEST_F(AsyncSocketByteEventHelperTest, ErrorDoubleSerrCmsg) {
+  const auto softwareTsSec = std::chrono::seconds(59);
+  const auto softwareTsNs = std::chrono::nanoseconds(11);
+  auto serrTs = cmsgForSockExtendedErrTimestamping();
+  serrTs.setSoftwareTimestamp(softwareTsSec, softwareTsNs);
+
+  AsyncSocket::ByteEventHelper helper = {};
+  helper.byteEventsEnabled = true;
+  helper.rawBytesWrittenWhenByteEventsEnabled = 0;
+  EXPECT_FALSE(helper.processCmsg(serrTs, 1 /* rawBytesWritten */));
+  EXPECT_THROW(
+      helper.processCmsg(serrTs, 1 /* rawBytesWritten */),
+      AsyncSocket::ByteEventHelper::Exception);
+}
+
+TEST_F(AsyncSocketByteEventHelperTest, ErrorExceptionSet) {
+  auto scmTs = cmsgForScmTimestamping(SCM_TSTAMP_SND, 0);
+  const auto softwareTsSec = std::chrono::seconds(59);
+  const auto softwareTsNs = std::chrono::nanoseconds(11);
+  auto serrTs = cmsgForSockExtendedErrTimestamping();
+  serrTs.setSoftwareTimestamp(softwareTsSec, softwareTsNs);
+
+  AsyncSocket::ByteEventHelper helper = {};
+  helper.byteEventsEnabled = true;
+  helper.rawBytesWrittenWhenByteEventsEnabled = 0;
+  helper.maybeEx = AsyncSocketException(
+      AsyncSocketException::AsyncSocketExceptionType::UNKNOWN, "");
+
+  // fails due to existing exception
+  EXPECT_FALSE(helper.processCmsg(scmTs, 1 /* rawBytesWritten */));
+  EXPECT_FALSE(helper.processCmsg(serrTs, 1 /* rawBytesWritten */));
+
+  // delete the exception, then repeat to prove exception was blocking
+  helper.maybeEx = folly::none;
+  EXPECT_FALSE(helper.processCmsg(scmTs, 1 /* rawBytesWritten */));
+  EXPECT_TRUE(helper.processCmsg(serrTs, 1 /* rawBytesWritten */));
+}
+
+struct AsyncSocketByteEventHelperTimestampTestParams {
+  AsyncSocketByteEventHelperTimestampTestParams(
+      uint32_t scmType,
+      AsyncTransport::ByteEvent::Type expectedByteEventType,
+      bool includeSoftwareTs,
+      bool includeHardwareTs)
+      : scmType(scmType),
+        expectedByteEventType(expectedByteEventType),
+        includeSoftwareTs(includeSoftwareTs),
+        includeHardwareTs(includeHardwareTs) {}
+  uint32_t scmType{0};
+  AsyncTransport::ByteEvent::Type expectedByteEventType;
+  bool includeSoftwareTs{false};
+  bool includeHardwareTs{false};
+};
+
+class AsyncSocketByteEventHelperTimestampTest
+    : public AsyncSocketByteEventHelperTest,
+      public testing::WithParamInterface<
+          AsyncSocketByteEventHelperTimestampTestParams> {
  public:
-  GMOCK_METHOD1_(, noexcept, , observerAttach, void(AsyncTransport*));
-  GMOCK_METHOD1_(, noexcept, , observerDetach, void(AsyncTransport*));
-  GMOCK_METHOD1_(, noexcept, , destroy, void(AsyncTransport*));
-  GMOCK_METHOD1_(, noexcept, , close, void(AsyncTransport*));
-  GMOCK_METHOD1_(, noexcept, , connect, void(AsyncTransport*));
-  GMOCK_METHOD2_(, noexcept, , evbAttach, void(AsyncTransport*, EventBase*));
-  GMOCK_METHOD2_(, noexcept, , evbDetach, void(AsyncTransport*, EventBase*));
+  static std::vector<AsyncSocketByteEventHelperTimestampTestParams>
+  getTestingValues() {
+    std::vector<AsyncSocketByteEventHelperTimestampTestParams> vals;
+
+    // software + hardware timestamps
+    {
+      vals.emplace_back(SCM_TSTAMP_SCHED, ByteEventType::SCHED, true, true);
+      vals.emplace_back(SCM_TSTAMP_SND, ByteEventType::TX, true, true);
+      vals.emplace_back(SCM_TSTAMP_ACK, ByteEventType::ACK, true, true);
+    }
+
+    // software ts only
+    {
+      vals.emplace_back(SCM_TSTAMP_SCHED, ByteEventType::SCHED, true, false);
+      vals.emplace_back(SCM_TSTAMP_SND, ByteEventType::TX, true, false);
+      vals.emplace_back(SCM_TSTAMP_ACK, ByteEventType::ACK, true, false);
+    }
+
+    // hardware ts only
+    {
+      vals.emplace_back(SCM_TSTAMP_SCHED, ByteEventType::SCHED, false, true);
+      vals.emplace_back(SCM_TSTAMP_SND, ByteEventType::TX, false, true);
+      vals.emplace_back(SCM_TSTAMP_ACK, ByteEventType::ACK, false, true);
+    }
+
+    return vals;
+  }
+};
+
+INSTANTIATE_TEST_CASE_P(
+    ByteEventTimestampTest,
+    AsyncSocketByteEventHelperTimestampTest,
+    ::testing::ValuesIn(
+        AsyncSocketByteEventHelperTimestampTest::getTestingValues()));
+
+/**
+ * Check timestamp parsing for software and hardware timestamps.
+ */
+TEST_P(AsyncSocketByteEventHelperTimestampTest, CheckEventTimestamps) {
+  const auto softwareTsSec = std::chrono::seconds(59);
+  const auto softwareTsNs = std::chrono::nanoseconds(11);
+  const auto hardwareTsSec = std::chrono::seconds(79);
+  const auto hardwareTsNs = std::chrono::nanoseconds(31);
+
+  auto params = GetParam();
+  auto scmTs = cmsgForScmTimestamping(params.scmType, 0);
+  auto serrTs = cmsgForSockExtendedErrTimestamping();
+  if (params.includeSoftwareTs) {
+    serrTs.setSoftwareTimestamp(softwareTsSec, softwareTsNs);
+  }
+  if (params.includeHardwareTs) {
+    serrTs.setHardwareTimestamp(hardwareTsSec, hardwareTsNs);
+  }
+
+  AsyncSocket::ByteEventHelper helper = {};
+  helper.byteEventsEnabled = true;
+  helper.rawBytesWrittenWhenByteEventsEnabled = 0;
+  folly::Optional<AsyncTransport::ByteEvent> maybeByteEvent;
+  maybeByteEvent = helper.processCmsg(serrTs, 1 /* rawBytesWritten */);
+  EXPECT_FALSE(maybeByteEvent.has_value());
+  maybeByteEvent = helper.processCmsg(scmTs, 1 /* rawBytesWritten */);
+
+  // common checks
+  ASSERT_TRUE(maybeByteEvent.has_value());
+  const auto& byteEvent = *maybeByteEvent;
+  EXPECT_EQ(0, byteEvent.offset);
+  EXPECT_GE(std::chrono::steady_clock::now(), byteEvent.ts);
+
+  EXPECT_EQ(params.expectedByteEventType, byteEvent.type);
+  if (params.includeSoftwareTs) {
+    EXPECT_EQ(softwareTsSec + softwareTsNs, byteEvent.maybeSoftwareTs);
+  }
+  if (params.includeHardwareTs) {
+    EXPECT_EQ(hardwareTsSec + hardwareTsNs, byteEvent.maybeHardwareTs);
+  }
+}
+
+struct AsyncSocketByteEventHelperOffsetTestParams {
+  uint64_t rawBytesWrittenWhenByteEventsEnabled{0};
+  uint64_t byteTimestamped;
+  uint64_t rawBytesWrittenWhenTimestampReceived;
+};
+
+class AsyncSocketByteEventHelperOffsetTest
+    : public AsyncSocketByteEventHelperTest,
+      public testing::WithParamInterface<
+          AsyncSocketByteEventHelperOffsetTestParams> {
+ public:
+  static std::vector<AsyncSocketByteEventHelperOffsetTestParams>
+  getTestingValues() {
+    std::vector<AsyncSocketByteEventHelperOffsetTestParams> vals;
+    const std::array<uint64_t, 5> rawBytesWrittenWhenByteEventsEnabledVals{
+        0, 1, 100, 4294967295, 4294967296};
+    for (const auto& rawBytesWrittenWhenByteEventsEnabled :
+         rawBytesWrittenWhenByteEventsEnabledVals) {
+      auto addParams = [&](auto params) {
+        // check if case is valid based on rawBytesWrittenWhenByteEventsEnabled
+        if (rawBytesWrittenWhenByteEventsEnabled <= params.byteTimestamped) {
+          vals.push_back(params);
+        }
+      };
+
+      // case 1
+      // bytes sent on receipt of timestamp == byte timestamped
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 0;
+        params.rawBytesWrittenWhenTimestampReceived = 0;
+        addParams(params);
+      }
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 1;
+        params.rawBytesWrittenWhenTimestampReceived = 1;
+        addParams(params);
+      }
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 101;
+        params.rawBytesWrittenWhenTimestampReceived = 101;
+        addParams(params);
+      }
+
+      // bytes sent on receipt of timestamp > byte timestamped
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 1;
+        params.rawBytesWrittenWhenTimestampReceived = 2;
+        addParams(params);
+      }
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 101;
+        params.rawBytesWrittenWhenTimestampReceived = 102;
+        addParams(params);
+      }
+
+      // case 2
+      // bytes sent on receipt of timestamp == byte timestamped, boundary test
+      // (boundary is at 2^32)
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 4294967294;
+        params.rawBytesWrittenWhenTimestampReceived = 4294967294;
+        addParams(params);
+      }
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 4294967295;
+        params.rawBytesWrittenWhenTimestampReceived = 4294967295;
+        addParams(params);
+      }
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 4294967296;
+        params.rawBytesWrittenWhenTimestampReceived = 4294967296;
+        addParams(params);
+      }
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 4294967297;
+        params.rawBytesWrittenWhenTimestampReceived = 4294967297;
+        addParams(params);
+      }
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 4294967298;
+        params.rawBytesWrittenWhenTimestampReceived = 4294967298;
+        addParams(params);
+      }
+
+      // case 3
+      // bytes sent on receipt of timestamp > byte timestamped, boundary test
+      // (boundary is at 2^32)
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 4294967293;
+        params.rawBytesWrittenWhenTimestampReceived = 4294967294;
+        addParams(params);
+      }
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 4294967294;
+        params.rawBytesWrittenWhenTimestampReceived = 4294967295;
+        addParams(params);
+      }
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 4294967295;
+        params.rawBytesWrittenWhenTimestampReceived = 4294967296;
+        addParams(params);
+      }
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 4294967296;
+        params.rawBytesWrittenWhenTimestampReceived = 4294967297;
+        addParams(params);
+      }
+
+      // case 4
+      // bytes sent on receipt of timestamp > byte timestamped, wrap test
+      // (boundary is at 2^32)
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 4294967275;
+        params.rawBytesWrittenWhenTimestampReceived = 4294967305;
+        addParams(params);
+      }
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 4294967295;
+        params.rawBytesWrittenWhenTimestampReceived = 4294967296;
+        addParams(params);
+      }
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 4294967285;
+        params.rawBytesWrittenWhenTimestampReceived = 4294967305;
+        addParams(params);
+      }
+
+      // case 5
+      // special case when timestamp enabled when bytes transferred > (2^32)
+      // bytes sent on receipt of timestamp == byte timestamped, boundary test
+      // (boundary is at 2^32)
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 6442450943;
+        params.rawBytesWrittenWhenTimestampReceived = 6442450943;
+        addParams(params);
+      }
+
+      // case 6
+      // special case when timestamp enabled when bytes transferred > (2^32)
+      // bytes sent on receipt of timestamp > byte timestamped, boundary test
+      // (boundary is at 2^32)
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 6442450943;
+        params.rawBytesWrittenWhenTimestampReceived = 6442450944;
+        addParams(params);
+      }
+
+      // case 7
+      // special case when timestamp enabled when bytes transferred > (2^32)
+      // bytes sent on receipt of timestamp > byte timestamped, wrap test
+      // (boundary is at 2^32)
+      {
+        AsyncSocketByteEventHelperOffsetTestParams params;
+        params.rawBytesWrittenWhenByteEventsEnabled =
+            rawBytesWrittenWhenByteEventsEnabled;
+        params.byteTimestamped = 6442450943;
+        params.rawBytesWrittenWhenTimestampReceived = 8589934591;
+        addParams(params);
+      }
+    }
+
+    return vals;
+  }
 };
 
+INSTANTIATE_TEST_CASE_P(
+    ByteEventOffsetTest,
+    AsyncSocketByteEventHelperOffsetTest,
+    ::testing::ValuesIn(
+        AsyncSocketByteEventHelperOffsetTest::getTestingValues()));
+
+/**
+ * Check byte offset handling, including boundary cases.
+ *
+ * See AsyncSocket::ByteEventHelper::processCmsg for details.
+ */
+TEST_P(AsyncSocketByteEventHelperOffsetTest, CheckCalculatedOffset) {
+  auto params = GetParam();
+
+  // because we use SOF_TIMESTAMPING_OPT_ID, byte offsets delivered from the
+  // kernel are offset (relative to bytes written by AsyncSocket) by the number
+  // of bytes AsyncSocket had written to the socket when enabling timestamps
+  //
+  // here we calculate what the kernel offset would be for the given byte offset
+  const uint64_t bytesPerOffsetWrap =
+      static_cast<uint64_t>(std::numeric_limits<uint32_t>::max()) + 1;
+
+  auto kernelByteOffset =
+      params.byteTimestamped - params.rawBytesWrittenWhenByteEventsEnabled;
+  if (kernelByteOffset > 0) {
+    kernelByteOffset = kernelByteOffset % bytesPerOffsetWrap;
+  }
+
+  auto scmTs = cmsgForScmTimestamping(SCM_TSTAMP_SND, kernelByteOffset);
+  const auto softwareTsSec = std::chrono::seconds(59);
+  const auto softwareTsNs = std::chrono::nanoseconds(11);
+  auto serrTs = cmsgForSockExtendedErrTimestamping();
+  serrTs.setSoftwareTimestamp(softwareTsSec, softwareTsNs);
+
+  AsyncSocket::ByteEventHelper helper = {};
+  helper.byteEventsEnabled = true;
+  helper.rawBytesWrittenWhenByteEventsEnabled =
+      params.rawBytesWrittenWhenByteEventsEnabled;
+
+  EXPECT_FALSE(helper.processCmsg(
+      scmTs,
+      params.rawBytesWrittenWhenTimestampReceived /* rawBytesWritten */));
+  const auto maybeByteEvent = helper.processCmsg(
+      serrTs,
+      params.rawBytesWrittenWhenTimestampReceived /* rawBytesWritten */);
+  ASSERT_TRUE(maybeByteEvent.has_value());
+  const auto& byteEvent = *maybeByteEvent;
+
+  EXPECT_EQ(params.byteTimestamped, byteEvent.offset);
+  EXPECT_EQ(softwareTsSec + softwareTsNs, byteEvent.maybeSoftwareTs);
+}
+
+#endif // FOLLY_HAVE_SO_TIMESTAMPING
+
 TEST(AsyncSocket, LifecycleObserverDetachAndAttachEvb) {
   auto cb = std::make_unique<StrictMock<MockAsyncSocketLifecycleObserver>>();
   EventBase evb;

folly/io/async/test/MockAsyncSocketObserver.h

+/*
+ * Copyright (c) Facebook, Inc. and its affiliates.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <folly/io/async/AsyncSocket.h>
+#include <folly/portability/GMock.h>
+
+namespace folly {
+namespace test {
+
+/*
+ * Mock class for AsyncSocketLifecycleObserver.
+ *
+ * Deriving from MockAsyncTransportLifecycleObserver results in diamond
+ * inheritance that creates a mess for Stict/Weak mocks; easier to just derive
+ * directly from AsyncSocket::LifecycleObserver and clone mocks
+ */
+class MockAsyncSocketLifecycleObserver : public AsyncSocket::LifecycleObserver {
+ public:
+  using AsyncSocket::LifecycleObserver::LifecycleObserver;
+  GMOCK_METHOD1_(, noexcept, , observerAttach, void(AsyncTransport*));
+  GMOCK_METHOD1_(, noexcept, , observerDetach, void(AsyncTransport*));
+  GMOCK_METHOD1_(, noexcept, , destroy, void(AsyncTransport*));
+  GMOCK_METHOD1_(, noexcept, , close, void(AsyncTransport*));
+  GMOCK_METHOD1_(, noexcept, , connect, void(AsyncTransport*));
+  GMOCK_METHOD2_(, noexcept, , evbAttach, void(AsyncTransport*, EventBase*));
+  GMOCK_METHOD2_(, noexcept, , evbDetach, void(AsyncTransport*, EventBase*));
+  GMOCK_METHOD2_(
+      ,
+      noexcept,
+      ,
+      byteEvent,
+      void(AsyncTransport*, const AsyncTransport::ByteEvent&));
+  GMOCK_METHOD1_(, noexcept, , byteEventsEnabled, void(AsyncTransport*));
+  GMOCK_METHOD2_(
+      ,
+      noexcept,
+      ,
+      byteEventsUnavailable,
+      void(AsyncTransport*, const AsyncSocketException&));
+
+  // additional handlers specific to AsyncSocket::LifecycleObserver
+  GMOCK_METHOD1_(, noexcept, , fdDetach, void(AsyncSocket*));
+  GMOCK_METHOD2_(, noexcept, , move, void(AsyncSocket*, AsyncSocket*));
+};
+
+} // namespace test
+} // namespace folly

folly/io/async/test/MockAsyncTransportObserver.h

+/*
+ * Copyright (c) Facebook, Inc. and its affiliates.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <folly/io/async/AsyncSocketException.h>
+#include <folly/io/async/AsyncTransport.h>
+#include <folly/portability/GMock.h>
+
+namespace folly {
+namespace test {
+
+class MockAsyncTransportLifecycleObserver
+    : public AsyncTransport::LifecycleObserver {
+ public:
+  using AsyncTransport::LifecycleObserver::LifecycleObserver;
+  GMOCK_METHOD1_(, noexcept, , observerAttach, void(AsyncTransport*));
+  GMOCK_METHOD1_(, noexcept, , observerDetach, void(AsyncTransport*));
+  GMOCK_METHOD1_(, noexcept, , destroy, void(AsyncTransport*));
+  GMOCK_METHOD1_(, noexcept, , close, void(AsyncTransport*));
+  GMOCK_METHOD1_(, noexcept, , connect, void(AsyncTransport*));
+  GMOCK_METHOD2_(, noexcept, , evbAttach, void(AsyncTransport*, EventBase*));
+  GMOCK_METHOD2_(, noexcept, , evbDetach, void(AsyncTransport*, EventBase*));
+  GMOCK_METHOD2_(
+      ,
+      noexcept,
+      ,
+      byteEvent,
+      void(AsyncTransport*, const AsyncTransport::ByteEvent&));
+  GMOCK_METHOD1_(, noexcept, , byteEventsEnabled, void(AsyncTransport*));
+  GMOCK_METHOD2_(
+      ,
+      noexcept,
+      ,
+      byteEventsUnavailable,
+      void(AsyncTransport*, const AsyncSocketException&));
+};
+
+/**
+ * Extends mock class to simplify ByteEvents tests.
+ */
+class MockAsyncTransportObserverForByteEvents
+    : public MockAsyncTransportLifecycleObserver {
+ public:
+  MockAsyncTransportObserverForByteEvents(
+      AsyncTransport* transport,
+      const MockAsyncTransportObserverForByteEvents::Config& observerConfig)
+      : MockAsyncTransportLifecycleObserver(observerConfig),
+        transport_(transport) {
+    ON_CALL(*this, byteEvent(testing::_, testing::_))
+        .WillByDefault(
+            testing::Invoke([this](
+                                AsyncTransport* transport,
+                                const AsyncTransport::ByteEvent& event) {
+              CHECK_EQ(this->transport_, transport);
+              byteEvents_.emplace_back(event);
+            }));
+    ON_CALL(*this, byteEventsEnabled(testing::_))
+        .WillByDefault(testing::Invoke([this](AsyncTransport* transport) {
+          CHECK_EQ(this->transport_, transport);
+          byteEventsEnabledCalled_++;
+        }));
+
+    ON_CALL(*this, byteEventsUnavailable(testing::_, testing::_))
+        .WillByDefault(testing::Invoke(
+            [this](AsyncTransport* transport, const AsyncSocketException& ex) {
+              CHECK_EQ(this->transport_, transport);
+              byteEventsUnavailableCalled_++;
+              byteEventsUnavailableCalledEx_.emplace(ex);
+            }));
+    transport->addLifecycleObserver(this);
+  }
+
+  folly::Optional<AsyncTransport::ByteEvent> getByteEventReceivedWithOffset(
+      const uint64_t offset, const AsyncTransport::ByteEvent::Type type) {
+    for (const auto& byteEvent : byteEvents_) {
+      if (type == byteEvent.type && offset == byteEvent.offset) {
+        return byteEvent;
+      }
+    }
+    return folly::none;
+  }
+
+  folly::Optional<uint64_t> maxOffsetForByteEventReceived(
+      const AsyncTransport::ByteEvent::Type type) {
+    folly::Optional<uint64_t> maybeMaxOffset;
+    for (const auto& byteEvent : byteEvents_) {
+      if (type == byteEvent.type &&
+          (!maybeMaxOffset.has_value() ||
+           maybeMaxOffset.value() <= byteEvent.offset)) {
+        maybeMaxOffset = byteEvent.offset;
+      }
+    }
+    return maybeMaxOffset;
+  }
+
+  bool checkIfByteEventReceived(
+      const AsyncTransport::ByteEvent::Type type, const uint64_t offset) {
+    for (const auto& byteEvent : byteEvents_) {
+      if (type == byteEvent.type && offset == byteEvent.offset) {
+        return true;
+      }
+    }
+    return false;
+  }
+
+  void waitForByteEvent(
+      const AsyncTransport::ByteEvent::Type type, const uint64_t offset) {
+    while (!checkIfByteEventReceived(type, offset)) {
+      transport_->getEventBase()->loopOnce();
+    }
+  }
+
+  // Exposed ByteEvent helper fields with const
+  const uint32_t& byteEventsEnabledCalled{byteEventsEnabledCalled_};
+  const uint32_t& byteEventsUnavailableCalled{byteEventsUnavailableCalled_};
+  const folly::Optional<AsyncSocketException>& byteEventsUnavailableCalledEx{
+      byteEventsUnavailableCalledEx_};
+  const std::vector<AsyncTransport::ByteEvent>& byteEvents{byteEvents_};
+
+ private:
+  const AsyncTransport* transport_;
+
+  // ByteEvents helpers
+  uint32_t byteEventsEnabledCalled_{0};
+  uint32_t byteEventsUnavailableCalled_{0};
+  folly::Optional<AsyncSocketException> byteEventsUnavailableCalledEx_;
+  std::vector<AsyncTransport::ByteEvent> byteEvents_;
+};
+
+} // namespace test
+} // namespace folly

folly/net/NetOps.h

@@ -116,6 +116,7 @@ struct mmsghdr {
 
 #ifdef MSG_ERRQUEUE
 #define FOLLY_HAVE_MSG_ERRQUEUE 1
+#define FOLLY_HAVE_SO_TIMESTAMPING 1
 /* for struct sock_extended_err*/
 #include <linux/errqueue.h>
 #endif

folly/net/test/MockNetOpsDispatcher.h

+/*
+ * Copyright (c) Facebook, Inc. and its affiliates.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <folly/net/NetOpsDispatcher.h>
+#include <folly/portability/GMock.h>
+
+namespace folly {
+namespace netops {
+namespace test {
+
+class MockDispatcher : public Dispatcher {
+ public:
+  MockDispatcher() = default;
+  virtual ~MockDispatcher() = default;
+
+  /**
+   * Configures mocked methods to forward calls to default implementation.
+   */
+  void forwardToDefaultImpl() {
+    ON_CALL(
+        *this,
+        getsockopt(testing::_, testing::_, testing::_, testing::_, testing::_))
+        .WillByDefault(testing::Invoke([this](
+                                           NetworkSocket s,
+                                           int level,
+                                           int optname,
+                                           void* optval,
+                                           socklen_t* optlen) {
+          return Dispatcher::getsockopt(s, level, optname, optval, optlen);
+        }));
+
+    ON_CALL(*this, sendmsg(testing::_, testing::_, testing::_))
+        .WillByDefault(testing::Invoke(
+            [this](NetworkSocket s, const msghdr* message, int flags) {
+              return Dispatcher::sendmsg(s, message, flags);
+            }));
+
+    ON_CALL(
+        *this,
+        setsockopt(testing::_, testing::_, testing::_, testing::_, testing::_))
+        .WillByDefault(testing::Invoke([this](
+                                           NetworkSocket s,
+                                           int level,
+                                           int optname,
+                                           const void* optval,
+                                           socklen_t optlen) {
+          return Dispatcher::setsockopt(s, level, optname, optval, optlen);
+        }));
+  }
+
+  MOCK_METHOD5(
+      getsockopt,
+      int(NetworkSocket s,
+          int level,
+          int optname,
+          void* optval,
+          socklen_t* optlen));
+
+  MOCK_METHOD3(
+      sendmsg, ssize_t(NetworkSocket s, const msghdr* message, int flags));
+
+  MOCK_METHOD5(
+      setsockopt,
+      int(NetworkSocket s,
+          int level,
+          int optname,
+          const void* optval,
+          socklen_t optlen));
+};
+
+} // namespace test
+} // namespace netops
+} // namespace folly