QDMA DPDK and Linux reference drivers 2019.1 release

Co-Authored-By: sujathabanoth-xlnx <sujathabanoth-xlnx@users.noreply.github.com>

QDMA/DPDK/RELEASE.txt

-Release: 2018.3
+Release: 2019.1
 ===============
 
-This release is based on DPDK v17.11.1 containing QDMA poll mode driver and
+This release is based on DPDK v18.11 containing QDMA poll mode driver and
 QDMA test application.
 
-This release includes a patch file for dpdk-pktgen v3.4.7 that extends
+This release includes a patch file for dpdk-pktgen v3.6.1 that extends
 dpdk-pktgen application to handle packets with packet sizes more than 1518 bytes
 and it disables the packet size classification logic in dpdk-pktgen to remove
 application overhead in performance measurement.
 This patch is used for performance testing with dpdk-pktgen application.
 
 The driver is validated against dpdk-pktgen and testpmd applications for API compliance.
-2018.3 driver is not backward compatible with 2018.2 and earlier QDMA IP.
 
 SUPPORTED FEATURES:
 ===================
@@ -23,20 +22,36 @@ SUPPORTED FEATURES:
 - Supports Polling Mode
 - Supports SR-IOV with 4 Physical Functions(PF) and 252 Virtual Functions(VF)
 - Allows Only Privileged/Physical functions to program the contexts and registers
-- Function Level Reset(FLR) Support
 - Mailbox Support
 - Supports Descriptor Prefetch
 - ST H2C to C2H and C2H to H2C loopback support
 - Zero-byte transfer support
 
 2018.3 Features
+---------------
 - Descriptor (8, 16, 32, 64 bytes) bypass support
 - Support for Completion queue descriptors of 64 bytes size
 - Support flexible BAR mapping for QDMA configuration register space
 - Support disabling overflow check in completion ring
 - Indirect programming of FMAP registers
 
+2019.1 Features
+---------------
+- Support DPDK v18.11 LTS
+- Interrupt support for Mailbox events
+- Support Completions in Memory mapped mode
+- Interoperability between Linux driver (as PF/VF) and DPDK driver (as PF/VF)
+- Error monitoring and error logging
+- Driver restructuring to partition QDMA access code such that it can be used across different drivers
+- Device configuration through additional driver APIs
 
-KNOWN ISSUES:
-============
-- Occasionally, C2H completions are not observed when payload and immediate data transfers are combined
\ No newline at end of file
+KNOWN BUGS:
+===========
+- Sometimes completions are not received when C2H PIDX updates are held for 64 descriptors
+
+
+LIMITATIONS:
+===========
+- Big endian systems are not supported
+- All VFs created by a PF should be shut down before terminating the parent PF. This is needed as all the QDMA VF configuration is through this parent PF.
+- For optimal QDMA streaming performance, packet buffers of the descriptor ring should be aligned to at least 256 bytes.
\ No newline at end of file

QDMA/DPDK/docs/git_doc/Makefile

+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+down:
+	wget -O kernel-doc "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/plain/scripts/kernel-doc?h=v4.14.52" && chmod +x kernel-doc
+
+genrst:
+	 mkdir -p $(BUILDDIR)
+	 ./kernel-doc -rst source/pmd_callbacks.h > build/pmd_callbacks.rst
+	 ./kernel-doc -rst source/rte_pmd_qdma.h > build/rte_pmd_qdma.rst
+
+html: down genrst
+	$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+clean:
+	rm -rf $(BUILDDIR)
+	rm -rf kernel-doc

QDMA/DPDK/docs/git_doc/README.txt

+###############################################################################
+
+               Xilinx QDMA DPDK Driver Documentation Generation
+
+###############################################################################
+
+
+1. Installation:
+
+	Xilinx QDMA DPDK driver documenation is designed based on Sphinx.
+	In oprder to generate the documentation, make sure to install the
+	Sphinx software. Details of required packages are available at
+	http://www.sphinx-doc.org/en/master/usage/installation.html
+
+	After installing the required packages, follow the steps below to generate the documentation.
+
+	Go to dpdk/docs/git_doc and run 'make html'
+  	[xilinx@]# make html
+	'build' directory is created. Open 'build/html/index.html' for QDMA DPDK driver documentation
+
+	To remove the generated documentation, run 'make clean'
+	[xilinx@]# make clean
+
+

QDMA/DPDK/docs/git_doc/html/.buildinfo

+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: f5075114aa9834768550066f6676a617
+tags: 645f666f9bcd5a90fca523b33c5a78b7

QDMA/DPDK/docs/git_doc/html/_sources/build.rst.txt

+.. _build_dpdk:
+
+Building QDMA DPDK Software
+===========================
+
+DPDK requires certain packages to be installed on host system.
+For a full list, refer to the official DPDK documentation:
+https://doc.dpdk.org/guides/linux_gsg/sys_reqs.html.
+
+**Note**: If the NUMA library is missing, it should be installed.
+For example:
+
+	For Ubuntu: ``sudo apt-get install libnuma-dev``
+
+	For Red Hat: ``sudo yum install numactl-devel``
+
+Modifying the driver for PCIe device ID
+---------------------------------------
+
+During the PCIe DMA IP customization in Vivado, user can specify a PCIe Device ID.
+This Device ID must be added to the driver to identify the PCIe QDMA device.
+The current driver is designed to recognize the PCIe Device IDs
+that get generated with the PCIe example design when this value has not been modified.
+If the PCIe Device ID is modified during IP customization,
+one needs to modify QDMA PMD to recognize this new ID.
+
+User can also remove PCIe Device IDs that will not be used by the end solution.
+To modify the PCIe Device ID in the driver,
+
+	For PF devices,
+		update ``struct rte_pci_id qdma_pci_id_tbl[]`` inside ``drivers/net/qdma/qdma_ethdev.c``
+	For VF devices,
+		update ``struct rte_pci_id qdma_vf_pci_id_tbl[]`` inside ``drivers/net/qdma/qdma_vf_ethdev.c``
+
+	Also add the device IDs in ``usertools/dpdk-devbind.py`` in ``xilinx_qdma_pf`` for PF device
+	and ``xilinx_qdma_vf`` for VF device as specified in later section.
+
+Once modified, the driver and application must be recompiled.
+
+Xilinx QDMA DPDK Software database structure
+--------------------------------------------
+
+Below Table describes the DPDK software database structure and its contents
+on the Xilinx GitHub https://github.com/Xilinx/dma_ip_drivers, subdirectory QDMA/DPDK.
+
++--------------------------+-------------------------------------------------------------+
+| Directory                | Description                                                 |
++==========================+=============================================================+
+| drivers/net/qdma         | Xilinx QDMA DPDK poll mode driver                           |
++--------------------------+-------------------------------------------------------------+
+| examples/qdma_testapp    | Xilinx CLI based test application for QDMA                  |
++--------------------------+-------------------------------------------------------------+
+| tools/0001-PKTGEN-3.6.1- | This is dpdk-pktgen patch based on dpdk-pktgen v3.6.1.      |
+| Patch-to-add-Jumbo-packet| This patch extends dpdk-pktgen application to handle packets|
+| -support.patch           | with packet sizes more than 1518 bytes and it disables the  |
+|                          | packet size classification logic to remove application      |
+|                          | overhead in performance measurement. This patch is used for |
+|                          | performance testing with dpdk-pktgen application.           |
++--------------------------+-------------------------------------------------------------+
+| RELEASE.txt              | Release Notes                                               |
++--------------------------+-------------------------------------------------------------+
+
+Setup: Download and modifications
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The reference driver code requires DPDK version 18.11.
+Follow the steps below to download the proper version of DPDK and apply
+driver code and test application supplied in the GitHub.
+
+Extract the DPDK driver software database from the Xilinx GitHub to the server where VCU1525
+is installed. Henceforth, this area is referred as <dpdk_sw_database>.
+
+Create a directory for the DPDK download on the server where the VCU1525
+is installed and move to this directory.
+
+::
+
+	mkdir <server_dir>/<dpdk_test_area>
+	cd <server_dir>/<dpdk_test_area>
+	git clone http://dpdk.org/git/dpdk-stable
+	cd dpdk-stable
+	git checkout v18.11
+	cp -r <dpdk_sw_database>/drivers/net/qdma ./drivers/net/
+	cp -r <dpdk_sw_database>/examples/qdma_testapp ./examples/
+
+Additionally, make below changes to the DPDK 18.11 tree to build QDMA driver,
+support 2K queues and populate Xilinx devices for binding.
+
+	1. To build QDMA driver
+
+	a. Add below lines to ``./config/common_base`` in DPDK 18.11 tree
+
+	::
+
+		#
+		#Complie Xilinx QDMA PMD driver
+		#
+		CONFIG_RTE_LIBRTE_QDMA_PMD=y
+		CONFIG_RTE_LIBRTE_QDMA_DEBUG_DRIVER=n
+
+		To enable driver debug logs, set
+		CONFIG_RTE_LIBRTE_QDMA_DEBUG_DRIVER=y
+
+	b. Add below lines to ``drivers/net/Makefile``, where PMDs are added
+
+	::
+
+		DIRS-$(CONFIG_RTE_LIBRTE_QDMA_PMD) += qdma
+
+	c. Add below lines to ``mk/rte.app.mk``, where PMDs are added
+
+	::
+
+		_LDLIBS-$(CONFIG_RTE_LIBRTE_QDMA_PMD) += -lrte_pmd_qdma
+
+	2. To add Xilinx devices for device binding, add below lines to	``./usertools/dpdk-devbind.py`` after cavium_pkx class, where PCI base class for devices are listed.
+
+	::
+
+		xilinx_qdma_pf = {'Class': '05', 'Vendor': '10ee', 'Device': '9011,9111,9211,9311,9014,9114,9214,9314,9018,9118,9218,9318,901f,911f,921f,931f,9021,9121,9221,9321,9024,9124,9224,9324,9028,9128,9228,9328,902f,912f,922f,932f,9031,9131,9231,9331,9034,9134,9234,9334,9038,9138,9238,9338,903f,913f,923f,933f,9041,9141,9241,9341,9044,9144,9244,9344,9048,9148,9248,9348',
+		'SVendor': None, 'SDevice': None}
+		xilinx_qdma_vf = {'Class': '05', 'Vendor': '10ee', 'Device': 'a011,a111,a211,a311,a014,a114,a214,a314,a018,a118,a218,a318,a01f,a11f,a21f,a31f,a021,a121,a221,a321,a024,a124,a224,a324,a028,a128,a228,a328,a02f,a12f,a22f,a32f,a031,a131,a231,a331,a034,a134,a234,a334,a038,a138,a238,a338,a03f,a13f,a23f,a33f,a041,a141,a241,a341,a044,a144,a244,a344,a048,a148,a248,a348',
+		'SVendor': None, 'SDevice': None}
+
+	Update entries in network devices class in ``./usertools/dpdk-devbind.py`` to add Xilinx devices
+
+	::
+
+		network_devices = [network_class, cavium_pkx, xilinx_qdma_pf, xilinx_qdma_vf]
+
+	3. To support 2K queues and 256 PCIe functions, update below configurations	in ``./config/common_base``
+
+	::
+
+		CONFIG_RTE_MAX_MEMZONE=7680
+		CONFIG_RTE_MAX_ETHPORTS=256
+		CONFIG_RTE_MAX_QUEUES_PER_PORT=2048
+
+
+Setup: Host system
+^^^^^^^^^^^^^^^^^^
+
+DPDK requires that hugepages are setup on the server.
+The following modifications must be made to the ``/boot/grub/grub.cfg`` on the host system
+
+- Add hugepages for DPDK
+
+	Add following parameter to ``/etc/default/grub file``
+
+	::
+
+		GRUB_CMDLINE_LINUX="default_hugepagesz=1GB hugepagesz=1G hugepages=20"
+
+	| This example adds 20 1GB hugepages, which are required to support 2048 queues, with descriptor ring of 1024 entries and each descriptor buffer length of 4KB.
+	| The number of hugepages required should be changed if the above configuration (queues, ring size, buffer size) changes.
+
+- Enable IOMMU for VM testing
+
+	Update ``/etc/default/grub`` file as below.
+
+	::
+
+		GRUB_CMDLINE_LINUX="default_hugepagesz=1GB hugepagesz=1G hugepages=20 iommu=pt intel_iommu=on"
+
+Execute the following command to modify the ``/boot/grub/grub.cfg`` with the configuration set in the above steps and permanently add them to the kernel command line.
+
+	::
+
+		update-grub
+
+Reboot host system after making the above modifications.
+
+Setup: Make Commands
+^^^^^^^^^^^^^^^^^^^^
+
+* Compile DPDK & QDMA driver
+
+	Execute the following to compile the driver.
+
+	::
+
+		cd <server_dir>/<dpdk_test_area>/dpdk-stable
+		make config T=x86_64-native-linuxapp-gcc install
+
+	- In the make output, verify that the QDMA files are being built.
+	  Below figure shows the QDMA files that are built as part of make.
+
+	  .. image:: images/make_output.png
+
+	  The following should appear when make completes
+
+	  ::
+
+		Build complete [x86_64-native-linuxapp-gcc]
+
+	- Verify that ``librte_pmd_qdma.a`` is installed in ``./x86_64-native-linuxapp-gcc/lib`` directory.
+
+
+	Additionally, for memory mapped mode, BRAM size can be configured with ``make`` command.
+	Default BRAM size is set to 512KB in the driver makefile.
+
+	::
+
+		make config T=x86_64-native-linuxapp-gcc BRAM_SIZE=<BRAM size in bytes in decimal> install
+
+* Compile Test application
+
+	Change to root user and compile the application
+
+	::
+
+		sudo su
+		cd examples/qdma_testapp
+		make RTE_SDK=`pwd`/../.. RTE_TARGET=x86_64-native-linuxapp-gcc
+
+	The following should appear when make completes
+
+	::
+
+		INSTALL-MAP qdma_testapp.map
+
+	Additionally, for memory mapped mode, BRAM size can be configured with make command.
+	Default BRAM size is set to 512KB in the driver makefile.
+
+	::
+
+		make BRAM_SIZE=<BRAM size in bytes in decimal> RTE_SDK=`pwd`/../.. RTE_TARGET=x86_64-native-linuxapp-gcc
+
+
+If any of above steps are missed or require code modifications,
+perform ``make clean`` before required modifications and re-building.
+For driver related modifications, perform ``make clean``
+from inside ``x86_64-native-linuxapp-gcc`` directory.
+

QDMA/DPDK/docs/git_doc/html/_sources/dev-guide.rst.txt

+****************
+Developers Guide
+****************
+
+.. toctree::
+   :maxdepth: 1
+
+   driver-design.rst
+   qdma_usecases.rst

QDMA/DPDK/docs/git_doc/html/_sources/dpdk-callbacks.rst.txt

+******************
+DPDK Callback APIs
+******************
+
+.. include:: ../build/pmd_callbacks.rst
+

QDMA/DPDK/docs/git_doc/html/_sources/driver-design.rst.txt

+Driver Design
+*************
+
+QDMA DPDK driver is implemented as an Ethernet poll mode driver in DPDK v18.11.
+It supports both streaming (network) and memory (compute) interface to QDMA.
+Below diagram gives a high level system overview of the QDMA DPDK driver.
+
+.. image:: /images/dpdk_overview.png
+   :align: center
+
+Driver interfaces and callbacks
+===============================
+
+The Driver registers two driver interfaces (``net_qdma`` for PFs and ``net_qdma_vf`` for VFs) with DPDK RTE.
+Below ``eth_dev_ops`` callbacks are implemented by both driver interfaces of the DPDK driver.
+
+::
+
+	static struct eth_dev_ops qdma_eth_dev_ops = {
+		.dev_configure		= qdma_dev_configure,
+		.dev_infos_get		= qdma_dev_infos_get,
+		.dev_start		= qdma_dev_start,
+		.dev_stop		= qdma_dev_stop,
+		.dev_close		= qdma_dev_close,
+		.link_update		= qdma_dev_link_update,
+		.rx_queue_setup		= qdma_dev_rx_queue_setup,
+		.tx_queue_setup		= qdma_dev_tx_queue_setup,
+		.rx_queue_release	= qdma_dev_rx_queue_release,
+		.tx_queue_release	= qdma_dev_tx_queue_release,
+		.rx_queue_start		= qdma_dev_rx_queue_start,
+		.rx_queue_stop		= qdma_dev_rx_queue_stop,
+		.tx_queue_start		= qdma_dev_tx_queue_start,
+		.tx_queue_stop		= qdma_dev_tx_queue_stop,
+		.stats_get		= qdma_dev_stats_get,
+	};
+
+Below sequence diagram depicts the application call flow to these driver callbacks and their high level operations.
+
+.. image:: images/dpdk_call_flow.png
+
+Implementation details of the callback APIs is described further in
+
+.. toctree::
+   :maxdepth: 1
+
+   dpdk-callbacks.rst
+
+Additional interfaces are exported by the driver for QDMA configuration. These new interfaces are described in
+
+.. toctree::
+   :maxdepth: 1
+
+   qdma_pmd_export.rst
+
+Resource Management
+===================
+
+.. toctree::
+   :maxdepth: 1
+
+   qdma_resmgmt.rst
+
+Mailbox Communication
+=====================
+
+.. toctree::
+   :maxdepth: 1
+
+   qdma_mailbox.rst
+
+QDMA Hardware Error Monitoring
+==============================
+
+There are various causes for errors in QDMA IP e.g. descriptor errors, DMA engine errors, etc.
+If the errors are enabled by SW, QDMA logs the status of error in the respective global status registers.
+
+DPDK driver executes a poll thread ``qdma_check_errors()`` every 1 second to check the error status and log the type of HW error occured on the console.
+
+DPDK Driver debug
+=================
+DPDK driver provides API ``rte_pmd_qdma_xdebug()`` to dump debug information. Below debug information can be dumped using this API
+
+- Dump QDMA registers
+- Dump queue contexts for the given queue id
+- Dump software information for the given queue id
+- Dump the specified descriptor ring of the given queue id
+
+User application can call this API to dump the required debug information.

QDMA/DPDK/docs/git_doc/html/_sources/features.rst.txt

+QDMA Features
+=============
+
+QDMA DPDK Driver supports the following list of features
+
+Hardware Features
+-----------------
+* SRIOV with 4 Physical Functions and 252 Virtual Functions
+* Memory Mapped (MM) and Stream (ST) interfaces per queue
+* 2048 queues sets
+
+   - 2048 H2C (host-to-card) Descriptor rings
+   - 2048 C2H (card-to-host) Descriptor rings
+   - 2048 Completion rings
+
+* Mailbox communication between PF and VF driver
+* Interrupt support for Mailbox events
+* HW Error reporting
+* Zero byte transfers
+* Immediate data transfers
+* Descriptor bypass (8, 16, 32, 64 descriptor sizes) support
+* Descriptor Prefetching
+* Streaming C2H completion entry coalescing
+* Disabling overflow check in completion ring
+* Streaming H2C to C2H and C2H to H2C loopback support
+* Dynamic queue configuration
+* Completion ring descriptors of 8, 16, 32, 64 bytes sizes
+* Flexible BAR mapping for QDMA configuration register space
+* ECC support
+* Completions in Memory mapped mode
+
+For details on Hardware Features refer to QDMA_Product_Guide_.
+
+.. _QDMA_Product_Guide: https://www.xilinx.com/support/documentation/ip_documentation/qdma/v3_0/pg302-qdma.pdf
+
+Software Features
+-----------------
+
+* Support DPDK v18.11 LTS
+* Support driver binding to igb_uio and vfio-pci
+* Support device arguments (module parameters) for device level configuration
+* Allows only Privileged Physical Functions to program the contexts and configuration registers
+* Device configuration through additional driver APIs
+* Interoperability between Linux driver (as PF/VF) and DPDK driver (as PF/VF)
+* SW versioning

QDMA/DPDK/docs/git_doc/html/_sources/index.rst.txt

+#################################
+Xilinx QDMA DPDK Poll Mode Driver
+#################################
+
+The Xilinx PCI Express Multi Queue DMA (QDMA) IP provides high-performance
+direct memory access (DMA) via PCI Express.
+
+Xilinx provides a DPDK poll mode driver based on DPDK v18.11 that runs on a
+PCI Express root port host PC to interact with the QDMA endpoint IP via PCI Express.
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Table of Contents
+
+   features.rst
+   system-requirements.rst
+   build.rst
+   userguide.rst
+   dev-guide.rst
+   perf.rst
+

QDMA/DPDK/docs/git_doc/html/_sources/perf.rst.txt

+QDMA DPDK Performance
+---------------------
+Refer to QDMA_Performance_Answer_Record_ for detailed performance results and test environment.
+
+.. _QDMA_Performance_Answer_Record: https://www.xilinx.com/support/answers/71453.html

QDMA/DPDK/docs/git_doc/html/_sources/qdma_mailbox.rst.txt

+*************
+QDMA Mailbox
+*************
+
+The QDMA Subsystem for PCIe provides an optional feature to support the Single Root I/O
+Virtualization (SR-IOV). SR-IOV classifies the functions as:
+
+- Physical Functions (PF) :
+	Full featured PCIe functions which include SR-IOV capabilities among others.
+
+- Virtual Functions (VF):
+	PCIe functions featuring configuration space with Base Address
+	Registers (BARs) but lacking the full configuration resources and are controlled by the PF
+	configuration. The main role of the VF is data transfer.
+
+VF can communicate to a PF through mailbox. Each function implements one 128B inbox and 128B outbox message buffer.
+These mailboxes are accessible to the driver via PCIe BAR of its own function.
+HW also provides ability to interrupt the driver for an incoming mailbox message to a PCIe function.
+For further details on the mailbox internals and mailbox registers, refer to QDMA_Product_Guide_.
+
+.. _QDMA_Product_Guide: https://www.xilinx.com/support/documentation/ip_documentation/qdma/v3_0/pg302-qdma.pdf
+
+Physical function (PF) is privileged with full access to QDMA registers and resources, but VFs updates only data handling registers and interrupts.
+VF drivers must communicate with the driver attached to the PF through the mailbox for configuration and resource allocation.
+The PF and VF drivers define the message formatting to be exchanged between the driver.
+
+The driver enables mailbox interrupt and registers a Rx handler for notification of incoming mailbox message.
+It also creates a Tx thread to send a message and poll for its response from the peer driver.
+
+Following are the list of messages supported between PF and VF driver
+
+::
+
+	enum mbox_msg_op {
+		/** @MBOX_OP_BYE: vf offline */
+		MBOX_OP_BYE,
+		/** @MBOX_OP_HELLO: vf online */
+		MBOX_OP_HELLO,
+		/** @: FMAP programming request */
+		MBOX_OP_FMAP,
+		/** @MBOX_OP_CSR: global CSR registers request */
+		MBOX_OP_CSR,
+		/** @MBOX_OP_QREQ: request queues */
+		MBOX_OP_QREQ,
+		/** @MBOX_OP_QNOTIFY_ADD: notify of queue addition */
+		MBOX_OP_QNOTIFY_ADD,
+		/** @MBOX_OP_QNOTIFY_DEL: notify of queue deletion */
+		MBOX_OP_QNOTIFY_DEL,
+		/** @MBOX_OP_QCTXT_WRT: queue context write */
+		MBOX_OP_QCTXT_WRT,
+		/** @MBOX_OP_QCTXT_RD: queue context read */
+		MBOX_OP_QCTXT_RD,
+		/** @MBOX_OP_QCTXT_CLR: queue context clear */
+		MBOX_OP_QCTXT_CLR,
+		/** @MBOX_OP_QCTXT_INV: queue context invalidate */
+		MBOX_OP_QCTXT_INV,
+		/** @MBOX_OP_INTR_CTXT_WRT: interrupt context write */
+		MBOX_OP_INTR_CTXT_WRT,
+		/** @MBOX_OP_INTR_CTXT_RD: interrupt context read */
+		MBOX_OP_INTR_CTXT_RD,
+		/** @MBOX_OP_INTR_CTXT_CLR: interrupt context clear */
+		MBOX_OP_INTR_CTXT_CLR,
+		/** @MBOX_OP_INTR_CTXT_INV: interrupt context invalidate */
+		MBOX_OP_INTR_CTXT_INV,
+
+		/** @MBOX_OP_HELLO_RESP: response to @MBOX_OP_HELLO */
+		MBOX_OP_HELLO_RESP = 0x81,
+		/** @MBOX_OP_FMAP_RESP: response to @MBOX_OP_FMAP */
+		MBOX_OP_FMAP_RESP,
+		/** @MBOX_OP_CSR_RESP: response to @MBOX_OP_CSR */
+		MBOX_OP_CSR_RESP,
+		/** @MBOX_OP_QREQ_RESP: response to @MBOX_OP_QREQ */
+		MBOX_OP_QREQ_RESP,
+		/** @MBOX_OP_QADD: notify of queue addition */
+		MBOX_OP_QNOTIFY_ADD_RESP,
+		/** @MBOX_OP_QNOTIFY_DEL: notify of queue deletion */
+		MBOX_OP_QNOTIFY_DEL_RESP,
+		/** @MBOX_OP_QCTXT_WRT_RESP: response to @MBOX_OP_QCTXT_WRT */
+		MBOX_OP_QCTXT_WRT_RESP,
+		/** @MBOX_OP_QCTXT_RD_RESP: response to @MBOX_OP_QCTXT_RD */
+		MBOX_OP_QCTXT_RD_RESP,
+		/** @MBOX_OP_QCTXT_CLR_RESP: response to @MBOX_OP_QCTXT_CLR */
+		MBOX_OP_QCTXT_CLR_RESP,
+		/** @MBOX_OP_QCTXT_INV_RESP: response to @MBOX_OP_QCTXT_INV */
+		MBOX_OP_QCTXT_INV_RESP,
+		/** @MBOX_OP_INTR_CTXT_WRT_RESP: response to @MBOX_OP_INTR_CTXT_WRT */
+		MBOX_OP_INTR_CTXT_WRT_RESP,
+		/** @MBOX_OP_INTR_CTXT_RD_RESP: response to @MBOX_OP_INTR_CTXT_RD */
+		MBOX_OP_INTR_CTXT_RD_RESP,
+		/** @MBOX_OP_INTR_CTXT_CLR_RESP: response to @MBOX_OP_INTR_CTXT_CLR */
+		MBOX_OP_INTR_CTXT_CLR_RESP,
+		/** @MBOX_OP_INTR_CTXT_INV_RESP: response to @MBOX_OP_INTR_CTXT_INV */
+		MBOX_OP_INTR_CTXT_INV_RESP,
+
+		/** @MBOX_OP_MAX: total mbox opcodes*/
+		MBOX_OP_MAX
+	};
+
+
+The following sequence diagrams shows the mailbox communication between PF and VF for VF configuration.
+
+.. image:: /images/mailbox_communication.png
+   :align: center
\ No newline at end of file

QDMA/DPDK/docs/git_doc/html/_sources/qdma_pmd_export.rst.txt

+***************************
+QDMA DPDK PMD Exported APIs
+***************************
+
+.. include:: ../build/rte_pmd_qdma.rst
+

QDMA/DPDK/docs/git_doc/html/_sources/qdma_resmgmt.rst.txt

+******************************
+QDMA Queue Resource Management
+******************************
+
+QDMA IP supports 2K queues. QDMA Resource Manager defines the strategy to allocate the queues across the available PFs and VFs.
+Resource Manager maintains a global resource linked list in the driver. It creates a linked list of nodes for each PCIe device (PCIe bus) it manages.
+Each device (bus) node in the Resource Manager list is initialized with queue base and number of queues to manage on that device.
+Given the request for a number of queues for a PCIe function on a PCIe device, Resource Manager finds the best fit available
+queue base with contiguous requested number of queues on that PCIe device.
+Resource Manager manages queue base allocation for all the PCIe devices and PCIe functions registered with it by the driver.
+
+Each device node in the global resource linked list maintains a "free list" with all the queues initially assigned to it, and a "device list" initialized to NULL.
+When a PCIe function is initialized and requests the required queues, Resource Manager finds the best fit available queue base
+with contiguous requested number of queues from the "free list" of the requested device node. If found, it creates a PCIe function node in the "device list"
+and assigns the queue base and requested queues to this PCIe function node and removes these queues from the "free list".
+
+Driver can request a specific queue base for a PCIe function and if the requested queue base is available,
+Resource Manager honours it, else, allocates the queues at a base determined by the best fit algorithm.
+
+Host system can have drivers in user space or kernel space to manage the devices connected.
+E.g. Linux QDMA Driver is a kernel mode driver and DPDK PMD is a User Space driver.
+Each driver maintains its own resouce manager with queue base and total queues to manage specified for each device it manages.
+Below, we demonstrate few examples on how Resource Manager is used in different combinations of drivers managing QDMA devices
+
+Single device managed by Single driver
+======================================
+
+.. image:: /images/resmgmt_sdev_sdri.PNG
+   :align: center
+
+In this scenario, one PCIe QDMA device is connected to the Host System and at any given time either the Linux kernel driver or the user space DPDK driver is
+managing the device. There is a single Resource Manager managing all the queues of the device and assigning the queues to functions based
+on the request from each function.
+
+Single device managed by multiple drivers
+==========================================
+
+.. image:: /images/resmgmt_sdev_mdri.PNG
+   :align: center
+
+In this scenario, one device is connected to the Host System. Linux kernel driver and user space DPDK driver are loaded to manage different PCIe functions of the device.
+Each driver will create its own Resource Manager using a queue base and total queues to manage the functions binded to them.
+Each Resource Manager has its own pool of queues and the PCIe functions get their queues resourced from this pool.
+
+Multiple devices on single Host managed by Single driver
+========================================================
+
+.. image:: /images/resmgmt_mdev_sdri.PNG
+   :align: center
+
+In this scenario, more than one devices are connected to the Host System.
+Either user space DPDK driver or Linux kernel driver is loaded to manage these devices.
+Driver will instantiate Resource Manager and create one node per PCIe device conected,
+resourcing each PCIe device (node) with the queue base and total number of queues to manage.
+PCIe functions corresponding to each PCIe device are allocated queues from the respective device nodes.
+

QDMA/DPDK/docs/git_doc/html/_sources/system-requirements.rst.txt

+.. _sys_req:
+
+System Requirements
+===================
+
+Xilinx Accelerator Card
+-----------------------
+
+1. VCU1525
+2. U200
+
+Host System
+-----------
+
+x86_64 host system with at least one Gen 3 x16 PCIe slot and minimum 32GB RAM
+on same CPU node for 2K queues.
+For VM testing, host system must support virtualization and it must be enabled in the BIOS.
+
+
+Host System Configuration
+-------------------------
+
+QDMA DPDK driver latest release is verified on following host system configuration for PF and VF functionality
+
++--------------------------+-------------------------------------------------------------+
+| Host System              | Configuration Details                                       |
++==========================+=============================================================+
+| Operating System         | Ubuntu 16.04.3 LTS, Ubuntu 18.04.1 LTS                      |
++--------------------------+-------------------------------------------------------------+
+| Linux Kernel             | 4.4.0-93-generic                                            |
++--------------------------+-------------------------------------------------------------+
+| CPU Cores                | 32                                                          |
++--------------------------+-------------------------------------------------------------+
+| RAM                      | 64GB on local NUMA node                                     |
++--------------------------+-------------------------------------------------------------+
+| Hypervisor               | KVM                                                         |
++--------------------------+-------------------------------------------------------------+
+| Qemu Version             | QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.15)|
++--------------------------+-------------------------------------------------------------+
+| Vivado Version           | 2019.1                                                      |
++--------------------------+-------------------------------------------------------------+
+
+.. _guest_system_cfg:
+
+Guest System Configuration
+--------------------------
+
+QDMA DPDK driver latest release is verified on the following guest system configuration for VF functionality
+
+========================= ==================================
+Guest System(VM)          Configuration Details
+========================= ==================================
+Operating System          Ubuntu 18.04 LTS
+Linux Kernel              4.15.1-20-generic
+RAM 					  4GB
+Cores              		  4
+========================= ==================================
+
+

QDMA/DPDK/docs/git_doc/html/_static/css/badge_only.css

+.fa:before{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:before,.clearfix:after{display:table;content:""}.clearfix:after{clear:both}@font-face{font-family:FontAwesome;font-weight:normal;font-style:normal;src:url("../font/fontawesome_webfont.eot");src:url("../font/fontawesome_webfont.eot?#iefix") format("embedded-opentype"),url("../font/fontawesome_webfont.woff") format("woff"),url("../font/fontawesome_webfont.ttf") format("truetype"),url("../font/fontawesome_webfont.svg#FontAwesome") format("svg")}.fa:before{display:inline-block;font-family:FontAwesome;font-style:normal;font-weight:normal;line-height:1;text-decoration:inherit}a .fa{display:inline-block;text-decoration:inherit}li .fa{display:inline-block}li .fa-large:before,li .fa-large:before{width:1.875em}ul.fas{list-style-type:none;margin-left:2em;text-indent:-0.8em}ul.fas li .fa{width:0.8em}ul.fas li .fa-large:before,ul.fas li .fa-large:before{vertical-align:baseline}.fa-book:before{content:""}.icon-book:before{content:""}.fa-caret-down:before{content:""}.icon-caret-down:before{content:""}.fa-caret-up:before{content:""}.icon-caret-up:before{content:""}.fa-caret-left:before{content:""}.icon-caret-left:before{content:""}.fa-caret-right:before{content:""}.icon-caret-right:before{content:""}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;border-top:solid 10px #343131;font-family:"Lato","proxima-nova","Helvetica Neue",Arial,sans-serif;z-index:400}.rst-versions a{color:#2980B9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27AE60;*zoom:1}.rst-versions .rst-current-version:before,.rst-versions .rst-current-version:after{display:table;content:""}.rst-versions .rst-current-version:after{clear:both}.rst-versions .rst-current-version .fa{color:#fcfcfc}.rst-versions .rst-current-version .fa-book{float:left}.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#E74C3C;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#F1C40F;color:#000}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:gray;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:solid 1px #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px}.rst-versions.rst-badge .icon-book{float:none}.rst-versions.rst-badge .fa-book{float:none}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book{float:left}.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge .rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width: 768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}img{width:100%;height:auto}}
+/*# sourceMappingURL=badge_only.css.map */

QDMA/DPDK/docs/git_doc/html/_static/doctools.js

+/*
+ * doctools.js
+ * ~~~~~~~~~~~
+ *
+ * Sphinx JavaScript utilities for all documentation.
+ *
+ * :copyright: Copyright 2007-2018 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+/**
+ * select a different prefix for underscore
+ */
+$u = _.noConflict();
+
+/**
+ * make the code below compatible with browsers without
+ * an installed firebug like debugger
+if (!window.console || !console.firebug) {
+  var names = ["log", "debug", "info", "warn", "error", "assert", "dir",
+    "dirxml", "group", "groupEnd", "time", "timeEnd", "count", "trace",
+    "profile", "profileEnd"];
+  window.console = {};
+  for (var i = 0; i < names.length; ++i)
+    window.console[names[i]] = function() {};
+}
+ */
+
+/**
+ * small helper function to urldecode strings
+ */
+jQuery.urldecode = function(x) {
+  return decodeURIComponent(x).replace(/\+/g, ' ');
+};
+
+/**
+ * small helper function to urlencode strings
+ */
+jQuery.urlencode = encodeURIComponent;
+
+/**
+ * This function returns the parsed url parameters of the
+ * current request. Multiple values per key are supported,
+ * it will always return arrays of strings for the value parts.
+ */
+jQuery.getQueryParameters = function(s) {
+  if (typeof s === 'undefined')
+    s = document.location.search;
+  var parts = s.substr(s.indexOf('?') + 1).split('&');
+  var result = {};
+  for (var i = 0; i < parts.length; i++) {
+    var tmp = parts[i].split('=', 2);
+    var key = jQuery.urldecode(tmp[0]);
+    var value = jQuery.urldecode(tmp[1]);
+    if (key in result)
+      result[key].push(value);
+    else
+      result[key] = [value];
+  }
+  return result;
+};
+
+/**
+ * highlight a given string on a jquery object by wrapping it in
+ * span elements with the given class name.
+ */
+jQuery.fn.highlightText = function(text, className) {
+  function highlight(node, addItems) {
+    if (node.nodeType === 3) {
+      var val = node.nodeValue;
+      var pos = val.toLowerCase().indexOf(text);
+      if (pos >= 0 &&
+          !jQuery(node.parentNode).hasClass(className) &&
+          !jQuery(node.parentNode).hasClass("nohighlight")) {
+        var span;
+        var isInSVG = jQuery(node).closest("body, svg, foreignObject").is("svg");
+        if (isInSVG) {
+          span = document.createElementNS("http://www.w3.org/2000/svg", "tspan");
+        } else {
+          span = document.createElement("span");
+          span.className = className;
+        }
+        span.appendChild(document.createTextNode(val.substr(pos, text.length)));
+        node.parentNode.insertBefore(span, node.parentNode.insertBefore(
+          document.createTextNode(val.substr(pos + text.length)),
+          node.nextSibling));
+        node.nodeValue = val.substr(0, pos);
+        if (isInSVG) {
+          var bbox = span.getBBox();
+          var rect = document.createElementNS("http://www.w3.org/2000/svg", "rect");
+       	  rect.x.baseVal.value = bbox.x;
+          rect.y.baseVal.value = bbox.y;
+          rect.width.baseVal.value = bbox.width;
+          rect.height.baseVal.value = bbox.height;
+          rect.setAttribute('class', className);
+          var parentOfText = node.parentNode.parentNode;
+          addItems.push({
+              "parent": node.parentNode,
+              "target": rect});
+        }
+      }
+    }
+    else if (!jQuery(node).is("button, select, textarea")) {
+      jQuery.each(node.childNodes, function() {
+        highlight(this, addItems);
+      });
+    }
+  }
+  var addItems = [];
+  var result = this.each(function() {
+    highlight(this, addItems);
+  });
+  for (var i = 0; i < addItems.length; ++i) {
+    jQuery(addItems[i].parent).before(addItems[i].target);
+  }
+  return result;
+};
+
+/*
+ * backward compatibility for jQuery.browser
+ * This will be supported until firefox bug is fixed.
+ */
+if (!jQuery.browser) {
+  jQuery.uaMatch = function(ua) {
+    ua = ua.toLowerCase();
+
+    var match = /(chrome)[ \/]([\w.]+)/.exec(ua) ||
+      /(webkit)[ \/]([\w.]+)/.exec(ua) ||
+      /(opera)(?:.*version|)[ \/]([\w.]+)/.exec(ua) ||
+      /(msie) ([\w.]+)/.exec(ua) ||
+      ua.indexOf("compatible") < 0 && /(mozilla)(?:.*? rv:([\w.]+)|)/.exec(ua) ||
+      [];
+
+    return {
+      browser: match[ 1 ] || "",
+      version: match[ 2 ] || "0"
+    };
+  };
+  jQuery.browser = {};
+  jQuery.browser[jQuery.uaMatch(navigator.userAgent).browser] = true;
+}
+
+/**
+ * Small JavaScript module for the documentation.
+ */
+var Documentation = {
+
+  init : function() {
+    this.fixFirefoxAnchorBug();
+    this.highlightSearchWords();
+    this.initIndexTable();
+    if (DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) {
+      this.initOnKeyListeners();
+    }
+  },
+
+  /**
+   * i18n support
+   */
+  TRANSLATIONS : {},
+  PLURAL_EXPR : function(n) { return n === 1 ? 0 : 1; },
+  LOCALE : 'unknown',
+
+  // gettext and ngettext don't access this so that the functions
+  // can safely bound to a different name (_ = Documentation.gettext)
+  gettext : function(string) {
+    var translated = Documentation.TRANSLATIONS[string];
+    if (typeof translated === 'undefined')
+      return string;
+    return (typeof translated === 'string') ? translated : translated[0];
+  },
+
+  ngettext : function(singular, plural, n) {
+    var translated = Documentation.TRANSLATIONS[singular];
+    if (typeof translated === 'undefined')
+      return (n == 1) ? singular : plural;
+    return translated[Documentation.PLURALEXPR(n)];
+  },
+
+  addTranslations : function(catalog) {
+    for (var key in catalog.messages)
+      this.TRANSLATIONS[key] = catalog.messages[key];
+    this.PLURAL_EXPR = new Function('n', 'return +(' + catalog.plural_expr + ')');
+    this.LOCALE = catalog.locale;
+  },
+
+  /**
+   * add context elements like header anchor links
+   */
+  addContextElements : function() {
+    $('div[id] > :header:first').each(function() {
+      $('<a class="headerlink">\u00B6</a>').
+      attr('href', '#' + this.id).
+      attr('title', _('Permalink to this headline')).
+      appendTo(this);
+    });
+    $('dt[id]').each(function() {
+      $('<a class="headerlink">\u00B6</a>').
+      attr('href', '#' + this.id).
+      attr('title', _('Permalink to this definition')).
+      appendTo(this);
+    });
+  },
+
+  /**
+   * workaround a firefox stupidity
+   * see: https://bugzilla.mozilla.org/show_bug.cgi?id=645075
+   */
+  fixFirefoxAnchorBug : function() {
+    if (document.location.hash && $.browser.mozilla)
+      window.setTimeout(function() {
+        document.location.href += '';
+      }, 10);
+  },
+
+  /**
+   * highlight the search words provided in the url in the text
+   */
+  highlightSearchWords : function() {
+    var params = $.getQueryParameters();
+    var terms = (params.highlight) ? params.highlight[0].split(/\s+/) : [];
+    if (terms.length) {
+      var body = $('div.body');
+      if (!body.length) {
+        body = $('body');
+      }
+      window.setTimeout(function() {
+        $.each(terms, function() {
+          body.highlightText(this.toLowerCase(), 'highlighted');
+        });
+      }, 10);
+      $('<p class="highlight-link"><a href="javascript:Documentation.' +
+        'hideSearchWords()">' + _('Hide Search Matches') + '</a></p>')
+          .appendTo($('#searchbox'));
+    }
+  },
+
+  /**
+   * init the domain index toggle buttons
+   */
+  initIndexTable : function() {
+    var togglers = $('img.toggler').click(function() {
+      var src = $(this).attr('src');
+      var idnum = $(this).attr('id').substr(7);
+      $('tr.cg-' + idnum).toggle();
+      if (src.substr(-9) === 'minus.png')
+        $(this).attr('src', src.substr(0, src.length-9) + 'plus.png');
+      else
+        $(this).attr('src', src.substr(0, src.length-8) + 'minus.png');
+    }).css('display', '');
+    if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) {
+        togglers.click();
+    }
+  },
+
+  /**
+   * helper function to hide the search marks again
+   */
+  hideSearchWords : function() {
+    $('#searchbox .highlight-link').fadeOut(300);
+    $('span.highlighted').removeClass('highlighted');
+  },
+
+  /**
+   * make the url absolute
+   */
+  makeURL : function(relativeURL) {
+    return DOCUMENTATION_OPTIONS.URL_ROOT + '/' + relativeURL;
+  },
+
+  /**
+   * get the current relative url
+   */
+  getCurrentURL : function() {
+    var path = document.location.pathname;
+    var parts = path.split(/\//);
+    $.each(DOCUMENTATION_OPTIONS.URL_ROOT.split(/\//), function() {
+      if (this === '..')
+        parts.pop();
+    });
+    var url = parts.join('/');
+    return path.substring(url.lastIndexOf('/') + 1, path.length - 1);
+  },
+
+  initOnKeyListeners: function() {
+    $(document).keyup(function(event) {
+      var activeElementType = document.activeElement.tagName;
+      // don't navigate when in search box or textarea
+      if (activeElementType !== 'TEXTAREA' && activeElementType !== 'INPUT' && activeElementType !== 'SELECT') {
+        switch (event.keyCode) {
+          case 37: // left
+            var prevHref = $('link[rel="prev"]').prop('href');
+            if (prevHref) {
+              window.location.href = prevHref;
+              return false;
+            }
+          case 39: // right
+            var nextHref = $('link[rel="next"]').prop('href');
+            if (nextHref) {
+              window.location.href = nextHref;
+              return false;
+            }
+        }
+      }
+    });
+  }
+};
+
+// quick alias for translations
+_ = Documentation.gettext;
+
+$(document).ready(function() {
+  Documentation.init();
+});

QDMA/DPDK/docs/git_doc/html/_static/graphviz.css

+/*
+ * graphviz.css
+ * ~~~~~~~~~~~~
+ *
+ * Sphinx stylesheet -- graphviz extension.
+ *
+ * :copyright: Copyright 2007-2018 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+img.graphviz {
+    border: 0;
+    max-width: 100%;
+}
+
+object.graphviz {
+    max-width: 100%;
+}

QDMA/DPDK/docs/git_doc/html/_static/js/theme.js

+require=(function e(t,n,r){function s(o,u){if(!n[o]){if(!t[o]){var a=typeof require=="function"&&require;if(!u&&a)return a(o,!0);if(i)return i(o,!0);var f=new Error("Cannot find module '"+o+"'");throw f.code="MODULE_NOT_FOUND",f}var l=n[o]={exports:{}};t[o][0].call(l.exports,function(e){var n=t[o][1][e];return s(n?n:e)},l,l.exports,e,t,n,r)}return n[o].exports}var i=typeof require=="function"&&require;for(var o=0;o<r.length;o++)s(r[o]);return s})({"sphinx-rtd-theme":[function(require,module,exports){
+var jQuery = (typeof(window) != 'undefined') ? window.jQuery : require('jquery');
+
+// Sphinx theme nav state
+function ThemeNav () {
+
+    var nav = {
+        navBar: null,
+        win: null,
+        winScroll: false,
+        winResize: false,
+        linkScroll: false,
+        winPosition: 0,
+        winHeight: null,
+        docHeight: null,
+        isRunning: null
+    };
+
+    nav.enable = function () {
+        var self = this;
+
+        jQuery(function ($) {
+            self.init($);
+
+            self.reset();
+            self.win.on('hashchange', self.reset);
+
+            // Set scroll monitor
+            self.win.on('scroll', function () {
+                if (!self.linkScroll) {
+                    self.winScroll = true;
+                }
+            });
+            setInterval(function () { if (self.winScroll) self.onScroll(); }, 25);
+
+            // Set resize monitor
+            self.win.on('resize', function () {
+                self.winResize = true;
+            });
+            setInterval(function () { if (self.winResize) self.onResize(); }, 25);
+            self.onResize();
+        });
+    };
+
+    nav.init = function ($) {
+        var doc = $(document),
+            self = this;
+
+        this.navBar = $('div.wy-side-scroll:first');
+        this.win = $(window);
+
+        // Set up javascript UX bits
+        $(document)
+            // Shift nav in mobile when clicking the menu.
+            .on('click', "[data-toggle='wy-nav-top']", function() {
+                $("[data-toggle='wy-nav-shift']").toggleClass("shift");
+                $("[data-toggle='rst-versions']").toggleClass("shift");
+            })
+
+            // Nav menu link click operations
+            .on('click', ".wy-menu-vertical .current ul li a", function() {
+                var target = $(this);
+                // Close menu when you click a link.
+                $("[data-toggle='wy-nav-shift']").removeClass("shift");
+                $("[data-toggle='rst-versions']").toggleClass("shift");
+                // Handle dynamic display of l3 and l4 nav lists
+                self.toggleCurrent(target);
+                self.hashChange();
+            })
+            .on('click', "[data-toggle='rst-current-version']", function() {
+                $("[data-toggle='rst-versions']").toggleClass("shift-up");
+            })
+
+        // Make tables responsive
+        $("table.docutils:not(.field-list)")
+            .wrap("<div class='wy-table-responsive'></div>");
+
+        // Add expand links to all parents of nested ul
+        $('.wy-menu-vertical ul').not('.simple').siblings('a').each(function () {
+            var link = $(this);
+                expand = $('<span class="toctree-expand"></span>');
+            expand.on('click', function (ev) {
+                self.toggleCurrent(link);
+                ev.stopPropagation();
+                return false;
+            });
+            link.prepend(expand);
+        });
+    };
+
+    nav.reset = function () {
+        // Get anchor from URL and open up nested nav
+        var anchor = encodeURI(window.location.hash);
+        if (anchor) {
+            try {
+                var link = $('.wy-menu-vertical')
+                    .find('[href="' + anchor + '"]');
+                $('.wy-menu-vertical li.toctree-l1 li.current')
+                    .removeClass('current');
+                link.closest('li.toctree-l2').addClass('current');
+                link.closest('li.toctree-l3').addClass('current');
+                link.closest('li.toctree-l4').addClass('current');
+            }
+            catch (err) {
+                console.log("Error expanding nav for anchor", err);
+            }
+        }
+    };
+
+    nav.onScroll = function () {
+        this.winScroll = false;
+        var newWinPosition = this.win.scrollTop(),
+            winBottom = newWinPosition + this.winHeight,
+            navPosition = this.navBar.scrollTop(),
+            newNavPosition = navPosition + (newWinPosition - this.winPosition);
+        if (newWinPosition < 0 || winBottom > this.docHeight) {
+            return;
+        }
+        this.navBar.scrollTop(newNavPosition);
+        this.winPosition = newWinPosition;
+    };
+
+    nav.onResize = function () {
+        this.winResize = false;
+        this.winHeight = this.win.height();
+        this.docHeight = $(document).height();
+    };
+
+    nav.hashChange = function () {
+        this.linkScroll = true;
+        this.win.one('hashchange', function () {
+            this.linkScroll = false;
+        });
+    };
+
+    nav.toggleCurrent = function (elem) {
+        var parent_li = elem.closest('li');
+        parent_li.siblings('li.current').removeClass('current');
+        parent_li.siblings().find('li.current').removeClass('current');
+        parent_li.find('> ul li.current').removeClass('current');
+        parent_li.toggleClass('current');
+    }
+
+    return nav;
+};
+
+module.exports.ThemeNav = ThemeNav();
+
+if (typeof(window) != 'undefined') {
+    window.SphinxRtdTheme = { StickyNav: module.exports.ThemeNav };
+}
+
+},{"jquery":"jquery"}]},{},["sphinx-rtd-theme"]);

QDMA/DPDK/docs/git_doc/html/_static/pygments.css

+.highlight .hll { background-color: #ffffcc }
+.highlight  { background: #eeffcc; }
+.highlight .c { color: #408090; font-style: italic } /* Comment */
+.highlight .err { border: 1px solid #FF0000 } /* Error */
+.highlight .k { color: #007020; font-weight: bold } /* Keyword */
+.highlight .o { color: #666666 } /* Operator */
+.highlight .ch { color: #408090; font-style: italic } /* Comment.Hashbang */
+.highlight .cm { color: #408090; font-style: italic } /* Comment.Multiline */
+.highlight .cp { color: #007020 } /* Comment.Preproc */
+.highlight .cpf { color: #408090; font-style: italic } /* Comment.PreprocFile */
+.highlight .c1 { color: #408090; font-style: italic } /* Comment.Single */
+.highlight .cs { color: #408090; background-color: #fff0f0 } /* Comment.Special */
+.highlight .gd { color: #A00000 } /* Generic.Deleted */
+.highlight .ge { font-style: italic } /* Generic.Emph */
+.highlight .gr { color: #FF0000 } /* Generic.Error */
+.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */
+.highlight .gi { color: #00A000 } /* Generic.Inserted */
+.highlight .go { color: #333333 } /* Generic.Output */
+.highlight .gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */
+.highlight .gs { font-weight: bold } /* Generic.Strong */
+.highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
+.highlight .gt { color: #0044DD } /* Generic.Traceback */
+.highlight .kc { color: #007020; font-weight: bold } /* Keyword.Constant */
+.highlight .kd { color: #007020; font-weight: bold } /* Keyword.Declaration */
+.highlight .kn { color: #007020; font-weight: bold } /* Keyword.Namespace */
+.highlight .kp { color: #007020 } /* Keyword.Pseudo */
+.highlight .kr { color: #007020; font-weight: bold } /* Keyword.Reserved */
+.highlight .kt { color: #902000 } /* Keyword.Type */
+.highlight .m { color: #208050 } /* Literal.Number */
+.highlight .s { color: #4070a0 } /* Literal.String */
+.highlight .na { color: #4070a0 } /* Name.Attribute */
+.highlight .nb { color: #007020 } /* Name.Builtin */
+.highlight .nc { color: #0e84b5; font-weight: bold } /* Name.Class */
+.highlight .no { color: #60add5 } /* Name.Constant */
+.highlight .nd { color: #555555; font-weight: bold } /* Name.Decorator */
+.highlight .ni { color: #d55537; font-weight: bold } /* Name.Entity */
+.highlight .ne { color: #007020 } /* Name.Exception */
+.highlight .nf { color: #06287e } /* Name.Function */
+.highlight .nl { color: #002070; font-weight: bold } /* Name.Label */
+.highlight .nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */
+.highlight .nt { color: #062873; font-weight: bold } /* Name.Tag */
+.highlight .nv { color: #bb60d5 } /* Name.Variable */
+.highlight .ow { color: #007020; font-weight: bold } /* Operator.Word */
+.highlight .w { color: #bbbbbb } /* Text.Whitespace */
+.highlight .mb { color: #208050 } /* Literal.Number.Bin */
+.highlight .mf { color: #208050 } /* Literal.Number.Float */
+.highlight .mh { color: #208050 } /* Literal.Number.Hex */
+.highlight .mi { color: #208050 } /* Literal.Number.Integer */
+.highlight .mo { color: #208050 } /* Literal.Number.Oct */
+.highlight .sb { color: #4070a0 } /* Literal.String.Backtick */
+.highlight .sc { color: #4070a0 } /* Literal.String.Char */
+.highlight .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */
+.highlight .s2 { color: #4070a0 } /* Literal.String.Double */
+.highlight .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */
+.highlight .sh { color: #4070a0 } /* Literal.String.Heredoc */
+.highlight .si { color: #70a0d0; font-style: italic } /* Literal.String.Interpol */
+.highlight .sx { color: #c65d09 } /* Literal.String.Other */
+.highlight .sr { color: #235388 } /* Literal.String.Regex */
+.highlight .s1 { color: #4070a0 } /* Literal.String.Single */
+.highlight .ss { color: #517918 } /* Literal.String.Symbol */
+.highlight .bp { color: #007020 } /* Name.Builtin.Pseudo */
+.highlight .vc { color: #bb60d5 } /* Name.Variable.Class */
+.highlight .vg { color: #bb60d5 } /* Name.Variable.Global */
+.highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */
+.highlight .il { color: #208050 } /* Literal.Number.Integer.Long */
\ No newline at end of file

QDMA/DPDK/docs/git_doc/html/dev-guide.html

+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+  <meta charset="utf-8">
+  
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  
+  <title>Developers Guide &mdash; QDMA DPDK Driver 2019.1 documentation</title>
+  
+
+  
+  
+
+  
+
+  
+  
+    
+
+  
+
+  
+  
+    <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+  
+
+  
+    <link rel="stylesheet" href="_static/graphviz.css" type="text/css" />
+  
+
+  
+        <link rel="index" title="Index"
+              href="genindex.html"/>
+        <link rel="search" title="Search" href="search.html"/>
+    <link rel="top" title="QDMA DPDK Driver 2019.1 documentation" href="index.html"/>
+        <link rel="next" title="Driver Design" href="driver-design.html"/>
+        <link rel="prev" title="User Guide" href="userguide.html"/> 
+
+  
+  <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+  <div class="wy-grid-for-nav">
+
+    
+    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+      <div class="wy-side-scroll">
+        <div class="wy-side-nav-search">
+          
+
+          
+            <a href="index.html" class="icon icon-home"> QDMA DPDK Driver
+          
+
+          
+          </a>
+
+          
+            
+            
+              <div class="version">
+                2019.1
+              </div>
+            
+          
+
+          
+<div role="search">
+  <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+    <input type="text" name="q" placeholder="Search docs" />
+    <input type="hidden" name="check_keywords" value="yes" />
+    <input type="hidden" name="area" value="default" />
+  </form>
+</div>
+
+          
+        </div>
+
+        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+          
+            
+            
+                <p class="caption"><span class="caption-text">Table of Contents</span></p>
+<ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="features.html">QDMA Features</a></li>
+<li class="toctree-l1"><a class="reference internal" href="system-requirements.html">System Requirements</a></li>
+<li class="toctree-l1"><a class="reference internal" href="build.html">Building QDMA DPDK Software</a></li>
+<li class="toctree-l1"><a class="reference internal" href="userguide.html">User Guide</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Developers Guide</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="driver-design.html">Driver Design</a></li>
+<li class="toctree-l2"><a class="reference internal" href="qdma_usecases.html">QDMA DPDK Driver UseCases</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="perf.html">QDMA DPDK Performance</a></li>
+</ul>
+
+            
+          
+        </div>
+      </div>
+    </nav>
+
+    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+      
+      <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+        <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+        <a href="index.html">QDMA DPDK Driver</a>
+      </nav>
+
+
+      
+      <div class="wy-nav-content">
+        <div class="rst-content">
+          
+
+ 
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+  <ul class="wy-breadcrumbs">
+    <li><a href="index.html">Docs</a> &raquo;</li>
+      
+    <li>Developers Guide</li>
+      <li class="wy-breadcrumbs-aside">
+        
+          
+            <a href="_sources/dev-guide.rst.txt" rel="nofollow"> View page source</a>
+          
+        
+      </li>
+  </ul>
+  <hr/>
+</div>
+          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+           <div itemprop="articleBody">
+            
+  <div class="section" id="developers-guide">
+<h1>Developers Guide<a class="headerlink" href="#developers-guide" title="Permalink to this headline">¶</a></h1>
+<div class="toctree-wrapper compound">
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="driver-design.html">Driver Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="qdma_usecases.html">QDMA DPDK Driver UseCases</a></li>
+</ul>
+</div>
+</div>
+
+
+           </div>
+          </div>
+          <footer>
+  
+    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+      
+        <a href="driver-design.html" class="btn btn-neutral float-right" title="Driver Design" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+      
+      
+        <a href="userguide.html" class="btn btn-neutral" title="User Guide" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+      
+    </div>
+  
+
+  <hr/>
+
+  <div role="contentinfo">
+    <p>
+        &copy; Copyright 2019, Xilinx, Inc.
+
+    </p>
+  </div>
+  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 
+
+</footer>
+
+        </div>
+      </div>
+
+    </section>
+
+  </div>
+  
+
+
+  
+
+    <script type="text/javascript">
+        var DOCUMENTATION_OPTIONS = {
+            URL_ROOT:'./',
+            VERSION:'2019.1',
+            COLLAPSE_INDEX:false,
+            FILE_SUFFIX:'.html',
+            HAS_SOURCE:  true
+        };
+    </script>
+      <script type="text/javascript" src="_static/jquery.js"></script>
+      <script type="text/javascript" src="_static/underscore.js"></script>
+      <script type="text/javascript" src="_static/doctools.js"></script>
+
+  
+
+  
+  
+    <script type="text/javascript" src="_static/js/theme.js"></script>
+  
+
+  
+  
+  <script type="text/javascript">
+      jQuery(function () {
+          SphinxRtdTheme.StickyNav.enable();
+      });
+  </script>
+   
+
+</body>
+</html>
\ No newline at end of file

QDMA/DPDK/docs/git_doc/source/dev-guide.rst

+****************
+Developers Guide
+****************
+
+.. toctree::
+   :maxdepth: 1
+
+   driver-design.rst
+   qdma_usecases.rst

QDMA/DPDK/docs/git_doc/source/dpdk-callbacks.rst

+******************
+DPDK Callback APIs
+******************
+
+.. include:: ../build/pmd_callbacks.rst
+

QDMA/DPDK/docs/git_doc/source/features.rst

+QDMA Features
+=============
+
+QDMA DPDK Driver supports the following list of features
+
+Hardware Features
+-----------------
+* SRIOV with 4 Physical Functions and 252 Virtual Functions
+* Memory Mapped (MM) and Stream (ST) interfaces per queue
+* 2048 queues sets
+
+   - 2048 H2C (host-to-card) Descriptor rings
+   - 2048 C2H (card-to-host) Descriptor rings
+   - 2048 Completion rings
+
+* Mailbox communication between PF and VF driver
+* Interrupt support for Mailbox events
+* HW Error reporting
+* Zero byte transfers
+* Immediate data transfers
+* Descriptor bypass (8, 16, 32, 64 descriptor sizes) support
+* Descriptor Prefetching
+* Streaming C2H completion entry coalescing
+* Disabling overflow check in completion ring
+* Streaming H2C to C2H and C2H to H2C loopback support
+* Dynamic queue configuration
+* Completion ring descriptors of 8, 16, 32, 64 bytes sizes
+* Flexible BAR mapping for QDMA configuration register space
+* ECC support
+* Completions in Memory mapped mode
+
+For details on Hardware Features refer to QDMA_Product_Guide_.
+
+.. _QDMA_Product_Guide: https://www.xilinx.com/support/documentation/ip_documentation/qdma/v3_0/pg302-qdma.pdf
+
+Software Features
+-----------------
+
+* Support DPDK v18.11 LTS
+* Support driver binding to igb_uio and vfio-pci
+* Support device arguments (module parameters) for device level configuration
+* Allows only Privileged Physical Functions to program the contexts and configuration registers
+* Device configuration through additional driver APIs
+* Interoperability between Linux driver (as PF/VF) and DPDK driver (as PF/VF)
+* SW versioning