# SPDX-License-Identifier: BSD-3-Clause
# Copyright 2018-2023 NXP
===============================================================================
NXP DPDK README
---------------
Supported Platforms (and their derivatives):
1. DPAA2 : LS108x, LS208x, LX216x
2. DPAA  : LS1043, LS1046
3. ENETC : LS1028
4. FEC   : i.MX 8M Mini, i.MX 8MP, i.MX93
5. QOS   : i.MX 8MP, i.MX93
===============================================================================
NXP DPDK provides a set of data plane libraries and network interface
controller driver for Layerscape platforms
This README provides information about building and executing DPDK based
applications for supported platforms.

NOTE: The NXP DPDK package is configured to generate single binary for
      all supported platforms. For this, configuration file is available
      in folder `config/defconfig_arm64-dpaa-linuxapp-gcc`. Further details are
      documented below.
===============================================================================

Components for Build & Execution Environment
--------------------------------------------

To successfully build and execute DPDK based applications,
following components are required:

1. DPDK source code
2. Cross compiled toolchain for ARM64 platform
3. Linux kernel, as per board being used
4. (Optional) OpenSSL package for OpenSSL driver with ARMCE support
5. (Optional) ARMv8_crypto package for ARMv8 crypto driver

Following information can be used to obtain these components:

    Fetching the DPDK code
    ~~~~~~~~~~~~~~~~~~~~~~

    Use following command to get the DPDK code

    - Internal git repository:
      $ git clone ssh://git@bitbucket.sw.nxp.com/gitam/dpdk.git -b 21.11-qoriq
      OR,
     - External Github Repository:
      $ git clone https://source.codeaurora.org/external/qoriq/qoriq-components/dpdk -b github.qoriq-os/21.11-qoriq


      $ cd dpdk # Change directory to cloned DPDK source code

    NOTE: Internal git repos is only available from within NXP intranet only.

    Linux kernel code
    ~~~~~~~~~~~~~~~~~

    Linux Kernel can be obtained either through the NXP SDK for the board or
    via the Github repository below:

      $ git clone https://source.codeaurora.org/external/qoriq/qoriq-components/linux -b github.qoriq-os/linux-5.10

    More than one kernel are available on the above link. Of those, currently
    Linux-4.19, 5.4, 5.10 has been verified with DPDK. Other kernels may work but
    they have not been tested.

    NOTE: While building DPDK on NXP boards, Linux Kernel headers might already
          be available in the rootfs. In this case, requirements for a compiled
          Linux Kernel source would not be required.

    Cross compiled toolchain For ARM64
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    Get the `gcc-7.5` or earlier toolchain from Linaro. e.g.

    # https://releases.linaro.org/components/toolchain/binaries/7.5-2019.12/aarch64-linux-gnu/

    Thereafter, set the environment variable:

    $ export CROSS=<path to uncompressed toolchain archive>/bin/aarch64-linux-gnu-

    you may also optionally install the Ubuntu based toolchain with:
    sudo apt-get install gcc-aarch64-linux-gnu


    Getting OpenSSL: (Cross Compiled Package) for ARM CE - crypto support
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    As OpenSSL library is required by DPDK application on LS* board, it needs
    to be cross compiled before being linked into DPDK compilation process.
    Follow the steps below to fetch and compile the OpenSSL package:

    Clone openssl repository

        $ git clone git://git.openssl.org/openssl.git
        $ cd openssl

    Checkout the specific tag to match openssl library in your target rootfs

        $ git checkout OpenSSL_1_1_1t

    Set the following environment variable:

        $ export CROSS_COMPILE=<path to uncompressed toolchain archive>/bin/aarch64-linux-gnu-

    Build and install 64 bit version of openssl

        $ ./Configure linux-aarch64 --prefix=<OpenSSL lib path>
        $ make depend
        $ make
        $ make install
        $ export PKG_CONFIG_PATH=<OpenSSL lib path>/lib/pkgconfig:$PKG_CONFIG_PATH
        $ export PKG_CONFIG_LIBDIR=<OpenSSL lib path>/lib:$PKG_CONFIG_LIBDIR

    NOTE: When building DPDK directly on NXP board, OpenSSL might already be
          available in the rootfs. In which case, separate compilation of
          OpenSSL package would not be required.


   Getting ARMv8 crypto: (Cross Compiled Package)
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   DPDK applications may use CAAM, Openssl or ARMv8 crypto library to perform
   IPsec operations on data. It needs to fetch and cross compiled the ARMv8
   crypto package to link with DPDK applications:

   Clone ARMv8 repository

	$ cd dpdk <dpdk repository>
	$ git submodule add https://github.com/ARM-software/AArch64cryptolib.git
	$ git submodule init
	$ git submodule update
	$ cd AArch64cryptolib

   Build ARMv8

	$ export CROSS=<path to cross-compile toolchain>
	$ make CC=${CROSS}gcc
	$ export PKG_CONFIG_PATH=<AArch64cryptolib>/pkgconfig:$PKG_CONFIG_PATH

   NOTE:--reconfigure in meson may not work. Better you delete the meson build
   folder and recreate it to use the updated PKG_CONFIG_PATH
===============================================================================

Building DPDK and Example Applications using meson
--------------------------------------------------

DPDK source code contains all necessary files for building the DPDK libraries
and the Example applications. Quick start information about build DPDK and the
example applications is provided below. For detailed information, refer the
DPDK online manuals at https://doc.dpdk.org/guides/linux_gsg/index.html.

More information on meson commands can be found at:
https://doc.dpdk.org/guides/prog_guide/build-sdk-meson.html

Pre requisite: Install pkg-config-aarch64-linux-gnu package on your machine

  sudo apt install pkg-config-aarch64-linux-gnu


Execute the following commands for cross compilation of dpdk libraries and
application:

  1. export CROSS_PATH=<path to toolchain>/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/

  2. export PATH=$PATH:$CROSS_PATH

  3. Optional: To compile using external library(eg: libcrypto), export these
               variables with proper paths to libraries.

               export PKG_CONFIG_LIBDIR="<path to libcrypto>/lib/"
               export PKG_CONFIG_PATH=$PKG_CONFIG_LIBDIR/pkgconfig

  4. meson arm64-build --cross-file config/arm/arm64_dpaa_linux_gcc -Dexamples=<list of example applications separated by commas>

     Example: To compile l2fwd and l3fwd applications, execute this command:
              meson arm64-build --cross-file config/arm/arm64_dpaa_linux_gcc -Dexamples=l2fwd,l3fwd

     NOTE: By default, both Static and Shared compilation is done by DPDK.

     NOTE: If installation is required in a specific directory, use following:

           meson arm64-build --cross-file config/arm/arm64_dpaa_linux_gcc -Dexamples=all -Dprefix=<location to install DPDK>

  5. ninja -C arm64-build install

All binaries will be there in arm64-build folder and a dpdk- prefix will be
added in the name of app binary. Example applications can be found in
arm64-build/examples directory.

  6. If you are changing configuration, you may need to reconfigure meson
  e.g.
        meson arm64-build --cross-file config/arm/arm64_dpaa_linux_gcc -Dexamples=all --reconfigure

 or with docs and debug mode
        meson arm64-build --cross-file config/arm/arm64_dpaa_linux_gcc -Dexamples=all -Dbuildtype=debug


    NOTE: If installation is required in a specific directory, use following:

        $ export DESTDIR=/path/toinsall

  Building DPDK Example Applications
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  The basic application like testpmd is compiled by default. It would be
  available in arm64-build/app or install directory.

  For building Example applications provided with DPDK source code, following
  environment variables are required to be set. It is assumed that DPDK code
  has already been compiled using the steps mentioned above.


  Compiling the Example applications:

    b) Build all applications

      $ meson arm64-build --cross-file config/arm/arm64_dpaa_linux_gcc -Dexamples=all

===============================================================================

Executing DPDK Applications
-------------------------

There are some pre-requisites for running DPDK applications on the DPAA,
DPAA2 or ENETC boards.

Following are some pre-requisites:

  For ENETC Platform:
  ~~~~~~~~~~~~~~~~~~

  1. Bring up the board with proper images using below "othbootargs":
	setenv othbootargs default_hugepagesz=2m hugepagesz=2m hugepages=256 isolcpus=1 iommu.passthrough=1

  2. Use "fsl-ls1028a-rdb-dpdk.dtb" device tree to test with DPDK.
     or
  2. Use "fdt" commands on uboot with default "fsl-ls1028a-rdb.dtb" device tree. fdt commands:
	# Load images
	tftp 0x82000000 <kernel image path>/Image
	tftp 0x92000000 <dtb path>/fsl-ls1028a-rdb.dtb
	tftp 0x94000000 <rootfs path>/fsl-image-networking-ls1028ardb.ext2.gz.u-boot

	#fdt commands to modify device tree for DPDK
	#set device tree address
	fdt addr 0x92000000
	fdt list /soc/pcie@1f0000000/
	fdt rm /soc/pcie@1f0000000/ethernet-switch@0,5/ports/port@5/ ethernet
	fdt rm /soc/pcie@1f0000000/ethernet-switch@0,5/ports/port@5/fixed-link pause
	fdt rm /soc/pcie@1f0000000/ethernet-switch@0,5/ports/port@4/fixed-link pause
	fdt rm /soc/pcie@1f0000000/mdio@0,3
	fdt rm /soc/pcie@1f0000000/ethernet@0,0/ phy-handle
	fdt rm /soc/pcie@1f0000000/ethernet@0,2/fixed-link pause
	fdt rm /soc/pcie@1f0000000/ethernet-switch@0,5/ports/port@0 phy-handle
	fdt rm /soc/pcie@1f0000000/ethernet-switch@0,5/ports/port@1 phy-handle
	fdt rm /soc/pcie@1f0000000/ethernet-switch@0,5/ports/port@2 phy-handle

	# boot command
	booti 0x82000000 0x94000000 0x92000000

  3. Mount the hugetlbfs required for DPDK applications:
	1. mkdir -p /dev/hugepages
	2. mount -t hugetlbfs hugetlbfs /dev/hugepages

  4. Get ethernet devices in userspace using below commands:
	1. For QDS board:
		echo 1 > /sys/bus/pci/devices/0000\:00\:00.1/sriov_numvfs
		echo 1 > /sys/bus/pci/devices/0000\:00\:00.0/sriov_numvfs
		echo vfio-pci > /sys/bus/pci/devices/0000\:00\:01.2/driver_override
		echo 0000:00:01.2 > /sys/bus/pci/drivers/fsl_enetc_vf/unbind
		echo 0000:00:01.2 > /sys/bus/pci/drivers/vfio-pci/bind

		echo vfio-pci > /sys/bus/pci/devices/0000\:00\:01.0/driver_override
		echo 0000:00:01.0 > /sys/bus/pci/drivers/fsl_enetc_vf/unbind
		echo 0000:00:01.0 > /sys/bus/pci/drivers/vfio-pci/bind

		ifconfig eno0 up
		ifconfig eno1 up

		Note: On QDS, There are some features limitation as VFs instead
		      of PFs to be used for IO. Below are those limitations:

		* Promiscuous and multicast cannot be enabled on VFs.
		* Port level hardware stats are not available.

	2. For RDB board, run below script:
	       /usr/local/dpdk/enetc/dpdk_configure_1028ardb.sh


  For DPAA2 Platform:
  ~~~~~~~~~~~~~~~~~~~

  1. Bring up the board with the DPAA2 images with proper DPNI-DPMAC
     configurations in the DPL file.

  2. Get the dpdk applications binaries and dynamic_dpl.sh on the DPAA2 board.
     It may already be present in rootfs at (/usr/local/dpdk/dpaa2).

      NOTE: If `pktgen` application is to be run, please increase the number of
            buffer pool available by setting the below environment variable
            before executing the dynamic_dpl.sh script:

            $ export DPBP_COUNT=16

      NOTE: If the application uses ordered queue, please enable the ordered queue
            by setting below environment variable before executing the dynamic_dpl.sh
            script:

            $ export ENABLE_ORDERED_QUEUE=1

  3. Run the following on the board:

    a) Change directory to location of Dynamic DPL script. It would ideally be
       placed in (/usr/local/dpdk/dpaa2). It is also part of the DPDK
       source code in `nxp` folder.

    b) source ./dynamic_dpl.sh dpmac.1 dpmac.2 dpmac.3 dpmac.4

       NOTE: Above command assumes that 4 ports are being configured for DPDK.
             Toggle the arguments for a different configuration.

    c) export DPRC=<dprc_container_created_by_dynamic_DPL>

  For DPAA Platform:
  ~~~~~~~~~~~~~~~~~~

  1. Bring up the board with the DPAA images with proper kernel/usdpaa
     configurations in the DTB file.

  2. Mount the hugepages for DPDK application:

    $ mkdir -p /dev/hugetlbfs # if this folder doesnt already exist
    $ echo 448 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
    $ mount -t hugetlbfs hugetlbfs /dev/hugetlbfs/

  3. By default the system will need fmc scripts to be run prior to uses.

     By default, =>
     Before running application, a binary `fmc` needs to be executed to
     configure the FMan on DPAA platforms. This requires two configuration files
     which are available in rootfs at (/usr/local/dpdk/dpaa) and
     also available as part of DPDK source code (./nxp/dpaa/).

     If the files are not available on the rootfs, upload them. Thereafter,
     execute the following command:

     $ fmc -c usdpaa_config_ls1043.xml -p usdpaa_policy_hash_ipv4_1queue.xml -a

     NOTE: There are multiple files named `usdpaa_policy_hash_ipv4_[*]queue.xml
           available in the folders mentioned above. The number defined before
           'queue.xml' is the number of queues which that configuration file
           configures when the above command is executed.

           One should be careful to use appropriate configuration (policy hash)
           file for required number of queues. Using a configuration and then
           not employing the number of queues can lead to packet loss as DPDK
           is designed to perform RSS over more than 1 queue configuration.

     NOTE: Also, files 'usdpaa_config_ls1043.xml' and 'usdpaa_config_ls1046.xml'
           are applicable for respective platform as mentioned in the name.
           Using configuration not meant for a board can render system unusable
           for DPDK applications.

  NOTE: For all the example applications, a port mask is provided for ports to
        be used by application for I/O. For DPAA platform, the number is
        defined by the order of detection which is platform specific. For DPAA2,
        the numbering is defined by order of arguments provided to dynamic_dpl
        script.
        It is important to take care of this mask as wrong values would result
        in either incorrectly configured I/O or I/O loss.
        For details of port numbering, refer the LSDK documentation.

  For i.MX Platform:
  ~~~~~~~~~~~~~~~~~~

  1.1 For i.MX8MM (FEC) Platform:
    Bring up the imx-8mm-evk-rev board with latest kernel and dtb.
    Enable User space mode of the kernel driver by using 'imx8mm-evk-dpdk.dtb'.
    On uboot set the dtb & boot linux using below commands:
    => setenv fdt_file imx8mm-evk-dpdk.dtb
    => boot

	or

  1.1 For i.MX8MM (FEC) Platform:
    
    Set the env. variable on uboot to enable user space mode:

    Note: All the variables should be defined with appropriate values
    => setenv dpdk_enable
    	"setenv fdt_file imx8mm-evk.dtb; setenv loadkernel 'fatload mmc ${mmcdev}:${mmcpart} ${loadaddr} Image';
    	setenv loadfdt 'fatload mmc ${mmcdev}:${mmcpart} ${fdt_addr} ${fdt_file}';
    	run loadkernel; run loadfdt; fdt addr ${fdt_addr};
	fdt rm /soc@0/bus@30800000/ethernet@30be0000/mdio/ethernet-phy@0 "reset-assert-us";
	fdt rm /soc@0/bus@30800000/ethernet@30be0000/mdio/ethernet-phy@0 "reset-deassert-us";
	fdt set /soc@0/bus@30800000/ethernet@30be0000 compatible "fsl,imx8mm-fec-uio";
	setenv bootcmd 'mmc dev ${mmcdev};run mmcargs; booti ${loadaddr} - ${fdt_addr};'"
    => saveenv
    => run dpdk_enable

  1.2 For i.MX8MP (FEC) Platform:

    Set the env. variable on uboot to enable user space mode:

    Note: All the variables should be defined with appropriate values
    => setenv dpdk_enable
    	"setenv fdt_file imx8mp-evk.dtb; setenv loadkernel 'fatload mmc ${mmcdev}:${mmcpart} ${loadaddr} Image';
     	setenv loadfdt 'fatload mmc ${mmcdev}:${mmcpart} ${fdt_addr} ${fdt_file}';
	run loadkernel; run loadfdt; fdt addr ${fdt_addr};
     	fdt rm /soc@0/bus@30800000/ethernet@30be0000/mdio/ethernet-phy@1 "reset-gpios";
     	fdt rm /soc@0/bus@30800000/ethernet@30be0000/mdio/ethernet-phy@1 "reset-assert-us";
     	fdt rm /soc@0/bus@30800000/ethernet@30be0000/mdio/ethernet-phy@1 "reset-deassert-us";
     	fdt set /soc@0/bus@30800000/ethernet@30be0000 compatible "fsl,imx8mm-fec-uio";
     	setenv bootcmd 'mmc dev ${mmcdev};run mmcargs; booti ${loadaddr} - ${fdt_addr};'"
    => saveenv
    => run dpdk_enable

     or

    => setenv fdt_file imx8mp-evk-dpdk.dtb
    => boot

  1.3 For i.MX93 (FEC) Platform:
    Bring up the iMX93 board with latest kernel.
    On uboot, enable User space mode of the kernel driver & boot linux after setting the below env. variable:

    Note: All the variables should be defined with appropriate values
    => setenv dpdk_enable
	"setenv loadkernel 'fatload mmc ${mmcdev}:${mmcpart} ${loadaddr} Image';
	 setenv loadfdt 'fatload mmc ${mmcdev}:${mmcpart} ${fdt_addr} ${fdt_file}';
	 run loadkernel; run loadfdt; fdt addr ${fdt_addr};
	 fdt set /soc@0/bus@42800000/ethernet@42890000 compatible "fsl,imx8mm-fec-uio";
	 setenv bootcmd 'mmc dev ${mmcdev}; run mmcargs; booti ${loadaddr} - ${fdt_addr};'"
    => saveenv
    => run dpdk_enable

    The descriptions of the variables used above are as follows:
    - mmcdev: MMC/SD device id.
    - mmcpart: This is the partition on the MMC/SD card containing the kernel image.
    - loadaddr: The address in the memory where the kernel is loaded.
    - Image: Linux kernel image.
    - fdt_addr: The address in the memory where the device tree is loaded.
    - fdt_file: Used to specify the filename of the device tree file (DTB).
    - mmcroot: The location of the root file system on the MMC SD card along with directives for the boot command for the rootfs.
      For example,  mmcargs=setenv bootargs ${jh_clk} console=${console} root=${mmcroot}
		    bootargs=console=ttyLP0,115200 earlycon root=/dev/mmcblk0p2 rootwait rw

  2. On Linux mount the hugetlbfs required for DPDK applications:
	1. mkdir -p /dev/hugepages
	2. mount -t hugetlbfs hugetlbfs /dev/hugepages

  3. Manually allocate hugepages
	e.g. echo 448 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

  4. Find DPDK applications(dpdk-l2fwd & dpdk-l3fwd) under the folder
     '/usr/share/dpdk/examples/' and dpdk-testpmd at '/usr/bin/'.

  5. Run-
     ./dpdk-l2fwd -c 0x1 -n 1 --vdev 'net_enetfec' -- -p 0x1 -T 0 -P
     ./dpdk-l3fwd -c 0x1 --vdev='net_enetfec'  -n 1 -- -p 0x1 --config="(0,0,0)" -P --parse-ptype

  6. Run/observe traffic
     For functional testing, send the traffic from packet generator which is connected to the port.

  NOTE: For i.MX platforms, all the applications, user must provide the "--vdev"
	argument with value "net_enetfec" to enable ethernet device. Maximum supported
	ethernet is currently 1 only.
  NOTE: For i.MX platforms, reserve the initial DDR memory from uboot via 'mem' environment variable in bootargs.
	Example: 'mem=4096M'



  NOTE: For all the applications, the value n must be always 1 for all platforms.

  NOTE: Platforms with CAAM JR support need the kernel support and they need
	to run following commands to assign job rings to userspace.
	#Find the address for job ring
		ls /sys/bus/platform/devices/*.jr
	#leave one and assign remaining rings to userspace drivers.
		echo 1720000.jr > /sys/bus/platform/drivers/caam_jr/unbind
		echo 1720000.jr > /sys/bus/platform/drivers/fsl-jr-uio/bind

		echo 1730000.jr > /sys/bus/platform/drivers/caam_jr/unbind
		echo 1730000.jr > /sys/bus/platform/drivers/fsl-jr-uio/bind

	You will need to use --vdev as "crypto_caam_jr0" "crypto_caam_jr1"
	to use these rings.

Steps for ENET-QOS:
-------------------

Steps to compile the kpage non-cacheable module:
------------------------------------------------
Under the dpdk-extras repo,
https://bitbucket.sw.nxp.com/projects/GITAM/repos/dpdk-extras/browse?at=refs%2Fheads%2Fmain

External repo:
https://github.com/nxp-qoriq/dpdk-extras

Refer README for kpage_ncache module in linux/kpage_ncache directory.

NOTE: Remember for i.MX platforms, reserve the initial DDR memory from uboot
via 'mem' environment variable in bootargs.
	Example: 'mem=4096M'

1.1 For i.MX8MP Platform:
-------------------------
Set the dpdk DTB file.
    => setenv fdt_file imx8mp-evk-dpdk.dtb
    => boot

OR

Set the env. variable on uboot to enable user space mode:

    Note: All the variables should be defined with appropriate values
    => setenv dpdk_enable "setenv fdt_file imx8mp-evk.dtb; setenv loadkernel 'fatload mmc ${mmcdev}:${mmcpart} ${loadaddr} Image'; 
       setenv loadfdt 'fatload mmc ${mmcdev}:${mmcpart} ${fdt_addr} ${fdt_file}';
       run loadkernel; run loadfdt; fdt addr ${fdt_addr};
       fdt set /soc@0/bus@30800000/ethernet@30bf0000 compatible "fsl,imx-enet-qos" "snps,dwmac-5.10a";
       setenv bootcmd 'mmc dev ${mmcdev};run mmcargs; booti ${loadaddr} - ${fdt_addr};'" 
    => saveenv
    => run dpdk_enable

1.2 For i.MX93 Platform:
------------------------
Set the env. variable on uboot to enable user space mode:
Note: All the variables should be defined with appropriate values

    => setenv dpdk_enable
       "setenv loadkernel fatload mmc ${mmcdev}:${mmcpart} ${loadaddr} Image;
        run loadkernel; run loadfdt; fdt addr 0x83000000;
        fdt set /soc@0/bus@42800000/ethernet@428a0000 compatible "fsl,imx-enet-qos" "snps,dwmac-5.10a";
        run mmcargs; sleep 1; booti ${loadaddr} - ${fdt_addr};"
    => saveenv
    => run dpdk_enable

Below steps are for running DPDK applications on i.MX8MP and i.MX93 boards.
2. Manually allocate hugepages
        e.g. echo 448 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
3. Load the non-cacheable module
	$ insmod kpage_ncache.ko
4. Find DPDK applications (dpdk-l2fwd & dpdk-testpmd) under the folder
	'/usr/share/dpdk/examples/' and dpdk-testpmd at '/usr/bin/'.

5. Run -
	./dpdk-l2fwd -c 0x3 -n 1 --vdev 'net_enetqos' -- -p 0x1 -q 1
	./dpdk-testpmd -c 0x3 -n 1 --vdev 'net_enetqos'  -- -i --portmask=0x2 --nb-ports=1 --forward-mode=io

	For testing the forwarding between both FEC and QOS port run:
		./dpdk-l2fwd -c 0xf -n 1 --vdev 'net_enetqos'  --vdev 'net_enetfec' -- -p 0x3 -T 0 -P
		./dpdk-testpmd -c 0xf -n 1 --vdev 'net_enetqos' --vdev 'net_enetfec'  -- -i --portmask=0x3 --nb-ports=2 --forward-mode=io

6. Run traffic
	For functional testing, send the traffic from packet generator
	which is connected to the port.

===============================================================================

  'dpdk-testpmd' Application
  ~~~~~~~~~~~~~~~~~~~~~

  'dpdk-testpmd' is part of standard DPDK build process. The binary is generated in
  the (./<target>/app/) folder. On a rootfs having DPDK binaries, the binary
  would be placed in (/usr/local/bin/).

  Execute following commands to run DPDK testpmd

  $./dpdk-testpmd -c 0xF -n 1 -- -i --portmask=0x3 --nb-cores=2

  Above command uses a port mask of first two ports. On DPAA and DPAA2 platforms
  this is defined by their respective configuration. See note above.

  This command would start the testpmd application in interactive mode, starting
  a shell for accepting further commands. For e.g:

  testpmd> show port info all

  Above command can be used to view all the ports which the framework has
  identified, with their detailed information.

  testpmd> <tab>

  On the `testpmd` prompt, <tab> key can be used for command completion as well
  as command help.

  On i.mx8mm platform
  ./dpdk-testpmd -c 0x6 -n 1 --vdev='net_enetfec' -- -i --portmask=0x1 --nb-ports=1 --forward-mode=io

  'dpdk-test' Application
  ~~~~~~~~~~~~~~~~~~

  Test application can be used to do functional testing of the drivers and libraries.
  The binary is tested for various algos and protocols in crypto pmds. 'dpdk-test' binary
  is generated at arm64-build/app/test while compiling DPDK.

  Run binary as
  $./dpdk-test --log-level=7

  (for DPAA_SEC)
  RTE>> cryptodev_dpaa_sec_autotest

  or (for DPAA2_SEC)
  RTE>> cryptodev_dpaa2_sec_autotest

  Run the 'autotest.py' script as
  $ python3 app/test/autotest.py ./dpdk-test arm64-dpaa-linuxapp-gcc

  NOTE: For 'autotest.py' script, all the dependent 'app/test/*.py' scripts and
	'python-pexpect' package is required on the board.
	$ sudo apt-get install python3-pexpect

  'dpdk-test-crypto-perf' Application
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  'test-crypto-perf' is part of standard DPDK build process. The binary is generated
  in (./<target>/build/app/test-crypto-perf/) folder.
  A script 'crypto_perf_test.sh' is added in nxp/ folder to run all supported cases
  in one go.

  Use the script as (for DPAA_SEC)
  $./crypto_perf_test.sh dpaa_sec
  or (for DPAA2_SEC)
  $./crypto_perf_test.sh dpaa2_sec

  Some default parameters are added in the script which can be modified as per
  requirement.


  Layer-2 Forwarding 'l2fwd' Application
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  Execute following commands to run the 'l2fwd' example application:

  $ ./dpdk-l2fwd -c 0x1 -n 1 -- -p 0x1 -q 1 -P

  OR

  $ ./dpdk-l2fwd -c 0x3 -n 1 -- -p 0x3 -q 1 -P

  In the above commands, the port mask has been modified to support first and
  first two ports, respectively. Also, the Core mask has been modified to
  execute on Core 0 and (Core 0 + Core 1), respectively.

  NOTE: For best performance, Core 0 should not be used for performing DPDK
        I/O. This is because large number of system services as well as some
        hardwired interrupts lines are services by Core 0.

        It is also advisable to use 'isolcpus=<>' when booting Linux Kernel.
        Value passed to this parameter should be the CPUs planned to be used
        for DPDK applications.


  Layer-2 Crypto Forwarding 'l2fwd-crypto' Application
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  Execute following commands to run the 'l2fwd-crypto' example application:

  #cipher mode only
  $ ./dpdk-l2fwd-crypto -c 0x1 -n 1 -- -p 0x1 --chain CIPHER_ONLY --cipher_op ENCRYPT --cipher_algo aes-cbc --cipher_key 00:01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f

  #auth mode only
  $ ./dpdk-l2fwd-crypto -c 0x1 -n 1 -- -p 0x1 -q 1 --chain HASH_ONLY --auth_algo sha1-hmac --auth_op GENERATE --auth_key_random_size 64

  #cipher-auth mode
  $ ./dpdk-l2fwd-crypto -c 0xf -n 1 -- -p 0x1 --chain CIPHER_HASH --cipher_op ENCRYPT --cipher_algo aes-cbc --auth_algo sha1-hmac --auth_op GENERATE --auth_key_random_size 64

  For caam_jr mode:
	use  --vdev 'crypto_caam_jr1'
  For openssl mode:
	use --vdev 'crypto_openssl'
  For armv8_crypto mode:
	use --vdev 'crypto_armv8'

  You need to disable the DPAA_HW SEC on DPAA devices by to use any other sec driver.
   export DPAA_SEC_DISABLE=1
   or use the export DPSECI_COUNT=0 for dpaa2 before creating the dprc.

  Layer-3 Forwarding 'l3fwd' Application
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  Execute following commands to run 'l3fwd' application

  For 1 core - 1 Port, 1 queue per port =>

  $ ./dpdk-l3fwd -c 0x1 -n 1 -- -p 0x1 --config="(0,0,0)"

  For 4 core - 2 Port, 2 queue per port =>

  $ ./dpdk-l3fwd -c 0xF -n 1 -- -p 0x3 -P --config="(0,0,0),(0,1,1),(1,0,2),(1,1,3)"

  For 4 core - 2 Port with dest mac =>

  $ ./dpdk-l3fwd -c 0xF -n 1 -- -p 0x3 -P --config="(0,0,0),(0,1,1),(1,0,2),(1,1,3)" --eth-dest=0,11:11:11:11:11:11 --eth-dest=1,00:00:00:11:11:11

  For 8 core - 4 Port with 4 queue per port =>

  $ ./dpdk-l3fwd -c 0xFF -n 1 -- -p 0x3 -P --config="(0,0,0),(0,1,1),(1,0,2),(1,1,3),(2,0,4),(2,1,5),(3,0,6),(3,1,7)"

  Hereafter, use the packet generator (Spirent, for example) to send traffic
  streams with below configuration of the destination IP address in the frames
  being sent:

      For traffic to port 1: 1.1.1.0/24
      For traffic to port 2: 2.1.1.0/24
      For traffic to port 3: 3.1.1.0/24
      For traffic to port 4: 4.1.1.0/24

  Layer-3 Forwarding 'l3fwd' Application using traffic bifurcation
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  L3fwd application has been modified to support traffic bifurcation, where traffic can be
  bifurcated such that some traffic goes to the kernel and other to DPDK using DPDMUX.
  Separate rules can be configured for this bifurcation. Below are some sample commands
  on LX2:

  1. Configure DPDMUX to have two DPNI's, one with kernel and other with user-space

  $ ls-addni --no-link
  $ source /usr/local/dpdk/dpaa2/dynamic_dpl.sh dpmac.5
  $ echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/unbind
  $ restool dprc disconnect dprc.2 --endpoint=dpni.3
  $ restool dpdmux create --default-if=1 --num-ifs=2 --method DPDMUX_METHOD_CUSTOM --manip=DPDMUX_MANIP_NONE --option=DPDMUX_OPT_CLS_MASK_SUPPORT --container=dprc.1
  $ restool dprc connect dprc.1 --endpoint1=dpdmux.0.0 --endpoint2=dpmac.5
  $ restool dprc connect dprc.1 --endpoint1=dpdmux.0.1 --endpoint2=dpni.2
  $ restool dprc connect dprc.1 --endpoint1=dpdmux.0.2 --endpoint2=dpni.3
  $ restool dprc assign dprc.1 --object=dpdmux.0 --child=dprc.2 --plugged=1
  $ echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind
  $ ifconfig eth0 1.1.1.2
  $ ifconfig eth0 promisc

  2. Run l3fwd application providing 'traffic-split-config' option for configuring DPDMUC rules:

  $ ./dpdk-l3fwd -c 0x6 -n 1 -- -p 0x1 --config="(0,0,1),(0,1,2)" -P --traffic-split-config="(3,2152,2)"

  Here "--traffic-split-config - (type,val,mux_conn_id)" is defined by
       'type' - 1: ETHTYPE,
		2: IP_PROTO,
		3: UDP_DST_PORT 	having value as 'val' based on which DPDMUX can
					split the traffic to mux_conn_id
		4: IP_FRAG_UDP_AND_GTP
		5: IP_FRAG_PROTO
		6: IP_FRAG_UDP_AND_GTP_AND_ESP

  IPsec Gateway 'dpdk-ipsec-secgw' Application
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  For IPsec application, two DUTs need to be configured as endpoint 0 (ep0) and endpoint 1 (ep1). Assuming that endpoint
  have 4 ports each:
   Connect Port 1 and Port 1 of the ep0 and ep1 to each other (back-to-back).
   Connect Port 0 and Port 0 of the ep0 and ep1 to packet generator (for example, Spirent).

  The sample configurations for both ep0 and ep1 are available at "nxp/ipsec".
  Custom port mappings, SA/SP and the routes can be configured in the corresponding
  configuration file named as ep0.cfg and ep1.cfg for respective endpoint.

  Endpoint0 Command:
  -----------------
  * Example command for 4 cores and 4 ports:
  dpdk-ipsec-secgw -c 0xf -n 1 -- -p 0xf -P -u 0xa --config="(0,0,0),(1,0,1),(2,0,2),(3,0,3)" -f
  ep0_64X64_proto.cfg

  Endpoint1 Command:
  -----------------
  * Example command for 4 cores and 4 ports:
  dpdk-ipsec-secgw -c 0xf -n 1 -- -p 0xf -P -u 0xa --config="(0,0,0),(1,0,1),(2,0,2),(3,0,3)" -f
  ep1_64X64_proto.cfg

  IPsec Gateway 'dpdk-ipsec-secgw' Application using eventdev
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  Execute following commands to run 'dpdk-ipsec-secgw' application with eventdev

  Note:
    1. For DPAA1 platforms

	-> disable push mode queue, using 'export DPAA_PUSH_QUEUES_NUMBER=0'

	-> virtual device should be "event_dpaa1" i.e. --vdev="event_dpaa1"

    2. For DPAA2 platforms

	-> virtual device should be "event_dpaa2" i.e. --vdev="event_dpaa2"

  -- With parallel queue configuration

  $ ./dpdk-ipsec-secgw -c 0xc --vdev="event_dpaa1" --vdev="event_dpaa1" -- -p 0x30 -P -u 0x20 -f ./ep0_64X64_proto.cfg --transfer-mode=event --event-schedule-type=parallel

  -- With atomic queue configuration

  $ ./dpdk-ipsec-secgw -c 0xc --vdev="event_dpaa1" --vdev="event_dpaa1" -- -p 0x30 -P -u 0x20 -f ./ep0_64X64_proto.cfg --transfer-mode=event --event-schedule-type=atomic

  -- With ordered queue configuration

  $ ./dpdk-ipsec-secgw -c 0xc --vdev="event_dpaa1" --vdev="event_dpaa1" -- -p 0x30 -P -u 0x20 -f ./ep0_64X64_proto.cfg --transfer-mode=event --event-schedule-type=ordered

  Note:
      * Above commands are for the 1st endpoint, to run on 2nd endpoint just change the configuration file "ep0_64X64_proto.cfg" to "ep1_64X64_proto.cfg"
      * There is no queue configuration for event mode. Only single QP is supported which will be shared by all the cores.

  Layer-2 Forwarding 'dpdk-l2fwd-event' Application
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  Execute following commands to run 'dpdk-l2fwd-event' application with eventdev

  Note:
    1. For DPAA1 platforms

	-> disable push mode queue, using 'export DPAA_PUSH_QUEUES_NUMBER=0'

	-> virtual device should be "event_dpaa1" i.e. --vdev="event_dpaa1"

    2. For DPAA2 platforms

	-> virtual device should be "event_dpaa2" i.e. --vdev="event_dpaa2"

    3. Only single instance of virtual device(vdev) is allowed.

  -- With parallel queue configuration

  For 1 core, 1 Port, 1 queue per port=>

  $ dpdk-l2fwd-event -c 0x14 -n 1 --vdev=event_dpaa2 -- -p 0x1 -T 0 --mode=eventdev -q 1 --eventq-sched=parallel

  -- With atomic queue configuration

  For 1 core, 1 Port, 1 queue per port=>

  $ dpdk-l2fwd-event -c 0x14 -n 1 --vdev=event_dpaa2 -- -p 0x1 -T 0 --mode=eventdev -q 1 --eventq-sched=atomic

  -- With ordered queue configuration

  For 1 core, 1 Port, 1 queue per port=>

  $ dpdk-l2fwd-event -c 0x14 -n 1 --vdev=event_dpaa2 -- -p 0x1 -T 0 --mode=eventdev -q 1 --eventq-sched=ordered

  Layer-3 Forwarding 'dpdk-l3fwd' Application using eventdev
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  Execute following commands to run 'l3fwd' application with eventdev

  Note:
    1. For DPAA1 platforms

	-> disable push mode queue, using 'export DPAA_PUSH_QUEUES_NUMBER=0'

	-> virtual device should be "event_dpaa1" i.e. --vdev="event_dpaa1"

    2. For DPAA2 platforms

	-> virtual device should be "event_dpaa2" i.e. --vdev="event_dpaa2"

    3. Only single instance of virtual device(vdev) is allowed.

  -- With parallel queue configuration

  For 1 core, 1 Port, 1 queue per port=>

  $ dpdk-l3fwd -c 0x2 -n 1 --vdev="event_dpaa2" -- -p 0x1 --mode=eventdev --eventq-sched=parallel --event-eth-rxqs=1 -P

  -- With atomic queue configuration

  For 1 core, 1 Port, 1 queue per port=>

  $ dpdk-l3fwd -c 0x2 -n 1 --vdev="event_dpaa2" -- -p 0x1 --mode=eventdev --eventq-sched=atomic --event-eth-rxqs=1 -P

  -- With ordered queue configuration

  For 1 core, 1 Port, 1 queue per port=>

  $ dpdk-l3fwd -c 0x2 -n 1 --vdev="event_dpaa2" -- -p 0x1 --mode=eventdev --eventq-sched=ordered --event-eth-rxqs=1 -P

================================================================================
Executing dpdk-pdump/dpdk-procinfo on DPAA2
===========================================

Both, 'dpdk-pdump' and 'dpdk-procinfo' are examples of secondary applications
which run along a primary DPDK process. Being different processes, rather than
thread, they work over the DPDK EAL's Multiprocess infrastructure to communicate
with the primary process.

  Layer-3 Forwarding 'l3fwd' Application using traffic bifurcation
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  L3fwd application has been modified to support traffic bifurcation, where traffic can be
  bifurcated such that some traffic goes to the kernel and other to DPDK using DPDMUX.
  Separate rules can be configured for this bifurcation. Below are some sample commands
  on LX2:

  1. Configure DPDMUX to have two DPNI's, one with kernel and other with user-space

  $ ls-addni --no-link
  $ source /usr/local/dpdk/dpaa2/dynamic_dpl.sh dpmac.5
  $ echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/unbind
  $ restool dprc disconnect dprc.2 --endpoint=dpni.3
  $ restool dpdmux create --default-if=1 --num-ifs=2 --method DPDMUX_METHOD_CUSTOM --manip=DPDMUX_MANIP_NONE --option=DPDMUX_OPT_CLS_MASK_SUPPORT --container=dprc.1
  $ restool dprc connect dprc.1 --endpoint1=dpdmux.0.0 --endpoint2=dpmac.5
  $ restool dprc connect dprc.1 --endpoint1=dpdmux.0.1 --endpoint2=dpni.2
  $ restool dprc connect dprc.1 --endpoint1=dpdmux.0.2 --endpoint2=dpni.3
  $ restool dprc assign dprc.1 --object=dpdmux.0 --child=dprc.2 --plugged=1
  $ echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind
  $ ifconfig eth0 1.1.1.2
  $ ifconfig eth0 promisc

  2. Run l3fwd application providing 'traffic-split-config' option for configuring DPDMUC rules:

  $ ./l3fwd -c 0x6 -n 1 -- -p 0x1 --config="(0,0,1),(0,1,2)" -P --traffic-split-config="(3,2152,2)"

  Here "--traffic-split-config - (type,val,mux_conn_id)" is defined by
       'type' -  1:ETHTYPE, 2:IP_PROTO, 3:UDP_DST_PORT having value as 'val' based on which DPDMUX can split the traffic to mux_conn_id

`dpdk-pdump` for capturing packets
----------------------------------

DPDK Supports executing secondary processes/applications which can query
the primary application (the first one) for information. For example, the
sample application 'dpdk-pdump' allows capturing of packets arriving on
other interfaces/devices available in DPDK.

The libpcap-based PMD has an external dependency on the libpcap development files
which must be installed on the board, if its not already installed.

	$ wget https://www.tcpdump.org/release/libpcap-1.10.1.tar.gz
	$ tar -xzvf libpcap-1.10.1.tar.gz
	$ cd libpcap-1.10.1
	$ ./configure --host=aarch64-linux-gnu --with-pcap=linux
	$ make install

To check if the library is installed:
	$ ldconfig -p | grep libpcap

Compiling 'dpdk-pdump':

Steps to cross compile libpcap:
------------------------------------------------
	$ git clone https://github.com/the-tcpdump-group/libpcap.git
	$ cd libpcap
	$ git checkout libpcap-1.10.1 -b libpcap-1.10.1
	$ ./configure --host=aarch64-linux-gnu --with-pcap=linux --prefix=<Path to install pcap directory>
	$ make install

Steps to cross compile dpdk-pdump:
------------------------------------------------

	$ export PKG_CONFIG_LIBDIR=$PKG_CONFIG_LIBDIR:<Path to installed pcap directory>/lib
	$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:<Path to installed pcap directory>/lib/pkgconfig
	$ pkg-config --modversion libpcap	#Verify pcap libraries can be used using pkgconfig.
	$ meson arm64-build --cross-file config/arm/arm64_dpaa_linux_gcc
	$ ninja -C arm64-build

	NOTE: "--reconfigure" in meson may not work after updating
	      PKG_CONFIG_PATH and PKG_CONFIG_LIBDIR. Better you delete the meson
	      build folder and recreate it to use the updated paths.

'dpdk-pdump'` would be available in the arm64-build/app folder along with 'testpmd'
and other inbuilt DPDK apps.

Executing 'dpdk-pdump':
    Pre-condition: disable ASLR:

    $ echo 0 > /proc/sys/kernel/randomize_va_space

    Note: As of now only testpmd application supports dpdk-pdump;

    Step 1: Execute testpmd using routine commands

    $ dpdk-testpmd -c 0xff -n 1 -- -i --portmask=0x3 --nb-ports=2

    Step 2: Get testpmd ready for IO by selecting the mode

    $ testpmd> set fwd io
    $ testpmd> start

    Step 3: Execute dpdk-pdump on a different terminal:

    $ dpdk-pdump -n 1 -- --pdump "port=0,queue=*,rx-dev=./rx.pcap"

    In the above command, "port=0, ..." parameters convey `dpdk-pdump` app
    that capture should be done on port 0 (for example, dpni.1) and packets
    being received on all queues "queue=*". Further, all the captured packets
    can be dumped to a PCAP file using "rx-dev=<path to pcap file>.

    In the above example, only Rx'd packets are being written to PCAP. For
    Tx'd packets, use something similar to "port=0,queue=*,tx-dev=./..." where
    the output PCAP file is different from Rx'd packets. Both, Rx and Tx,
    options can be simultaneously provided.

    Step 4: Stop 'dpdk-pdump' using Ctrl+C and copy the pcap file for reading
    through external application like Wireshark

Executing `dpdk-procinfo`:

    'dpdk-procinfo' is an inbuilt application which allows dumping information
    like memory and statistics of a running (primary) DPDK application. It is
    available in the app/ folder just like testpmd and dpdk-pdump.

    Note: This can work with any other DPDK application (sample application
    like l3fwd) unlike the restriction for 'dpdk-pdump'.

    Pre-condition: disable ASLR:

    $ echo 0 > /proc/sys/kernel/randomize_va_space

    Step 1: Execute any DPDK application using routine commands. For example,
    testpmd

    $ testpmd -c 0xff -n 1 -- -i --portmask=0x3 --nb-ports=2

    Step 2: Execute 'dpdk-procinfo' on a separate terminal:

    $ dpdk-procinfo -- -m

    Above command would dump to screen the memory layout of the primary DPDK
    application. There are other switches also available like '-s' which can
    dump the statistics.

    Note: dpdk-procinfo will work without providing blocklist only if there is
    single primary process running (no other secondary). If you are using multi
    process application, you will need to provide blacklist to dpdk-procinfo.

===============================================================================

Building Pktgen Application
======================================

Pktgen is a packet generator powered by DPDK. It requires DPDK environment for
compilation and DPDK compliant infrastructure for execution. DPAA and DPAA2 DPDK
PMD (Poll Mode Drivers) can be used by Pktgen for building a packet generator
using the DPAA infrastructure.

Steps to compile pktgen on host:
----------------------------------

These steps assumes that compiled DPDK binaries are available in DPDK build
directory. Refer the section "`dpdk-pdump` for capturing packets"
for compilation steps of DPDK on host.

	$ export RTE_SDK=<path to compiled DPDK source code containing build folder>
	$ export RTE_TARGET=arm64-build
	$ export PKG_CONFIG_PATH=<path to meson-private in arm64-build folder>
	$ git clone -b pktgen-19.12.0 http://dpdk.org/git/apps/pktgen-dpdk
	$ cd pktgen-dpdk/
	$ meson build -Dc_args="-DRTE_FORCE_INTRINSICS"
	$ ninja -C build install

Pktgen app location: pktgen-dpdk/build/app/pktgen
Pktgen Pktgen.lua location: pktgen-dpdk/Pktgen.lua

Pktgen Tag being used: pktgen-19.12.0

===============================================================================


Building OVS Application
======================================

Steps to cross compile ovs
--------------------------
These steps assumes that compiled DPDK and openssl libraries are installed in
specific directories.
Refer to "Building DPDK and Example Applications using meson" for DPDK
compilation steps.

Clone OVS repository:
	$ git clone ssh://git@bitbucket.sw.nxp.com/dqns/ovs-dpdk.git
	$ cd ovs-dpdk
	$ git checkout branch-3.1-qoriq

Build OVS:
	$ export PKG_CONFIG_LIBDIR=$PKG_CONFIG_LIBDIR:<Path to installed DPDK directory>/lib:<Path to installed openssl directory>/lib
	$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:<Path to installed DPDK directory>/lib/pkgconfig:<Path to installed openssl directory>/lib/pkgconfig
	$ pkg-config --modversion libdpdk     #Verify dpdk libraries can be used using pkgconfig.
	$ pkg-config --modversion libssl      #Verify openssl libraries can be used using pkgconfig.
	$ export CC=aarch64-linux-gnu-gcc
	$ ./boot.sh
	$ ./configure --with-dpdk=static --host=aarch64-fsl-linux
	  CFLAGS="-g -Wno-cast-align -Ofast"
	  --with-openssl=<Path to installed openssl directory>
	$ make 'CFLAGS=-g -Ofast' -j 4

	NOTE: Use complete absolute path(Dont use '~') in ./configure command to
	      compile using openssl. Otherwise it will not be able to find the
	      openssl libraries.

OVS binaries:
	ovsdb/ovsdb-server
	ovsdb/ovsdb-tool
	utilities/ovs-ofctl
	utilities/ovs-vsctl
	vswitchd/ovs-vswitchd
	vswitchd/vswitch.ovsschema

=============================================
IP-fragmentation & Reassembly Appilcations:

The IP Fragmentation and Reassembly Library implements IPv4 and IPv6 packet
fragmentation and reassembly.

Steps to execute IP-fragmentation:
----------------------------------
1. Run the following command:
	./dpdk-ip_fragmentation -c 0xf -n 1 -- -p 0x20
2. Send different sized packets from spirent and capture the frames to verify
   the fragments formed in every case.

Steps to execute Reassembly application:
-----------------------------------------
1. Run the following command:
	./dpdk-ip_reassembly -c 0xf -n 1 -- -p 0x20 --maxflows=1024
2. Create fragmented packets on Spirent and capture the output to verify the
   reassembled packet.

=========================
VM VFIO Direct Assignment
=========================

Refer to nxp/dpaa2/README_VM_VFIO_DIRECT for further information.

=================================================
Achieving best performance from DPDK Applications
=================================================

There are multiple configurations points for tuning a DPDK application over
the supported NXP boards. Some broad points are listed below. For detailed
information, refer the `Performance Reproducibility Guide` available as part
of NXP LSDK documentation.

   DPAA/DPAA2: Avoid using Core 0
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   It has been observed that with Ubuntu rootfs (or, for that matter, any other
   fully featured/function distro), there would be large number of system
   services which would require/consume CPU cycles.

   By using `isolcpus=<core list>` in Linux bootargs (u-boot),
   '<core list>` CPUs can be prevented from being used by Linux Kernel for
   scheduling system services and applications using its scheduling algorithm.
   For example, `isolcpus=1-7`, only Core 0 would be used by Linux Kernel for
   scheduling its tasks. Cores 1-7 can then be used by DPDK Applications
   without interruption from Kernel.

   Also, in the folder (/usr/local/dpdk/), `disable_services.sh` script has
   been provided for disabling all services on a Ubuntu 16.04 rootfs which are
   known to impact DPDK performance. (Only for Stock 16.04).

   For DPAA, following Linux bootargs parameter should be used:

   `isolcpus=1-3`

   For DPAA2, following Linux bootargs parameter should be used:

   `isolcpus=1-7`

   DPAA/DPAA2: Hugepage Configuration
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   It is preferrable to use 1G Hugepages for best performance.

   For DPAA1, following Linux bootargs parameter should be used:

   `default_hugepagesz=2m hugepagesz=2m hugepages=448`

   For DPAA2, following Linux bootargs parameter should be used:

   `default_hugepagesz=1024m hugepagesz=1024m hugepages=8`

   One can also modify the hugepages on linux command line. e.g.

    $echo 400 > /proc/sys/vm/nr_hugepages

================================================================================

Layout of Extra files in 'nxp' folder
=====================================

nxp/
        build_dpdk.sh
        COPYING
        disable_services.sh
        dpaa
                usdpaa_config_ls1043.xml
                usdpaa_config_ls1046.xml
                usdpaa_policy_hash_ipv4_1queue.xml
                usdpaa_policy_hash_ipv4_2queue.xml
                usdpaa_policy_hash_ipv4_4queue.xml
        dpaa2
                destroy_dynamic_dpl.sh
                dynamic_dpl.sh
                vm_linux.conf
                vm_dpdk.conf
        README
        debug_dump.sh

- nxp/build_dpdk.sh
  is a script for compilation of DPDK for various platforms
  and modes (shared, static, debug). This can be executed once CROSS and other
  environment variables are set. Help can be obtained by executing with '-h'
  argument.

- nxp/disable_services.sh
  is a script to disable all services on a Ubuntu 16.04 stock rootfs. These
  services are known to impact DPDK performance, specially when the Core 0
  is used for DPDK I/O.

- nxp/dpaa
  folder contains all DPAA platform specific extra files. This includes the
  configuration files (XMLs) and Policy files.

- nxp/dpaa2
  folder contains all DPAA2 platform specific extra files. This includes the
  'dynamic_dpl.sh' for creating a container, as well as 'destroy_dynamic_dpl.sh'
  for destroying the creating container.

- debug_dump.sh
  is a scripts to check the system config. it generate the output in a file.

================================================================================

Applications validated DPAA/DPAA2 Platforms
-------------------------------------------

 1. l2fwd
 2. l3fwd
 3. l2fwd-crypto
 4. l2fwd-keepalive
 5. link_status_interrupt
       (link_status_interrupt -c 0xf -n 1  --log-level=8  -- -p 0x30 -q 1 -T 30)
 6. ip_fragmentation
 7. ip_reassembly
 8. ipv4_multicast
 9. ipsecgw
 10. cmdline
 11. timer
 12. vhost
 13. ethtool
 14. l3fwd-acl
 15. skelton
 16. rxtx_callback

================================================================================

Running DPDK Applications Without Root Privileges
===================================================

On root user:
--------------------

1. Edit /etc/security/limits.conf file with the following to adjust the resource limits needed to ensure normal DPDK operation. Reboot the board after saving the changes.

       #<domain>       <type>          <item>          <value>
       username        soft            memlock         unlimited
       username        hard            memlock         unlimited
       username        hard            nofile          4096
       username        hard            locks           unlimited

2. source /usr/local/dpdk/dpaa2/dynamic_dpl.sh dpmac.1 dpmac.2
3. The following permissions should be adjusted:
	chmod -R 777 /run/user
	chmod -R 777 /dev/hugepages
	chmod -R 777 /dev/vfio

On non-root user:
-------------------

1. export DPRC=dprc.2
2. Execute the following command to run the 'l2fwd' example application:
       ./dpdk-l2fwd -c 0x3 -n 1 -- -p 0x3 -q 1

================================================================================

Environment configuration variables
------------------------------------

Common for DPAA1 & DPAA2
------------------------
1. NXP_CHRT_PERF_MODE		Internally sets each I/O thread to have chrt priority as 90.


DPAA1:
------
1. DPAA_RX_TAILDROP_THRESHOLD - Size of the Rx queue taildrop, in packets. Default is 256.
				If set to 0, this would disable taildrop.
				(Only applicable for ethernet devices)

2. DPAA_TX_TAILDROP_THRESHOLD - Size of the Tx queue taildrop, in packets. Default is 0.
				If set to 0, this would disable taildrop.
				(Only applicable for ethernet devices)

3. DPAA_SEC_DP_DUMP_LEVEL	SEC datapath debug messages dump level.
				0 - No dump.
				1 - dump only sec error.
				2 - dump sec error along with useful debugging
				    information.
				By default, 1 is configured.

DPAA2:
------

1. DPAA2_TX_CGR_OFF		Disable the TX congestion control - i.e. infinite size of TX queues
				By default TX congestion control is enabled. Threshold is
				set to number of tx descriptors configured by application.

2. DPAA2_RX_TAILDROP_OFF	Disable RX taildrop.
				By default RX taildrop is enabled. Threshold is set to
				number of rx descriptors configured by application.

3. DPAA2_PARSE_ERR_DROP		Start dropping the error packets in hardware (parse errors).
				By default, these (parse) error packets are not dropped by hardware.

4. DPAA2_PORTAL_INTR_THRESHOLD	Portal Interrupt threshold w.r.t number of packets for epoll.
				By default, it is set to 3.

5. DPAA2_PORTAL_INTR_TIMEOUT	Portal interrupt timeout w.r.t time for no packet received.
				By default, it is set to 255(0xFF).

6. DPAA2_HOST_START_CPU		Define the CPU id for the virtual m/c CPU - so that right
				QMAN HW stashing can be configured.

7. DPAA2_NO_PREFETCH_RX		Disable prefetch RX mode - better latency and allows different
				size of pull request(number of packets to be received) in each
				RX call.
				By default, prefetch RX mode is enabled.

8. DPAA2_DATA_STASHING_OFF	Switch off data stashing when this flag is enabled. This gives
				better performance on LS1088 with number of cores >= 4.

9. DPAA2_LOOPBACK		QBMAN based loopback. send the received fd back without touching
				the payload.
				By default, loopback mode is disabled.

10. DPAA2_STRICT_ORDERING_ENABLE  Enable strict ordering when using ordered queues.
				  By default, loose ordering is enabled while using
				  ordered queues.

11. DPAA2_SEC_DP_DUMP_LEVEL	SEC datapath debug messages dump level.
				0 - No dump.
				1 - dump only sec error.
				2 - dump sec error along with useful debugging
				    information.
				By default, 1 is configured.

12. DPAA2_TX_CONF		Enable TX confirmation mode.
				By default, TX confirmation mode is disabled.

13. DPAA2_ENABLE_ERROR_QUEUE	Enable RX error queue for error packets. All the error packets
				are received on this queue instead of getting discarded/received
				with normal flow.
				By default, checksum error and parse error packets are received
				with normal flow, others are discarded.

14. DPAA2_ENABLE_SOFT_PARSER	Enable Soft parsing of RX packets by H/W(MC). Information
				related to a packet is shared by H/W along with packet.
				By default, soft parsing is disabled.

15. DPAA2_FLOW_CONTROL_LOG	Enable flow control logs(for debugging).
				By default, flow control logs are disabled.

16. DPAA2_FLOW_CONTROL_MISS_FLOW  Set queue id for packets not bound for any queue.
				  Set it to -1 to drop the missed flow packets.
				  By default, it is set to queue 0.

================================================================================

================================================================================

Devargs
-------

Note: Devargs can only be passed via '-w', -'b' or '--vdev' option

DPAA2:
-----

1. fslmc:dpni.<x>,drv_loopback=1	Enable low level loopback in the driver

2. fslmc:dpni.<x>,drv_no_prefetch=1	Disable prefetching of packet pull command
					which is issued  in the previous cycle.

3. fslmc.dpni.<x>,drv_tx_conf=1		Enable Tx confirmation mode for a given interface
================================================================================

==========================================================================
Execution steps to run Virtio_user as Exceptional Path
==========================================================================

The below commands assume 2 ports - where port 0 is the DPAA interface and port 1 is the tap interface, such that the traffic from DPAA interface is forwarded to tap interface and vice-versa.

Sending traffic from physical interface to tap interface
--------------------------------------------------------------------------------------

        1. Run testpmd
                ./dpdk-testpmd -l 2-3 -n 1 --vdev=virtio_user0,path=/dev/vhost-net,queues=2,queue_size=1024 -- -i --txq=2 --rxq=2 --txd=1024 --rxd=1024
        2. Start testpmd
                testpmd> port start all
                testpmd> start
        3. Enable tap interface on kernel
                ifconfig tap0 up
        4. Send traffic with mac address of tap interface from spirent (i.e. on port 0 of Board)
        5. Check the traffic on tap interface using the following kernel command
                tcpdump -i tap0

Sending traffic from tap interface to physical interface
--------------------------------------------------------------------------------------

        1. Run testpmd
                ./dpdk-testpmd -l 2-3 -n 1 --vdev=virtio_user0,path=/dev/vhost-net,queues=2,queue_size=1024 -- -i --txq=2 --rxq=2 --txd=1024 --rxd=1024
        2. Start testpmd
                testpmd> port start all
                testpmd> start
        3. On kernel:
			a. Configure IP address and enable tap interface
				ifconfig tap0 <ip_address0> up
			b. Set static ARP to send ping packets
				arp -s <ip_address1> <mac_address_of_physical_port>
			c. ping <ip_address1>
        4. Check and validate packets received on spirent


Running DPDK Applications Without Root Privileges
===================================================

On root user:
--------------------

1. Edit /etc/security/limits.conf file with the following to adjust the resource limits needed to ensure normal DPDK operation. Reboot the board after saving the changes.

       #<domain>       <type>          <item>          <value>
       username        soft            memlock         unlimited
       username        hard            memlock         unlimited
       username        hard            nofile          4096
       username        hard            locks           unlimited

2. source /usr/local/dpdk/dpaa2/dynamic_dpl.sh dpmac.1 dpmac.2
3. The following permissions should be adjusted:
	chmod -R 777 /run/user
	chmod -R 777 /dev/hugepages
	chmod -R 777 /dev/vfio

On non-root user:
-------------------

1. export DPRC=dprc.2
2. Execute the following command to run the 'l2fwd' example application:
       ./dpdk-l2fwd -c 0x3 -n 1 -- -p 0x3 -q 1 -P


======================================================================================================================================================

Minimum NXP LSDK version supported: LSDK2004 (https://lsdk.github.io/)
DPDK base version used: Release 20.11
More info on DPDK :  www.dpdk.org

NXP contact: hemant.agrawal@nxp.com, dpdk-team@nxp1.onmicrosoft.com
