============================
R E A D M E    N O T E S
============================
QLogic qfle3i VMware ESX Native Driver for ESXi6.5/6.7
Copyright (c) 2015-2017 QLogic Corporation
Copyright (c) 2016-2018 Cavium Inc.
All rights reserved

Table of Contents
=================
1.  Introduction
2.  Using the Driver
3.  Driver Parameters
4.  Feature support
5.  Statistics
6.  Features under Development.
7.  Known Issues
8.  Contacting Support

1.   Introduction
     =============
This file describes the qfle3i Native VMware ESXi iSCSI driver for QLogic 57XXX based 10Gb
PCIE Converged Network Controllers.

qfle3i is a native driver intended to replace bnx2i.

OS Support:

   * ESXi6.5
   * ESXi6.7


2.   Using the Driver
     =================

Installing the Driver:
---------------------
You can use the driver zip bundle .zip or .vib to update existing driver or install new driver. Unzip the driver installer zip bundle and extract .vib file from it.

   The .vib or .zip file may be installed directly on an ESX server from the command line, or through the VMware Update Manager (VUM).

COMMAND LINE INSTALLATION
-------------------------

New Installation
----------------

   For new installs, you should perform the following steps:
   1. Copy the VIB or ZIP to the ESX server.  Technically, you can place the file anywhere that is accessible to the ESX console shell, but for these instructions, we'll assume the location is in '/tmp'.

   Here's an example of using the Linux 'scp' utility to copy the file
   from a local system to an ESX server located at 10.10.10.10:

#scp qfle3i-1.0.0.2-1vmw.650.0.0.7044253.x86_64.vib root@10.10.10.10:/tmp

2. Issue the following command (full path to the VIB must be specified):
   #esxcli software vib install -v {VIBFILE}

   In the example above, this would be:

   #esxcli software vib install -v /tmp/qfle3i-1.0.0.2-1vmw.650.0.0.7044253.x86_64.vib

In case you are installing using zip:
#esxcli software vib install -d /tmp/qfle3i-1.0.0.2-1vmw.650.0.0.7044253.x86_64.zip

Note: Depending on the certificate used to sign the VIB, you may need to change the host acceptance level.  To do this, use the following command:

   #esxcli software acceptance set --level=<level>

   Also, depending on the type of VIB being installed, you may have to put ESX into maintenance mode.  This can be done through the VI Client, or by adding the '--maintenance-mode' option to the above esxcli command.

Upgrade Installation
--------------------

The upgrade process is similar to a new install, except the command that should be issued is following:

   #esxcli software vib update -v /tmp/qfle3i-1.0.0.2-1vmw.650.0.0.7044253.x86_64.vib
Or

#esxcli software vib update -d /tmp/qfle3i-1.0.0.2-1vmw.650.0.0.7044253.x86_64.zip

VUM INSTALLATION
----------------

   The VMware Update Manager (VUM) is a plugin for the Virtual Center Server (vCenter Server).  You can use the VUM UI to install a VIB by importing the associated offline bundle package (a ZIP file that contains the VIB and metadata).  You can then create an add-on baseline and remediate the host(s) with this baseline. Please refer the vCenter Server documentation for more details on VUM.

Removing the Driver:
--------------------

   Removing the driver package can be accomplished using the following command:

   #esxcli software vib remove --vibname <vib-name>

   Eg. esxcli software vib remove --vibname qfle3i

Uninstalling Driver:
-------------------

Following sequence must be followed to gracefully uninstall qfle3i driver:

[If SAN datastore has been created/mounted]
* esxcli storage filesystem unmount -l <datastore>

* esxcli storage core device set  -f --state=off -d <device-name>
eg: esxcli storage core device set  -f --state=off -d
	t10.94544500000000006F7093495932C12C08CCFC5D6B4BEC39

* esxcli iscsi session remove -A vmhbaX
eg: esxcli iscsi session remove -A vmhba32

* esxcli storage core claiming unclaim -t driver -D qfle3i

* vmkload_unload -u qfle3i



3.   Driver Parameters
     ==================

The qfle3i driver settings can be queried using command:

#esxcli system module parameters list -m qfle3i
Or
# esxcfg-module -i qfle3i
 Parameters:
  qfle3i_chip_cmd_max: int
   Max IOs queued to chip.
   Default: 24
  qfle3i_esx_mtu_max: int
   Max MTU Supported for Offload Sessions
   Default: 9000
   Valid Range: 1500-9000
  qfle3i_max_sectors: int
   Maximum sectors supported by the driver
	Valid range: 64 - 256. -1 will set it to default value.
	The default value are 256 for 10G NIC, 127 for 1G NIC.

  qfle3i_max_task_pgs: int
   Maximum pages allocated for iSCSi tasks per connection
    Default: 2
	Valid range: 2 - 8
  qfle3i_nopout_when_cmds_active: int
   iSCSI NOOP even when connection is not idle
   Default: 1
   Valid range: 0-1
  cmd_cmpl_per_work: int
   Number of CQE's processed per work
   Default: 256
  en_hba_poll: int
   Enable HBA poll timer
   Default: 0
  en_tcp_dack: int
   Enable TCP Delayed ACK. This parameter is enabled by default.
   Enabling TCP Delayed ACK helps improve network performance by
  combining several ACKs in a single response.
  Default: 1
   Note: It has been observed that certain iSCSI targets do not handle
  ACK piggybacking.For those targets, if this feature is turned on,
  the host will not be able to login to the target. If such behavior is
  observed, it is recommended that delayed Ack feature should be disabled.
  error_mask1: uint
   Config FW iSCSI Error Mask #1
   Default: 0
  error_mask2: uint
   Config FW iSCSI Error Mask #2
   Default: 0
  event_coal_div: int
   Event Coalescing Divide Factor.
   Default: 1
  event_coal_min: int
   Event Coalescing Minimum Commands
   Default: 24
  ooo_enable: int
   Enable TCP out-of-order support
   Default: 1
   Valid Range: 0 (disable) - 1 (enable)
  qfle3i_debug_level: int
   Bit mask to enable/disable debug logs
   Default: 0x0
	DEFAULT_LEVEL: 0x001
	Initalization: 0x002
	Conn Setup   : 0x004
	TMF          : 0x008
	iSCSI NOP    : 0x010
	CNIC IF      : 0x020
	ITT CLEANUP  : 0x040
	CONN EVT     : 0x080
	SESS Recovery: 0x100
	Internal     : 0x200
	IO Path      : 0x400
	APP INTERFACE: 0x800

  rq_size: int
   Configure RQ size
   Default: 16
  sq_size: int
   Configure SQ size
   Default: 128
  tcp_buf_size: int
   TCP send/receive buffer size
   Default: (64 * 1024)
  time_stamps: int
   Enable TCP TimeStamps
   Default: 1
   Valid range: 0 (disable) - 1 (enable)

The qfle3i driver settings can be changed using:

#esxcli system module parameters set -m qfle3i -p <param>=<value>

Or

#esxcfg-module -s <parameter>=<value> qfle3i

4.   Feature Support
     ================

1. Single Function Mode
2. IPv4
3. Jumbo MTU
4. Scatter Gather IO
5. Multi Function Support
6. SCSI Error Handling
7. Link Disruptions
8. OOO Support
9. IPv6 Support
10. Cleanup Path

5.   Statistics
     ===========

Driver supports multiple statistics that can be fetched using below command:
<TBD>

6.   Features under development
     ==========================

1. ARP Redirect Handling

7.   Known Issues
     ============
 - Login issues when 'Delayed Ack' is enabled
   Due to an internal Qlogic timer issue, we fail to login to some targets
  like 3Par and EMC. Hence a workaround is recommended to disable “delayed-ack”
  using module param with below mentioned command.
    esxcfg-module -s 'en_tcp_dack=0' qfle3i
  This workaround is needed only till the timer issue is fixed.

 - Port parity error recovery support:
	In case of hw error eg: parity error, the iscsi port will clear all iscsi
	contexts and connections, but remain in FAILED state.

8.   Contacting Support
     ===================

Please feel free to contact your QLogic approved reseller or QLogic
Technical Support at any phase of integration for assistance. QLogic
Technical Support can be reached by the following methods:

 Web:    http://support.qlogic.com
 E-mail: support@qlogic.com


(c) Copyright 2016. All rights reserved worldwide. QLogic, the QLogic
logo, and the Powered by QLogic logo are registered trademarks of
QLogic Corporation. All other brand and product names are trademarks
or registered trademarks of their respective owners.
