Tuesday, November 17, 2020

How to use DiskSpd to simulate Veeam Backup & Replication disk actions

This HOW-TO contains information on how to use Microsoft© DiskSpd to simulate Veeam Backup & Replication disk actions to measure disk performance.

The test file created by DiskSpd does not contain any diagnostic information and must be removed manually after testing has concluded. All diagnostic information regarding the performance test is displayed in the command line.

Active full or forward incremental

C:\diskspd> diskspd.exe -c25G -b512K -w100 -Sh -d600 D:\testfile.dat

-w100 indicates 100% writes and 0% reads. Sequential I/O is used by default.

Reverse incremental

C:\diskspd> diskspd.exe -c100G -b512K -w67 -r4K -Sh -d600 D:\testfile.dat

-w67 indicates 67% writes and 33% reads to simulate 2 write and 1 read operations that happen in reverse incremental backup jobs.

-r4K enables random I/O that are 4KB aligned, for a more realistic simulation.

After the test has finished, take Total IO MB/s from the results and divide it by 3. This is because for every processed block Veeam needs to do 3 I/O operations, thus the effective speed is 3 times slower.

Transforms, merges, and other synthetic operations

Includes transformation of incrementals to rollbacks, merge operations in forever forward incremental backups and backup copy jobs, and creation of synthetic full backup files and GFS points.

C:\diskspd\x86fre> diskspd.exe -c100G -b512K -w50 -r4K -Sh -d600 D:\testfile.dat

-w50 indicates 50% writes and 50% reads to simulate reading data from one file and writing that data into another (or in the case of transform, reading the same number of blocks from two files as are written to two other files).

-r4K enables random I/O that are 4KB aligned, for a more realistic simulation.

After the test has finished, take Total IO MB/s from the results and divide it by 2 (4 for transform to rollbacks). This is because for every processed block Veeam needs to do 2 I/O operations, thus the effective speed is 2 times slower. For transform to rollbacks, each block must be read out of the full backup file and written into the rollback before the corresponding block can be read out of the incremental and written into the full, which results in 4x I/O.

To estimate an expected time to complete the synthetic operation, in seconds:

For synthetic full backup and GFS points: divide the expected size of the new full backup file (typically the same as previous full backup files) by the effective speed.

For all other synthetic operations, add the sizes of all of the incremental files which will be merged or transformed, and then divide the resulting sum by the effective speed. Typically only the oldest incremental file is merged, whereas all incremental files are transformed to rollbacks.

Slow restore or Surebackup

This is typically when you're restoring from deduplication appliances with sub-optimal settings. As a workaround in case of slow restore, manually copy the backup files elsewhere (e.g. Veeam server), import and restore from there.

Worst case scenario where the backup file is heavily fragmented inside, which implies a lot of random read I/O:

C:\diskspd\x86fre> diskspd.exe -b512K -r4K -Sh -d600 \\nas\share\VeeamBackups\Job\Job2014-01-23T012345.vbk

-r4K enables random I/O that are 4KB aligned, for a more realistic simulation.

Best case scenario where the backup file is not fragmented inside (no parallel processing), which implies linear read I/O: 

C:\diskspd\x86fre> diskspd.exe -b512K -Sh -d600 \\nas\share\VeeamBackups\Job\Job2014-01-23T012345.vbk

In both cases you need to pick an existing .vbk file as the target. Only read operations will be performed.

Direct disk access speed

C:\diskspd\x86fre> diskspd.exe -Sh -d600 #X

Where X is the number of disk that you see in Disk Management.

This test will not overwrite any data, it is a safe test, and it works for Offline disks too. You can simulate and measure maximum possible reading speed in SAN or hot-add modes, however this of course will not take any VDDK overhead into account.

Tuesday, October 6, 2020

Changing Qlogic Queue Depth Windows

 The maximum SCSI queue depth value, as determined by the Microsoft port driver developer, is 254. QLogic sets the default value to 32. The Microsoft storport.sys driver has total control of the depth of the device queue; QLogic Miniport merely sets the value using the predefined Storport interface call and procedure as determined by Microsoft.

 
Use the following procedure to change the queue depth parameter:
 
1. Click

Start, select Run, and open the REGEDIT/REGEDT32 program.

2. Select

HKEY_LOCAL_MACHINE and follow the tree structure down to the QLogic driver as follows:
HKEY_LOCAL_MACHINE
SYSTEM
CurrentControlSet
Services
ql2300
Parameters
Device

3. Double-click DriverParameter to edit the string, for example:

DriverParameter REG_SZ qd=32

4. Add "qd=" to the "Value data:" field. If the string "qd=" does not exist, append it to end of the string with a semicolon, for example:

;qd=<value>

5. Enter a value up to 254 (0xFE).

The default value is 32 (0x20).

6. Click OK.
7. Exit the Registry Editor, then shutdown and reboot the system.

Thursday, September 24, 2020

Reset iLO4 certificate

 During troubleshooting of issue at my customer in OneView, I have noticed that there is a SSL certificate issue with one of iLO4 of BL460gen9 ... 

After seraching HPE support page, I have stumbled to this article:

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=emr_na-a00042194en_us

Advisory: (Revision) HP Integrated Lights-Out (iLO) - iLO 3 and iLO 4 Self-Signed SSL Certificate May Have an Expiration Date Earlier Than the Issued Date

The HP Integrated Lights-Out 3 (iLO 3) and HP Integrated Lights-Out 4 (HP iLO 4) may have a self-signed Secure Socket Layer (SSL) certificate containing the incorrect year for the expiration date as follows:




RESOLUTION

Import a Trusted SSL Certificate signed by a Certification Authority (CA) into iLO 3 and iLO 4. Refer to the iLO User Guide for more details.

The HP iLO 4 User Guide is located at the following URL: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c03334051 The HP iLO 3 User Guide is located at the following URL: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c02774507

If a Certification Authority is not available, perform either of the following steps as a workaround:

Downgrade the iLO firmware

  1. Downgrade the iLO 3 firmware to Version 1.50 or the iLO 4 firmware to Version 1.13.
  2. Boot the server and press F9 to enter the ROM-Based Setup Utility (RBSU).
  3. In the RBSU Date and Time Menu, set the system date to 01-01-2013.
  4. Save changes and exit the RBSU.
  5. Wait until the server completes the Power-On Self-Test (POST).
  6. Log into iLO, verify the iLO date is January 1, 2013, and force iLO to generate a new self-signed SSL certificate by either changing the iLO hostname or restoring iLO to factory defaults.
  7. After iLO has a new self-signed SSL certificate containing a valid expiration date, the iLO 3 firmware can be upgraded to Version 1.55 and the iLO 4 firmware can be upgraded to Version 1.20.

OR

iLO 4 firmware version 2.55 (or later)

The iLO 4 self-signed certificate can also be regenerated using a REST command that was added in iLO 4 firmware 2.55 (or later). To regenerate the self-signed certificate using this REST command use the following syntax:

DELETE https://{iLO}/redfish/v1/Managers/{item}/SecurityService/HttpsCert/

Below is an example using curl.

curl -X DELETE https://ilo.example.net/redfish/v1/Managers/1/SecurityService/HttpsCert/

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-length: 299
Content-type: application/json; charset=utf-8
Date: Sat, 20 Jan 2018 03:38:28 GMT
ETag: W/"78E0DBAE"
X-Frame-Options: sameorigin
X_HP-CHRP-Service-Version: 1.0.3

{"Messages":[{"MessageID":"iLO.0.10.ImportCertSuccessfuliLOResetinProgress"}],"Type":"ExtendedError.1.0.0","error":{"@Message.ExtendedInfo": [{"MessageID":"iLO.0.10.ImportCertSuccessfuliLOResetinProgress"}],"code":"iLO.0.10.ExtendedInfo","message":"See @Message.ExtendedInfo for mo re information."}}

The expiration date on the self-signed SSL certificate is targeted to be resolved in a future version of the iLO 3 and iLO 4 firmware.


So I have tried to use 2nd option, as I have 2.55+ firmware.

You will need CURL for windows, and some extra command parameters to add so that CURL delete process can work:

curl -X DELETE https://iLO_IP_ADDRESS/redfish/v1/Managers/1/SecurityService/HttpsCert/ -i --insecure -u username:password -L

How to use DiskSpd to simulate Veeam Backup & Replication disk actions

This HOW-TO contains information on how to use Microsoft© DiskSpd to simulate Veeam Backup & Replication disk actions to measure disk pe...