Search This Blog

Tuesday 13 November 2012

Windows 2012 - How to configure Multi Path iSCSI I/O

This is how to configure Multi Path I/O for iSCSI on Windows 2012 Server. I want to use this for our Hyper-V implementation to increase through put and redundancy.

Setup iSCSI NICs

media_1352814175635.png

In this server I have eight NICS, I have chosen to use two NICS for iSCSI and here you can see I have chosen to use one onboard Broadcom NIC and one PCI-e slot Intel NIC. Each NIC is configured with an IP address in the subnet of the storage network. In this case it is

10.12.0.40 255.255.255.0
10.12.0.41 255.255.255.0

The SAN is a HP MSA P2000 G3 iSCSI LFF and I have configured the Host NICS as

10.12.0.21 255.255.255.0
10.12.0.22 255.255.255.0
10.12.0.23 255.255.255.0
10.12.0.24 255.255.255.0
10.12.0.25 255.255.255.0
10.12.0.26 255.255.255.0
10.12.0.27 255.255.255.0
10.12.0.28 255.255.255.0

NIC Configuration

media_1352814185588.png

On each NIC you can remove services that are not required for iSCSI so I have unchecked Client for Microsoft Networks and File and Printer Sharing for Microsoft Networks.

Set IP Address

media_1352814192494.png

I will be using IP v4 for this implementation.

Confirm IP Address

media_1352814203472.png

Use static IP addresses to reduce need for DHCP and network overhead for that protocol. You do not need a gateway if the storage network is not to be routable. Each NIC needs to not use DNS to again improve performance so choose the Advanced option.

Do not register in DNS

media_1352814214374.png

Uncheck the option for Register this connections addresses in DNS. We do not want any IP from the iSCSI network in DNS.

Advanced NIC Settings

media_1352814239144.png

Each NIC has advanced settings and some relate to Power Management, we do not want any interruptions in the iSCSI network so we will change the advanced settings with the Configure option.

Power Management

media_1352814245991.png

Uncheck the option Allow the computer to turn off this device to save power.

Add MPIO Role

media_1352814312820.png

Now we will add the Multi Path Input Output (MPIO) role to the server so that we can use MPIO. From the Server Management dashboard choose the Manage Add Roles and Features option.

Add Features

media_1352814331835.png

Follow through the add roles and features and then at the Select Features option choose Multipath I/O. In this example I have already installed this feature which is why the (Installed) is displayed. The server will now installt he MPIO feature.

MPIO Tools

media_1352814339190.png

Once the feature is installed you can then choose Tools MPIO from the Server Management Dashboard.

MPIO Properties - Immeadiate Reboot Required

media_1352814348070.png

In the MPIO dialog choose the Discover Multi-Paths tab and then check the Add support for iSCSI devices option. The server will now require an immeadiate reboot so be prepared.

iSCSI Initiator

media_1352814363133.png

Now the server has rebooted we are ready to setup iSCSI, this is done from the Server Management and Tools, iSCSI Initiator.

Connecting to a Target

media_1352814377613.png

iSCSI works be connecting to a Target, the target is most likely a disk SAN or similar, in our case it is the HP MSA P2000 G3 iSCSI SAN. A target is an IP address that is configured on the iSCSI port on the SAN. 10.12.0.20 is the first IP address assigned to my Controller A iSCSI A1 port so I choose the Quick Connect option.

Quick Connect

media_1352814386418.png

The quick connect will now communicate to the SAN using the iSCSI NIC on the server and the iSCSI port on the SAN. It negotiates and we see under the Discovered Targets section the IQN of the SAN. You can see in the IQN name the hp:storage.p2000 text, this is part of the IQN name of our SAN. You can check this information on your iSCSI Storage Device as these will be different across manufacturers. Click Done to return to the iSCSI Initiator.

Add the first Multi Path

media_1352814405756.png

Select the target and then click on Properties to add the next path to the iSCSI Storage Device.

Sessions

media_1352814432848.png

This dialog will show the existing sessions to the iSCSI Storage Device, we have only added one session so far so we will only have one path to the iSCSI Storage Device and if we removed the network cable for the iSCSI NIC we would lose connection to the target. What we want is to be able to lose one connection and know that the second iSCSI NIC can carry on the iSCSI traffic. So choose Add Session to add the second iSCSI session.

Connect to Target

media_1352814440662.png

When you add the new session you are asked do you want to use Multi Path,, check the Enable Multi-Path option and then choose the Advanced option.

Advanced Settings

media_1352814449116.png

In this dialog we are going to choose which type of adapter we are going to use, as we have no Hardware Based Adapters (HBA) we will use the Microsoft iSCSI Initator which is software based so select this from the Local Adapter dropdown.

Intiator IP

media_1352814455490.png

From the Initiator IP dropdown choose the IP Address you have assigned to the second iSCSI NIC, in this case this is the IP address 10.12.0.40. This will now connect the second iSCSI NIC to our target so that both iSCSI NICS can communicate with the iSCSI Storage Device. Choose OK.

Confirm MPIO for each Session

media_1352814484856.png

A session will now be created with a long GUID, check the new session and then click on the Devices button to see what devices are connected in this session. We are looking to see two devices, one for each of the iSCSI Target IP addresses.

Devices

media_1352814491811.png

I recommend that you create a LUN on your iSCSI Storage Device in advance as you then have a device to see as connected, here I can see the disk I have created on LUN 0. I now choose the MPIO button.

Device Details

media_1352814619540.png

This displays the MPIO details and the Load Balance Policy. This is the way that the MPIO trys to communicate with the iSCSI Target, we would like it to Round Robin. This means that the first IP address is sent a packet and then the next and so on until the packets come round to the start again. The benefit here is that all paths get used and you can have multiple packets sent at once so you get better performance. If a path is down due to a cable failure or swich failure the round robin notices this and ignore the path and sends the packet on the next active path. So you have a high performance and redundant iSCSI infrastructure.

To see the IP addresses used for a path, click on a path and then choose the Details button.

MPIO Details

media_1352814625966.png

In the details of the path you can see the Source and Target IP address details. Here we can see the Source is the iSCSI NIC on the server 10.12.0.41 and the target is the IP address of Controller A iSCSI A1 port 10.12.0.20.

MPIO Details on second path

media_1352814641581.png

On the second path you can see the Source is now the other iSCSI NIC on the Server and the Target is the Controller A iSCSI A1 port so we have two paths now to this target.

Confirm the MPIO

media_1352814656390.png

You can confirm the MPIO in use with a command line tool called mpclaim. Here I have ran the command mpclaim -v c:\config.txt This will output the MPIO configuration in verbose mode to a text file so it is easy to read.

Text File Output

media_1352814690888.png

I open the Config.txt file and I can see the MPIO states we have 2 Paths so I know the paths I have created are live. So all I need now is to go do this all again for each target IP addresss on my iSCSI Storage Device to built the multiple paths.

14 comments:

  1. Great stuff! I was struggling with getting this to work for a couple of days.

    ReplyDelete
  2. No worries Steve, glad it help to you!

    ReplyDelete
  3. Hi Kyle, I have a P2000 also, Currently i just have 2 iscsi connections from one server to the p2000 i am only getting about 150mb/s throughput, to test the fault tolerance i pull one cable it keep copying files, put it back in wait 30 secs and pull the other cable. when i pull the 2nd cable the copying hangs up for 30secs and then continues. My setup seems identical to yours except my connections show as Active/UNOPTIMIZED. is this why its not working properly for me? how do it get it to show as Active/Optimized? Keep in mind I do not have any switches in use for the iscsi connections. This is a great article by the way.

    ReplyDelete
  4. any ideas on this? also on my setup it shows the max connections as 1. could you tell me if yours shows more than that?

    ReplyDelete
  5. I ran that mpclaim command on my vmware virtualized windows 2012 servers and they both crashed. yes, i ran it on both to see if i could repeat the crash. The crash line states: DRIVER_IRQL_NOT_LESS_OR_EQUAL

    Good article otherwise :)

    I still cant find anything useful on MCS though.


    ReplyDelete
  6. Thank you for this!! 4000 miles away from home working on a new iSCSI environment and this was a life saver.

    ReplyDelete
  7. Hello Kyle,
    Thanks for this great stuff ...
    It's help me a lot !!

    Did You install or configure something on the target server (i think not) ?
    In the case where i've got TWO nic on my target ... did i have in this case ... to install or configure something on the target server (install the feature MPIO too ?) ??

    Thanks!
    Patrick

    ReplyDelete
  8. Thanks for sharing. Really useful.

    ReplyDelete
  9. Hi, on my system there was another step that I had to complete before I was able to configure MPIO. After installing MPIO, I had to run mpiocpl.exe, and within that dialog, specifically add MPIO capability for iSCSI. After a reboot, the MPIO functionality was available.

    ReplyDelete
  10. Hi Kyle Heath,

    I have just read your article. I realized that a missing at here.

    You said. This is IP address NIC card

    10.12.0.40 255.255.255.0
    10.12.0.41 255.255.255.0

    IP of P2000G3

    10.12.0.21 255.255.255.0
    10.12.0.22 255.255.255.0
    10.12.0.23 255.255.255.0
    10.12.0.24 255.255.255.0
    10.12.0.25 255.255.255.0
    10.12.0.26 255.255.255.0
    10.12.0.27 255.255.255.0
    10.12.0.28 255.255.255.0

    But at the end of the article . I saw 2 path the following:

    Source Portal Target Portal

    10.12.0.41/48064 10.12.0.20/3260

    10.12.0.40/48064 10.12.0.20/3260


    IP address 10.12.0.20. Why do you have this IP address. Can you explain or this is a IP Virtual.

    Thanks,

    ReplyDelete
  11. great stuff!
    i have a question, i've had an older HP MSA2012i NAS running for about 8 years now, and i have it setup so the two iscsi networks are on 2 separate subnets. have i been doing this all wrong?

    Networks
    iscsi-A 10.1.1.0 /24
    iscsi-B 10.2.1.0 /24
    LAN 192.168.1.0 /24

    Controller A
    target 10.1.1.1 /24
    target 10.1.1.6 /24

    Controller B
    target 10.2.1.1 /24
    target 10.2.1.6 /24

    host
    10.1.1.8 /24
    10.2.1.8 /24

    now i'm setting up a new server 2012 R2 and i'm starting to look at this again and i'm seeing setups being configured where both controllers (and entire iscsi network) are on 1 meshed network.

    I'm also teaming NICs for the iscsi connections (that's ok to do right?)

    and are the LUNs necessary? i'v been using 0's for all 2 iscsi connections to my 2 servers.

    ReplyDelete
  12. Thank you so much for sharing the article as I found this blog while doing a search on IT solutions Perth.

    ReplyDelete
  13. Just Super...great, this will help lot many folks

    ReplyDelete