Welcome back! In this post, I’m getting into somehow different topic than my usual ones. Here, the subject matter that I’m diving into is storage traffic switching with Cisco MDS series switches. If you’re new to this subject, here are some basic introductions: First, Storage Area Network (SAN) is much like the well-known Local Area Network (LAN), with an exception that it carries storage traffic or SCSI commands among storage devices (target) and servers/clients (initiator). There are mainly two technologies that are used for SAN: Fibre Channel (FC) which utilizes fibre channel protocol and iSCSI which utilizes IP.

Lab Overview

Devices

In this lab, I’ve got two Cisco MDS 9148S switches, two Cisco UCS C240-M4 servers and two Storage appliances. With these devices in hand, I intend to setup a SAN topology using Fibre Channel. Although Cisco MDS 9148 series switches have 48 fibre channel ports, only the first 12 of them can acquire port license by default and the remaining ports have to be activated by purchasing additional licenses from Cisco. So, all of my cabling and configuration will be done on these first 12 ports of each switch.

Transceivers and Cabling

For cabling, I’m going to use SR-type SFP transceivers on all of my devices and multi-mode fiber cables for inter-device connectivity. I then connect all of my storage devices and servers to the Cisco MDS switches as follows:

Switch 1:

FC 1/1: Server 1 -> HBA 1 -> Port 1
FC 1/2: Server 2 -> HBA 1 -> Port 1
FC 1/3 : Storage 2 -> Controller A -> Channel 5
FC 1/4: Storage 2 -> Controller B -> Channel 5
FC 1/5: Storage 1 -> Controller A -> Channel 5
FC 1/6: Storage 1 -> Controller B -> Channel 5
FC 1/11-12: Switch 2 (ISL trunks)

Switch 2:

FC 1/1: Server 1 -> HBA 1 -> Port 0
FC 1/2: Server 2 -> HBA 1 -> Port 0
FC 1/3 : Storage 2 -> Controller A -> Channel 4
FC 1/4: Storage 2 -> Controller B -> Channel 4
FC 1/5: Storage 1 -> Controller A -> Channel 4
FC 1/6: Storage 1 -> Controller B -> Channel 4
FC 1/11-12: Switch 1 (ISL trunks)

Cisco MDS Configuration

Initial Setup

When we’re trying to configure a newly purchased Cisco MDS switch, we can simply utilize the setup utility provided by Cisco. To do so, we need to connect the device’s console to our laptop or PC and power on the device. Once the switch boots up, it prompts for the admin password and asks if we’d like to run the setup wizard, to which I answer Yes. Then, it asks for some more options, for which we can basically choose default values. You can refer to this Cisco documentation to learn about it.

 Switch 1 Config

After the cabling is completed and the initial configuration is done, let’s begin the further configuration by setting up the description in every connected switch ports and creating a port channel for aggregation of links fc1/11 and fc1/12.

LAB-MDS-1# conf t
LAB-MDS-1(config)# int fc1/1-12
LAB-MDS-1(config-int)# port-license acquire
LAB-MDS-1(config-int)# no shutdown
LAB-MDS-1(config-int)# int fc1/1
LAB-MDS-1(config-int)# switchport description ***UCS-2-Port-1***
LAB-MDS-1(config-int)# int fc1/2
LAB-MDS-1(config-int)# switchport description ***UCS-1-Port-1***
LAB-MDS-1(config-int)# int fc1/3
LAB-MDS-1(config-int)# switchport description ***Storage-2-Ctr-A-Channel-5***
LAB-MDS-1(config-int)# int fc1/4
LAB-MDS-1(config-int)# switchport description ***Storage-2-Ctr-B-Channel-5***
LAB-MDS-1(config-int)# int fc1/5
LAB-MDS-1(config-int)# switchport description ***Storage-1-Ctr-A-Channel-5***
LAB-MDS-1(config-int)# int fc1/6
LAB-MDS-1(config-int)# switchport description ***Storage-1-Ctr-B-Channel-5***
LAB-MDS-1(config-int)# int fc1/11
LAB-MDS-1(config-int)# switchport description ***ISL-Trunk-1***
LAB-MDS-1(config-int)# channel-group 1
LAB-MDS-1(config-int)# int fc1/12
LAB-MDS-1(config-int)# switchport description ***ISL-Trunk-2***
LAB-MDS-1(config-int)# channel-group 1

Let’s also create the port-channel that we’ve called on the interfaces fc1/11 and fc1/12.

interface vsan100
  no shutdown

interface port-channel1
  channel mode active
  switchport description ***Port-Channel-ISL-Trunk***
  switchport rate-mode dedicated

Then, let’s create the virtual SAN (VSAN) and associate the interfaces to this VSAN.

LAB-MDS-1(config)# vsan database
LAB-MDS-1(config-vsan-database)# vsan 1 suspend
LAB-MDS-1(config-vsan-database)# vsan 100 name "My-VSAN"

LAB-MDS-1(config-vsan-database)# vsan 100 interface port-channel1
LAB-MDS-1(config-vsan-database)# vsan 100 interface fc1/1-10

LAB-MDS-1(config)# system default zone default-zone permit
LAB-MDS-1(config)# system default zone distribute full
LAB-MDS-1(config)# zone mode enhanced vsan 100

Here I’ve created a VSAN with id 100 and included all of my available interfaces into it. Simultaneously, I also suspended vsan 1, so that it won’t remain being the native vsan.

Now, comes probably the most important task in SAN setup i.e. Zoning. Zoning allows us to control the traffic flowing between the devices, similar to the Access Control Lists in IP network. There are mainly two ways to setup zones in Cisco MDS: interface based zoning and WWN based zoning. When interface based zoning is done, only the member interfaces can communicate with each other. So, if the devices are connected on different ports of switch, the zoning might not work correctly. Whereas with WWN based zones, devices can only communicate with other devices with member PWWNs. Thus, it works correctly even when ports are changed, but doesn’t work if the ports or HBA cards on server or storage are changed. Here, I’m using PWWN based zoning. If you choose to use interface based zoning, that’s simply fine.

In this case, I’m also creating fcaliases for each PWWNs of devices so that I can identify and manage them easily. I’m then going to associate these fcaliases in the zone configuration.

fcalias name UCS-1-Port-0 vsan 100
 member pwwn 10:00:00:90:fa:de:f5:9a

fcalias name UCS-1-Port-1 vsan 100
 member pwwn 10:00:00:90:fa:de:f5:9b

fcalias name Storage-1-Ctr-A-Channel-5 vsan 100
 member pwwn 22:00:00:d0:23:05:43:2a

fcalias name Storage-1-Ctr-B-Channel-5 vsan 100
 member pwwn 22:00:00:d0:23:15:43:2a

fcalias name Storage-1-Ctr-A-Channel-4 vsan 100
 member pwwn 21:00:00:d0:23:05:43:2a

fcalias name Storage-1-Ctr-B-Channel-4 vsan 100
 member pwwn 21:00:00:d0:23:15:43:2a

fcalias name Storage-2-Ctr-A-Channel-4 vsan 100
 member pwwn 21:00:00:d0:23:05:6b:6f

fcalias name Storage-2-Ctr-A-Channel-5 vsan 100
 member pwwn 22:00:00:d0:23:05:6b:6f

fcalias name Storage-2-Ctr-B-Channel-5 vsan 100
 member pwwn 22:00:00:d0:23:15:6b:6f

fcalias name Storage-2-Ctr-B-Channel-4 vsan 100
 member pwwn 21:00:00:d0:23:15:6b:6f

fcalias name UCS-2-Port-1 vsan 100
 member pwwn 10:00:00:90:fa:81:56:4d

fcalias name UCS-2-Port-0 vsan 100
 member pwwn 10:00:00:90:fa:81:56:4c

fcalias name My-VSAN vsan 100

Now, I need to create zones to allow communication between each server and storage. That way, these devices will have their own separate communication path, which won’t be accessible to outsider devices.

zone name UCS1-SAN1 vsan 100
    member fcalias UCS-1-Port-0
    member fcalias UCS-1-Port-1
    member fcalias Storage-1-Ctr-A-Channel-5
    member fcalias Storage-1-Ctr-B-Channel-5
    member fcalias Storage-1-Ctr-A-Channel-4
    member fcalias Storage-1-Ctr-B-Channel-4

zone name UCS1-SAN2 vsan 100
    member fcalias UCS-1-Port-0
    member fcalias UCS-1-Port-1
    member fcalias Storage-2-Ctr-A-Channel-4
    member fcalias Storage-2-Ctr-A-Channel-5
    member fcalias Storage-2-Ctr-B-Channel-5
    member fcalias Storage-2-Ctr-B-Channel-4

zone name UCS2-SAN1 vsan 100
    member fcalias Storage-1-Ctr-A-Channel-5
    member fcalias Storage-1-Ctr-B-Channel-5
    member fcalias Storage-1-Ctr-A-Channel-4
    member fcalias Storage-1-Ctr-B-Channel-4
    member fcalias UCS-2-Port-1
    member fcalias UCS-2-Port-0

zone name UCS2-SAN2 vsan 100
    member fcalias Storage-2-Ctr-A-Channel-4
    member fcalias Storage-2-Ctr-A-Channel-5
    member fcalias Storage-2-Ctr-B-Channel-5
    member fcalias Storage-2-Ctr-B-Channel-4
    member fcalias UCS-2-Port-1
    member fcalias UCS-2-Port-0

At last, I need to create a zoneset belonging to the vsan 100 and containing the zones created above. And every time I make changes to the zones and commit the zoneset, the zones are synchronized in both of the switches via ISL link.

zoneset name Zoneset_1-V100 vsan 100
    member UCS1-SAN1
    member UCS1-SAN2
    member UCS2-SAN1
    member UCS2-SAN2

zone commit vsan 100
zoneset activate name Zoneset_1-V100 vsan 100

Switch 2 Config

conf t
int fc1/1-12
port-license acquire
no shutdown

int fc1/1
switchport description ***UCS-2-Port-0***

int fc1/2
switchport description ***UCS-1-Port-0***

int fc1/3
switchport description ***Storage-2-Ctr-A-Channel-4***
int fc1/4
switchport description ***Storage-2-Ctr-B-Channel-4***
int fc1/5
switchport description ***Storage-1-Ctr-A-Channel-4***
int fc1/6
switchport description ***Storage-1-Ctr-B-Channel-4***
int fc1/11
switchport description ***ISL-Trunk-1***
channel-group 1
int fc1/12
switchport description ***ISL-Trunk-2***
channel-group 1

interface vsan100
  no shutdown

interface port-channel1
  channel mode active
  switchport description ***Port-Channel-ISL-Trunk***
  switchport rate-mode dedicated

system default zone default-zone permit
system default zone distribute full
zone mode enhanced vsan 100

Since all of the fcalias, zones and zoneset information are synchronized to this switch from switch-1, we don’t need to do much setup on this switch.

Well, this is it! I hope you’ve learned something from this article. Please let me know of your suggestion and feedback in the Comments section below. See you later. Thank you!