Multihoming with OTV

Multihoming has always been of great interest in almost all technologies. This is primarily due to the fact that today's world never wants to accept down time. The great thing about OTV is that it is inherently ready for multihoming [redundancy / high availability]. This is where the real concept of 'Authoritative Edge Device' [aka AED] comes into the picture.

ISIS plays a major role here and does internal peering with the other edge device to provide the multihoming support. If you are aware of the basic OTV configuration, we usually have used the command 'otv site-identifier <x>'. ISIS ensures the connectivity of the site edge-devices via both the access and the via join-interface.
 

Edge devices on the same site almost replicate each other's configuration, apart from the join-interface IP addresses and physical interfaces being used by them. For better understanding let us consider the below topology and its respective configuration.

NOTE: OTV [unicast-core] configuration used

Topology:

Configuration:

 

S1ED1:

otv site bridge-domain 2
 otv isis hello-interval 3

otv site-identifier 0000.0000.0002

router ospf 100
 router-id 1.1.1.1

spanning-tree mode mst
spanning-tree extend system-id
spanning-tree mst configuration
 name multihome
 revision 1
 instance 1 vlan 2-4094

interface Port-channel11
 description "Connected to CORE via Ten 0/0/0 & Ten 0/0/1"
 ip address 10.1.1.1 255.255.255.0
 ip ospf 100 area 100

interface TenGigabitEthernet0/0/0
 description "Connected to CORE"
 no shutdown! added by me 
 no ip address
 channel-group 11 mode active

interface TenGigabitEthernet0/0/1
 description "Connected to CORE" 
 no shutdown                         ! added by me
 no ip address
 channel-group 11 mode active

interface GigabitEthernet0/0/0
 description "Connected to Switch - SW1" 
 no shutdown                         ! added by me
 no ip address
 negotiation auto
 service instance 2 ethernet
  encapsulation dot1q 2
  bridge-domain 2
 !
 service instance 11 ethernet
  encapsulation dot1q 11
  bridge-domain 11
 !
 service instance 12 ethernet
  encapsulation dot1q 12
  bridge-domain 12
 !
 service instance 4094 ethernet
  encapsulation untagged
  bridge-domain 4094

interface Overlay2
 no shutdown                         ! added by me
 no ip address
 otv join-interface Port-channel11
 otv use-adjacency-server 10.1.3.1 unicast-only
 otv adjacency-server unicast-only
 otv isis hello-interval 3
 service instance 11 ethernet
  encapsulation dot1q 11
  bridge-domain 11
 !
 service instance 12 ethernet
  encapsulation dot1q 12
  bridge-domain 12

S1ED2:

otv site bridge-domain 2
 otv isis hello-interval 3

otv site-identifier 0000.0000.0002

router ospf 100
 router-id 2.2.2.2

spanning-tree mode mst
spanning-tree extend system-id
spanning-tree mst configuration
 name multihome
 revision 1
 instance 1 vlan 2-4094

interface GigabitEthernet0/0/0
 description "Connected to CORE" 
 no shutdown                         ! added by me

 ip address 10.1.2.1 255.255.255.0
 ip ospf 100 area 100

interface GigabitEthernet0/0/1
 description "Connected to Switch - SW2" 
 no shutdown                         ! added by me
 no ip address
 negotiation auto
 service instance 2 ethernet
  encapsulation dot1q 2
  bridge-domain 2
 !
 service instance 11 ethernet
  encapsulation dot1q 11
  bridge-domain 11
 !
 service instance 12 ethernet
  encapsulation dot1q 12
  bridge-domain 12
 !
 service instance 4094 ethernet
  encapsulation untagged
  bridge-domain 4094

interface Overlay2
 no shutdown                         ! added by me
 no ip address
 otv join-interface GigabitEthernet0/0/0
 otv use-adjacency-server 10.1.3.1 unicast-only
 otv adjacency-server unicast-only
 otv isis hello-interval 3
 service instance 11 ethernet
  encapsulation dot1q 11
  bridge-domain 11
 !
 service instance 12 ethernet
  encapsulation dot1q 12
  bridge-domain 12

S2ED1:

otv site bridge-domain 3
otv site-identifier 0000.0000.0003

router ospf 100
 router-id 4.4.4.4

interface GigabitEthernet2/1/0
 description "Connected to CORE"
 no shutdown                         ! added by me
 ip address 10.1.3.1 255.255.255.0
 ip ospf 100 area 100

interface GigabitEthernet2/1/1
 description "Connected to Switch - SW3"
 no shutdown                         ! added by me
 no ip address
 negotiation auto
 service instance 3 ethernet
  encapsulation dot1q 3
  bridge-domain 3
 !
 service instance 11 ethernet
  encapsulation dot1q 11
  bridge-domain 11
 !
 service instance 12 ethernet
  encapsulation dot1q 12
  bridge-domain 12
 
interface Overlay2
 no shutdown                         ! added by me
 no ip address
 otv join-interface GigabitEthernet2/1/0
 otv adjacency-server unicast-only
 otv isis hello-interval 3
 service instance 11 ethernet
  encapsulation dot1q 11
  bridge-domain 11
  !
 service instance 12 ethernet
  encapsulation dot1q 12
  bridge-domain 12

SW1:

interface GigabitEthernet3/1
 description "Connected to S1ED1"
 no shutdown                         ! added by me
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 
interface GigabitEthernet3/2
 description "Connected to SW2"
 no shutdown                         ! added by me
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 
interface GigabitEthernet3/3
 description "Connected to VM1"
 no shutdown                         ! added by me
 switchport
 switchport access vlan 11
 switchport mode access

SW2:

interface GigabitEthernet4/1
 description "Connected to SW1"
 no shutdown                         ! added by me
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 no ip address
 
interface GigabitEthernet5/1
 description "Connected to S1ED2"
 no shutdown                         ! added by me
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 no ip address

SW3:

interface GigabitEthernet5/1
 description "Connected to S2ED1"
 no shutdown                        ! added by me
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 no ip address 

interface GigabitEthernet5/2
 description "Connected to VM2"
 no shutdown                       ! added by me
 switchport
 switchport access vlan 11
 switchport mode access
 no ip address
 media-type rj45

CORE:

router ospf 100
 router-id 3.3.3.3
 
interface Port-channel11
 description "Connected to CORE via Ten 1/0/0 & Ten 1/0/1"
 ip address 10.1.1.2 255.255.255.0
 ip ospf 100 area 100

interface TenGigabitEthernet1/0/0
 description "Connected to S1ED1"
 no shutdown                       ! added by me
 no ip address
 channel-group 11 mode active

interface TenGigabitEthernet1/0/1
 description "Connected to S1ED1"
 no shutdown                       ! added by me
 no ip address
 channel-group 11 mode active
 
interface GigabitEthernet0/0/1
description "Connected to S1ED2"
no shutdown                        ! added by me
ip address 10.1.2.2 255.255.255.0
ip ospf 100 area 100
negotiation auto

interface GigabitEthernet0/0/5
 description "Connected to S2ED1"
 no shutdown                       ! added by me
 ip address 10.1.3.2 255.255.255.0
 ip ospf 100 area 100
 negotiation auto

Very few additions come in when dealing with a multi-homed OTV setup which results in easier deployments. The configuration which comes in new is related to spanning-tree which plays its role on the Layer-2 [access] side of the edge device.

Also, from the configuration you can see the usage of port-channel as the join-interface for one of the connections between the ED and the core. Usage of port-channels is supported even on the L2 [access-interface] side.

There is one more point about multi-homing in OTV, and that is, the edge-devices always work in load-balancing mode and not in active-standby mode. We will look at this in a bit more detail in a while.

Before we send out a ping from VM1 to VM2, a few commands which would be useful when identifying peer edge-devices / remote edge-devices:

S1ED1 & S1ED2 are multi-homing edge-devices @S1
S2ED1 is a single-homed edge-device @S2

S1ED1#show otv isis neighbors
Tag Overlay2:
System Id      Type Interface   IP Address      State Holdtime Circuit Id
S1ED2          L1   Ov2         10.1.2.1        UP    8        S1ED1.01          
S2ED1          L1   Ov2         10.1.3.1        UP    7        S1ED1.01          
Tag Site:
System Id      Type Interface   IP Address      State Holdtime Circuit Id
0006.F685.4500 L1   OTV-Site                    UP    8        S1ED1.01          
S1ED1#
 
S2ED1#show otv isis neighbors
Tag Overlay2:
System Id      Type Interface   IP Address      State Holdtime Circuit Id
S1ED1          L1   Ov2         10.1.1.1        UP    2        S1ED1.01          
S1ED2          L1   Ov2         10.1.2.1        UP    7        S1ED1.01          
Tag Site:
System Id      Type Interface   IP Address      State Holdtime Circuit Id
S2ED1#

As seen above "Tag Site" entry is seen under S1ED1, but its empty in S2ED1, stating the existance of another edge-device @S1

S1ED1#show otv site
Site Adjacency Information (Site Bridge-Domain: 2)
Overlay2 Site-Local Adjacencies (Count: 2)
  Hostname       System ID      Last Change Ordinal    AED Enabled Status
  S1ED2          0006.F685.4500 12:37:24    0          site       overlay
* S1ED1          A80C.0DED.FA00 12:37:25    1          site       overlay

S1ED1#
 
S2ED1#show otv site
Site Adjacency Information (Site Bridge-Domain: 3)
Overlay2 Site-Local Adjacencies (Count: 1)
  Hostname       System ID      Last Change Ordinal    AED Enabled Status
* S2ED1          0023.33ED.E000 12:37:47    0          site       overlay

S2ED1#

Here again, we see two entries for the 'show otv site' on S1ED1 and just one entry on S2ED1. The thing to note in this command is the 'Ordinal' column.
 
Ordinal values are the one's which divide the vlan's between the edge-devices. OTV edge-devices divide even and odd vlan's between themselves.
An edge-device with an ordinal value of '0' means, it's the authoritative edge-device [AED] for all the even VLAN's and the one which shows the value '1' is the AED for all the odd VLAN's
 
This can be further nailed down by looking the following command:

S1ED1#show otv vlan
Key:  SI - Service Instance
Overlay 2 VLAN Configuration Information
 Inst VLAN  Bridge-Domain  Auth  Site Interface(s)
 0    11    11             yes   Gi0/0/0:SI11
 0    12    12             no    Gi0/0/0:SI12
 Total VLAN(s): 2
 Total Authoritative VLAN(s): 1
S1ED1#

As seen from the above commands, S1ED1 has the Ordinal value of '1' and with that its the AED for all odd VLAN's and 'show otv vlan' indicates the ownership of the odd VLAN's with an yes in the 'Auth' column of the output.
 
Now, let us send the ping traffic from VM1 to VM2. Here, VM1 and VM2 are our hosts on 'VLAN 11'. [VM1: 172.16.11.11; VM2: 172.16.11.111]

[root@localhost ~]# ping 172.16.11.111 -c 5
PING 172.16.11.111 (172.16.11.111) 56(84) bytes of data.
64 bytes from 172.16.11.111: icmp_seq=1 ttl=64 time=2.04 ms
64 bytes from 172.16.11.111: icmp_seq=2 ttl=64 time=0.418 ms
64 bytes from 172.16.11.111: icmp_seq=3 ttl=64 time=0.402 ms
64 bytes from 172.16.11.111: icmp_seq=4 ttl=64 time=0.409 ms
64 bytes from 172.16.11.111: icmp_seq=5 ttl=64 time=0.461 ms
--- 172.16.11.111 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.402/0.747/2.048/0.651 ms
[root@localhost ~]#

Now, lets have a quick look at the OTV MAC contents:

S1ED1#show l2fib bridge-domain 11 table unicast
Bridge Domain : 11
  Unicast Address table size : 3
  Unicast Address table information :
    Mac: 000c.29c9.f541, Adjacency: OTV Encap: 10.1.3.1
    Mac: 000c.29e3.f2bc, Adjacency: Serv Inst: Gi0/0/0:11
    Mac: ffff.ffff.ffff, Adjacency: Olist: 5121, Ports: 3
S1ED1#

We see the MAC address "000c.29c9.f541" which is that of VM2 given the encapsulation 10.1.3.1 which is nothing but the join-interface IP address of S2ED1.

S2ED1#show l2fib bridge-domain 11 table unicast
Bridge Domain : 11
  Unicast Address table size : 3
  Unicast Address table information :
    Mac: 000c.29c9.f541, Adjacency: Serv Inst: Gi2/1/1:11
    Mac: 000c.29e3.f2bc, Adjacency: OTV Encap: 10.1.1.1
    Mac: ffff.ffff.ffff, Adjacency: Olist: 5121, Ports: 3
S2ED1#

S2ED1 shows the other way around, wherein it points to S1ED1's join-interface IP address to reach VM1.

Let us check the same table on S1ED2:

S1ED2#show l2fib bridge-domain 11 table unicast
Bridge Domain : 11
  Unicast Address table size : 0
S1ED2#

This is because S1ED2 is not the AED for bridge-domain 11.

Now, let's put redundancy to test: I will bring down S1ED1's join-interface 'Port-Channel 11' via shutdown command on the 'core' and then again ping from VM1 to VM2:

[root@localhost ~]# ping 172.16.11.111 -c 5
PING 172.16.11.111 (172.16.11.111) 56(84) bytes of data.
64 bytes from 172.16.11.111: icmp_seq=2 ttl=64 time=0.481 ms
64 bytes from 172.16.11.111: icmp_seq=3 ttl=64 time=0.468 ms
64 bytes from 172.16.11.111: icmp_seq=4 ttl=64 time=0.515 ms
64 bytes from 172.16.11.111: icmp_seq=5 ttl=64 time=0.457 ms
--- 172.16.11.111 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.457/0.480/0.515/0.026 ms
[root@localhost ~]#

We see above 'one' ping is lost since there was a down time, but, there was recovery as well wherein due to which ping was 80% successful.

S1ED2#show l2fib bridge-domain 11 table unicast
Bridge Domain : 11
  Unicast Address table size : 3
  Unicast Address table information :
    Mac: 000c.29c9.f541, Adjacency: OTV Encap: 10.1.3.1
    Mac: 000c.29e3.f2bc, Adjacency: Serv Inst: Gi0/0/1:11
    Mac: ffff.ffff.ffff, Adjacency: Olist: 5125, Ports: 2
S1ED2#

S2ED1#show l2fib bridge-domain 11 table unicast
Bridge Domain : 11
  Unicast Address table size : 3
  Unicast Address table information :
    Mac: 000c.29c9.f541, Adjacency: Serv Inst: Gi2/1/1:11
    Mac: 000c.29e3.f2bc, Adjacency: OTV Encap: 10.1.2.1
    Mac: ffff.ffff.ffff, Adjacency: Olist: 5121, Ports: 2
S2ED1#

S1ED2 which had no entry related to bridge-domain 11, now shows the required information, and on the same lines, S2ED1 which was showing the IP address of S1ED1, now has S1ED2.

Now, I will un-shut the core interface to bring up the S1ED1 again:

[root@localhost ~]# ping 172.16.11.111 -c 5
PING 172.16.11.111 (172.16.11.111) 56(84) bytes of data.
64 bytes from 172.16.11.111: icmp_seq=1 ttl=64 time=1001 ms
64 bytes from 172.16.11.111: icmp_seq=2 ttl=64 time=2.44 ms
64 bytes from 172.16.11.111: icmp_seq=3 ttl=64 time=0.410 ms
64 bytes from 172.16.11.111: icmp_seq=4 ttl=64 time=0.399 ms
64 bytes from 172.16.11.111: icmp_seq=5 ttl=64 time=0.412 ms
--- 172.16.11.111 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.399/201.117/1001.919/400.401 ms, pipe 2
[root@localhost ~]#

S1ED1#show l2fib bridge-domain 11 table unicast
Bridge Domain : 11
  Unicast Address table size : 3
  Unicast Address table information :
    Mac: 000c.29c9.f541, Adjacency: OTV Encap: 10.1.3.1
    Mac: 000c.29e3.f2bc, Adjacency: Serv Inst: Gi0/0/0:11
    Mac: ffff.ffff.ffff, Adjacency: Olist: 5121, Ports: 3
S1ED1#

S2ED1#show l2fib bridge-domain 11 table unicast
Bridge Domain : 11
  Unicast Address table size : 3
  Unicast Address table information :
    Mac: 000c.29c9.f541, Adjacency: Serv Inst: Gi2/1/1:11
    Mac: 000c.29e3.f2bc, Adjacency: OTV Encap: 10.1.1.1
    Mac: ffff.ffff.ffff, Adjacency: Olist: 5121, Ports: 3
S2ED1#

As observed, S1ED1 becomes the AED for 'VLAN 11' again, and it's reflected on S2ED1 as well.

This concludes a very basic multi-homing setup. I will take this forward by using HSRP between the sites and also send a bit of inter-VLAN traffic across as well.