[Nexus9K] VXLAN EVPN サービスチェイニング – GPO & ePBRを試してみた
前回VXLAN EVPN環境で、マルチテナント上のエンドポイント間セキュリティとしてCisco NexusにおけるGPO（Group Policy Option）を紹介しました。
このGPOはサービスチェイニングにも活用できます。利用用途として、例えばWeb→DB通信だけをFWとIPSへ通す、テナント間通信を共通FWへ集約する、外部公開系をLB→FWへ通す、NAT装置を挿入するなど、サービスを挟み込むことが可能です。
ということで、動作概要と実機で簡単な動作をみてみたいと思います。
GPOを使ったサービスチェイニング
GPOを使ったサービスチェイニングは、VXLANファブリック内に接続されるサーバやアプリケーションを「Web」「DB」「App」などのセキュリティグループに分類し、グループ間の通信にFW、IPS、LBなどのグループをサービスチェーン（差し込む）する仕組みです。通信の送信元・宛先IPではなく、ワークロードの役割を基準に制御できるため、VM移動やIP変更があってもポリシーを維持しやすく、East-West通信のマイクロセグメンテーションや複数サイトへの展開に向きます。
GPOでエンドポイント同士の通信を引っ掛けて、サービスチェイニングさせる際にはNexusのePBR(Enhanced Policy-based Redirect)機能を使います。
ePBRは、対象トラフィックをサービス装置へ実際にリダイレクトし、順序制御、死活監視、負荷分散、障害時のDrop／Bypassなどを行う転送エンジンです。従来型ePBRはACLで通信を選別し、入口インターフェースへポリシーを適用するため、IPアドレスや物理・論理トポロジーへの依存が比較的大きくなります。GPO利用時は、セキュリティグループ間ので対象を選びePBRを呼び出します。
検証内容と結果
EPG（Endpoint Group）間通信に、サービス相当のEPGを経由させるテストをしてみます。前回のGPOの構成をベースにテストしました。
検証構成
今回は以下のようなシングルテナント構成で実施しました。All IPv6 eBGP構成がベースとなっています。
それぞれのリーフスイッチに接続されたエンドポイントを異なるセレクト条件(IPv4/IPv6/VLAN)で定義します。
。
テスト内容と結果
テストは以下の図の通り、EPG1からEPG2への通信に対してFWを経由する設定をいれて想定どおり動作するか確認しました。
またGPO+ePBRの際に、ICMP Probeを利用してEPG4がUpしていることも確認しています。ICMP Probeのするために、各Leaf上でLoopback IFを作成し、サービスチェイニングするEPGと同一のVRFをLoopback IFに設定して同一テナントに所属させます。そして各Loopback IFからターゲットに対してICMPを利用してProbeを行います。
テスト結果
試験結果としては各テストパターンともに想定した通りのEPG1−EPG2間の通信に対して、FWを経由することを各LeafスイッチのIFカウンタから確認しています。
それではコンフィグとステータスを確認します。
GPO ePBR関連のコンフィグ
まずGPOとePBRにフォーカスして、各スイッチに適用するコンフィグを見てみたいと思います。GPOの設定部分は前回の記事と同じ内容となります。
各リーフのGPOの基本設定
feature security-group
feature sla sender
feature pbr
feature epbr
system routing template-security-groups
### system routing template-security-groupsでrebootが必要
次にEPGを定義します。各ローカルリーフスイッチで定義します。IPv4/IPv6/VLAN/IFなど様々な条件でグループ条件を設定することが可能です。
各リーフのEPGの設定
NXOS15:
security-group 100 name EPG1
match connected-endpoints vrf vrf-3000 ipv4 20.20.20.1/32
match connected-endpoints vrf vrf-3000 ipv6 2001:20:20:20::1/128
security-group 200 name EPG2
match connected-endpoints vrf vrf-3000 ipv4 21.21.21.0/24
match connected-endpoints vrf vrf-3000 ipv6 2001:21:21:21::1/128
NXOS16:
security-group 220 name SG_One_Arm_Fw
type layer4-7
match interface Vlan220
次に各スイッチのGPO関連のポリシーとePBRの設定なります。ePBRではICMP Probeを利用してエンドポイントの死活監視をしています。ICMP Probe用のSource addressとなるLoopback IFもここで作成しています。
なおポリシー関連の設定名ですが、HTTPサーバ宛のWebトラフィックをFWに折り曲げるという前提で作成した名称です。実際の通信対象がICMPのため、設定名がややこしくなってしまいました。
各リーフのポリシーの設定
NXOS15:
interface loopback11
description SLA Probe loopback interface
vrf member vrf-3000
ip address 10.11.0.1/32 tag 12345
ipv6 address 2001:10:11::1/128 tag 12345
icam monitor scale
NXOS16:
interface loopback11
description SLA Probe loopback interface
vrf member vrf-3000
ip address 10.11.0.3/32 tag 12345
ipv6 address 2001:10:11::3/128 tag 12345
icam monitor scale
NXOS15/16共通:
class-map type security match-any icmp
match ip icmp
match ipv6 icmpv6
class-map type security match-any web_icmp
match ip icmp
class-map type security match-any web_icmp-v6
match ipv6 icmpv6
policy-map type security HTTPs_Redirect_To_Fw
class web_icmp
log
service-chain SRV_Chain_One_Arm_Fw
policy-map type security HTTPs_Redirect_To_Fw-v6
class web_icmp-v6
log
service-chain SRV_Chain_One_Arm_Fw-v6
policy-map type security permit_icmp
class icmp
log
epbr service SRV_One_Arm_Fw-v6
vrf vrf-3000
security-group 220
probe icmp source-interface loopback11
service-end-point ipv6 2001:22:22:22::1
epbr service SRV_One_Arm_Fw
vrf vrf-3000
security-group 220
probe icmp source-interface loopback11
service-end-point ip 22.22.22.1
epbr service-chain SRV_Chain_One_Arm_Fw-v6
10 set service SRV_One_Arm_Fw-v6
epbr service-chain SRV_Chain_One_Arm_Fw
10 set service SRV_One_Arm_Fw
vrf context vrf-3000
vni 3000
security contract source 100 destination 200 policy HTTPs_Redirect_To_Fw
security contract source 100 destination 200 policy HTTPs_Redirect_To_Fw-v6
security enforce tag 9999 default deny
全体コンフィグ
ここで全体のコンフィグを見てみましょう。
以下テスト１の全体コンフィグとなります。NXOS15とNXOS16の太字部分が上記で説明したGPOコンフィグ部分となります。NXOS14はSpineなので特にGPOのコンフィグはありません。
NXOS14
NXOS14# show run
version 10.6(2) Bios:version 05.53
hostname NXOS14
<snip>
feature nxapi
nv overlay evpn
feature ospf
feature bgp
feature pim
feature grpc
feature netflow
feature analytics
feature lldp
clock timezone JST 9 0
clock protocol ptp vdc 1
feature nv overlay
feature ngoam
feature telemetry
feature nxsecure
<snip>
route-map NEXT-HOP-UNCH permit 10
set ip next-hop unchanged
set ipv6 next-hop unchanged
<snip>
interface Ethernet1/32
description to NXOS16 Ethernet1/1
ipv6 address use-link-local-only
ipv6 nd ra-interval 4 min 3
ipv6 nd ra-lifetime 10
no shutdown
<snip>
interface Ethernet1/34
description to NXOS15 Ethernet1/1
ipv6 address use-link-local-only
ipv6 nd ra-interval 4 min 3
ipv6 nd ra-lifetime 10
no shutdown
<snip>
interface loopback0
description SPINE_LOOPBACK
ipv6 address 2001:db8:100::14/128
icam monitor scale
line console
line vty
boot nxos bootflash:/nxos64-cs.10.6.2.F.bin
router bgp 65000
router-id 14.14.14.14
bestpath as-path multipath-relax
address-family ipv6 unicast
network 2001:db8:100::14/128
address-family l2vpn evpn
retain route-target all
neighbor Ethernet1/32
remote-as 65016
description BGP_UNNUMBERED_TO_NXOS16
address-family ipv6 unicast
address-family l2vpn evpn
send-community
send-community extended
route-map NEXT-HOP-UNCH out
neighbor Ethernet1/34
remote-as 65015
description BGP_UNNUMBERED_TO_NXOS15
address-family ipv6 unicast
address-family l2vpn evpn
send-community
send-community extended
route-map NEXT-HOP-UNCH out
NXOS15
NXOS15# show run
version 10.6(3) Bios:version 01.12
hostname NXOS15
<snip>
feature nxapi
cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature pim
feature grpc
feature pbr
feature interface-vlan
feature vn-segment-vlan-based
feature netflow
feature analytics
feature lacp
feature dhcp
feature lldp
clock timezone JST 9 0
clock protocol ptp vdc 1
feature sla sender
feature nv overlay
feature security-group
feature ngoam
feature telemetry
feature epbr
feature nxsecure
<snip>
security-group 100 name EPG1
match connected-endpoints vrf vrf-3000 ipv4 20.20.20.1/32
match connected-endpoints vrf vrf-3000 ipv6 2001:20:20:20::1/128
security-group 200 name EPG2
match connected-endpoints vrf vrf-3000 ipv4 21.21.21.0/24
match connected-endpoints vrf vrf-3000 ipv6 2001:21:21:21::1/128
class-map type security match-any icmp
match ip icmp
match ipv6 icmpv6
class-map type security match-any web_icmp
match ip icmp
class-map type security match-any web_icmp-v6
match ipv6 icmpv6
policy-map type security HTTPs_Redirect_To_Fw
class web_icmp
log
service-chain SRV_Chain_One_Arm_Fw
policy-map type security HTTPs_Redirect_To_Fw-v6
class web_icmp-v6
log
service-chain SRV_Chain_One_Arm_Fw-v6
policy-map type security permit_icmp
class icmp
log
fabric forwarding anycast-gateway-mac 2020.0000.00aa
ipv6 switch-packets lla
vlan 1,200,210,300
vlan 200
vn-segment 2000
vlan 210
vn-segment 2100
vlan 300
vn-segment 3000
route-map RM_EVPN_NH permit 10
set ipv6 next-hop 2001:db8:100::15
<snip>
vrf context vrf-3000
vni 3000
security contract source 100 destination 200 policy HTTPs_Redirect_To_Fw
security contract source 100 destination 200 policy HTTPs_Redirect_To_Fw-v6
security enforce tag 9999 default deny
rd auto
address-family ipv4 unicast
route-target import 3000:3000
route-target import 3000:3000 evpn
route-target export 3000:3000
route-target export 3000:3000 evpn
address-family ipv6 unicast
route-target import 3000:3000
route-target import 3000:3000 evpn
route-target export 3000:3000
route-target export 3000:3000 evpn
system routing template-security-groups
<snip>
interface Vlan200
no shutdown
vrf member vrf-3000
ip address 20.20.20.254/24
ipv6 address 2001:20:20:20::254/64
fabric forwarding mode anycast-gateway
interface Vlan210
no shutdown
vrf member vrf-3000
ip address 21.21.21.254/24
ipv6 address 2001:21:21:21::254/64
fabric forwarding mode anycast-gateway
interface Vlan300
no shutdown
vrf member vrf-3000
ip forward
ipv6 forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 2000
ingress-replication protocol bgp
member vni 2100
ingress-replication protocol bgp
member vni 3000 associate-vrf
<snip>
interface Ethernet1/51
description to VPC9
switchport
switchport mode trunk
switchport trunk allowed vlan 200,210
no shutdown
interface Ethernet1/52
description to NXOS14 Ethernet1/1
ipv6 address use-link-local-only
ipv6 nd ra-interval 4 min 3
ipv6 nd ra-lifetime 10
no shutdown
<snip>
interface loopback0
description VTEP_LOOPBACK_IPV6
ipv6 address 2001:db8:100::15/128
interface loopback11
description SLA Probe loopback interface
vrf member vrf-3000
ip address 10.11.0.1/32 tag 12345
ipv6 address 2001:10:11::1/128 tag 12345
icam monitor scale
line console
line vty
boot nxos bootflash:/nxos64-cs.10.6.3.F.bin
router bgp 65015
router-id 15.15.15.15
bestpath as-path multipath-relax
address-family ipv6 unicast
network 2001:db8:100::15/128
address-family l2vpn evpn
neighbor Ethernet1/52
remote-as 65000
description BGP_UNNUMBERED_TO_NXOS14
address-family ipv6 unicast
address-family l2vpn evpn
send-community
send-community extended
route-map RM_EVPN_NH out
vrf vrf-3000
address-family ipv4 unicast
network 10.11.0.1/32
advertise l2vpn evpn
address-family ipv6 unicast
network 2001:10:11::1/128
advertise l2vpn evpn
evpn
vni 2000 l2
rd auto
route-target import 2000:2000
route-target export 2000:2000
vni 2100 l2
rd auto
route-target import 2100:2100
route-target export 2100:2100
epbr service SRV_One_Arm_Fw-v6
vrf vrf-3000
security-group 220
probe icmp source-interface loopback11
service-end-point ipv6 2001:22:22:22::1
epbr service SRV_One_Arm_Fw
vrf vrf-3000
security-group 220
probe icmp source-interface loopback11
service-end-point ip 22.22.22.1
epbr service-chain SRV_Chain_One_Arm_Fw-v6
10 set service SRV_One_Arm_Fw-v6
epbr service-chain SRV_Chain_One_Arm_Fw
10 set service SRV_One_Arm_Fw
NXOS16
NXOS16# show run
version 10.6(3) Bios:version 01.12
hostname NXOS16
<snip>
feature nxapi
cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature pim
feature grpc
feature pbr
feature interface-vlan
feature vn-segment-vlan-based
feature netflow
feature analytics
feature lacp
feature dhcp
feature lldp
clock timezone JST 9 0
clock protocol ptp vdc 1
feature sla sender
feature nv overlay
feature security-group
feature ngoam
feature telemetry
feature epbr
<snip>
security-group 220 name SG_One_Arm_Fw
type layer4-7
match interface Vlan220
security-group 300 name EPG3
match vlan 200
security-group 400 name EPG4
match vlan 220
class-map type security match-any icmp
match ip icmp
match ipv6 icmpv6
class-map type security match-any web_icmp
match ip icmp
class-map type security match-any web_icmp-v6
match ipv6 icmpv6
policy-map type security HTTPs_Redirect_To_Fw
class web_icmp
log
service-chain SRV_Chain_One_Arm_Fw
policy-map type security HTTPs_Redirect_To_Fw-v6
class web_icmp-v6
log
service-chain SRV_Chain_One_Arm_Fw-v6
policy-map type security permit_icmp
class icmp
log
fabric forwarding anycast-gateway-mac 2020.0000.00aa
ipv6 switch-packets lla
vlan 1,200,220,300
vlan 200
vn-segment 2000
vlan 220
vn-segment 2200
vlan 300
vn-segment 3000
route-map RM_EVPN_NH permit 10
set ipv6 next-hop 2001:db8:100::16
service dhcp
<snip>
vrf context vrf-3000
vni 3000
security contract source 100 destination 200 policy HTTPs_Redirect_To_Fw
security contract source 100 destination 200 policy HTTPs_Redirect_To_Fw-v6
security enforce tag 9999 default deny
rd auto
address-family ipv4 unicast
route-target import 3000:3000
route-target import 3000:3000 evpn
route-target export 3000:3000
route-target export 3000:3000 evpn
address-family ipv6 unicast
route-target import 3000:3000
route-target import 3000:3000 evpn
route-target export 3000:3000
route-target export 3000:3000 evpn
system routing template-security-groups
interface Vlan1
interface Vlan200
no shutdown
vrf member vrf-3000
ip address 20.20.20.254/24
ipv6 address 2001:20:20:20::254/64
fabric forwarding mode anycast-gateway
interface Vlan220
no shutdown
vrf member vrf-3000
ip address 22.22.22.254/24
ipv6 address 2001:22:22:22::254/64
fabric forwarding mode anycast-gateway
interface Vlan300
no shutdown
vrf member vrf-3000
ip forward
ipv6 forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 2000
ingress-replication protocol bgp
member vni 2200
ingress-replication protocol bgp
member vni 3000 associate-vrf
<snip>
interface Ethernet1/51
switchport
switchport mode trunk
switchport trunk allowed vlan 200,220
no shutdown
interface Ethernet1/52
speed 40000
no negotiate auto
ipv6 address use-link-local-only
ipv6 nd ra-interval 4 min 3
ipv6 nd ra-lifetime 10
no shutdown
<snip>
interface loopback0
description VTEP_LOOPBACK_IPV6
ipv6 address 2001:db8:100::16/128
interface loopback11
description SLA Probe loopback interface
vrf member vrf-3000
ip address 10.11.0.3/32 tag 12345
ipv6 address 2001:10:11::3/128 tag 12345
icam monitor scale
line console
line vty
boot nxos bootflash:/nxos64-cs.10.6.3.F.bin
router bgp 65016
router-id 16.16.16.16
bestpath as-path multipath-relax
address-family ipv6 unicast
network 2001:db8:100::16/128
address-family l2vpn evpn
neighbor Ethernet1/52
remote-as 65000
description BGP_UNNUMBERED_TO_NXOS14
address-family ipv6 unicast
address-family l2vpn evpn
send-community
send-community extended
route-map RM_EVPN_NH out
evpn
vni 2000 l2
rd auto
route-target import 2000:2000
route-target export 2000:2000
vni 2200 l2
rd auto
route-target import 2200:2200
route-target export 2200:2200
epbr service SRV_One_Arm_Fw-v6
vrf vrf-3000
security-group 220
probe icmp source-interface loopback11
service-end-point ipv6 2001:22:22:22::1
epbr service SRV_One_Arm_Fw
vrf vrf-3000
security-group 220
probe icmp source-interface loopback11
service-end-point ip 22.22.22.1
epbr service-chain SRV_Chain_One_Arm_Fw-v6
10 set service SRV_One_Arm_Fw-v6
epbr service-chain SRV_Chain_One_Arm_Fw
10 set service SRV_One_Arm_Fw
各ステータス確認コマンド
GPOとePBRを中心にステータスを見てみます。なおIPv6関連のコマンド出力が一部切れている事象が見受けられました。
NXOS15
NXOS15# show epbr service-chain
Service-chain : SRV_Chain_One_Arm_Fw state:UP
service:SRV_One_Arm_Fw, sequence:10, fail-action:Drop
load-balance:Source-Destination-ipprotocol, action:Redirect
state:UP
IP 22.22.22.1 track 1 [UP]
Service-chain : SRV_Chain_One_Arm_Fw-v6 state:UP
service:SRV_One_Arm_Fw-v6, sequence:10, fail-action:Drop
load-balance:Source-Destination-ipprotocol, action:Redirect
state:UP
NXOS15# V6 2001:22:22:22::1 track 2 [UP]
NXOS15#
NXOS15#
NXOS15# show contracts
VRF SGT DGT Policy Dir Stats Class Action OperSt
--------------------------------------------------------------------------------------------------------
vrf-3000 100 200 HTTPs_Redirect_To_Fw
bidir 600 web_icmp redir,log enabled
100 200 HTTPs_Redirect_To_Fw-v6
bidir 458 web_icmp-v6 redir,log enabled
NXOS15#
NXOS15# show epbr service
Legend:
Operational State (Op-STS): UP:Reachable, DOWN:Unreachable,
SVC-ADMIN-DOWN:Service shut
ADMIN-DOWN:Admin shut, OPER-DOWN:Out-of-service
Probe: Protocol/Frequency(sec)/Timeout(sec)/Retry-Up-Count/Retry-Down-Count
Hold-down Threshold: Count/Time(sec)
Service mode: Full:Full-Duplex, Half:Half-Duplex
Type: L3:Layer-3, L2:Layer-2
Threshold: Value(percent)
Mode: A: Active, HS: Hot-Standby
Name Type Service mode VRF
==========================================================================================
SRV_One_Arm_Fw L3 Full vrf-3000
Security-group Reverse security-group Threshold
=======================================================
220
Endpoint IP/Intf Track SLA Op-ST Probe Hold-down Role Weight
Reverse IP/Intf Track SLA Op-ST Probe
=================================================================================================
22.22.22.1/ 1 20001 UP ICMP/0/0/0/0 A 1
Name Type Service mode VRF
==========================================================================================
SRV_One_Arm_Fw-v6 L3 Full vrf-3000
Security-group Reverse security-group Threshold
=======================================================
220
Endpoint IP/Intf Track SLA Op-ST Probe Hold-down Role Weight
Reverse IP/Intf Track SLA Op-ST Probe
=================================================================================================
NXOS15# 22:22::1/ 2 20002 UP ICMP/0/0/0/0 A 1
NXOS15#
NXOS15#
NXOS15# show logging ip access-list cache
Src IP Dst IP S-Port D-Port Src Intf Protocol Hits
------------------------------------------------------------------------------------------
21.21.21.1 20.20.20.1 0 0 Ethernet1/51 (1)ICMP 0
2001:21:21:21::1 2001:20:20:20::1 0 0 Ethernet1/51 (58)IPv6-ICMP 0
2001:20:20:20::1 2001:21:21:21::1 0 0 Ethernet1/51 (58)IPv6-ICMP 0
20.20.20.1 21.21.21.1 0 0 Ethernet1/51 (1)ICMP 0
NXOS15#
NXOS16
NXOS16# show contracts
VRF SGT DGT Policy Dir Stats Class Action OperSt
--------------------------------------------------------------------------------------------------------
vrf-3000 100 200 HTTPs_Redirect_To_Fw
bidir 1600 web_icmp redir,log enabled
100 200 HTTPs_Redirect_To_Fw-v6
bidir 887 web_icmp-v6 redir,log enabled
NXOS16#
NXOS16# show epbr service
Legend:
Operational State (Op-STS): UP:Reachable, DOWN:Unreachable,
SVC-ADMIN-DOWN:Service shut
ADMIN-DOWN:Admin shut, OPER-DOWN:Out-of-service
Probe: Protocol/Frequency(sec)/Timeout(sec)/Retry-Up-Count/Retry-Down-Count
Hold-down Threshold: Count/Time(sec)
Service mode: Full:Full-Duplex, Half:Half-Duplex
Type: L3:Layer-3, L2:Layer-2
Threshold: Value(percent)
Mode: A: Active, HS: Hot-Standby
Name Type Service mode VRF
==========================================================================================
SRV_One_Arm_Fw L3 Full vrf-3000
Security-group Reverse security-group Threshold
=======================================================
220
Endpoint IP/Intf Track SLA Op-ST Probe Hold-down Role Weight
Reverse IP/Intf Track SLA Op-ST Probe
=================================================================================================
22.22.22.1/ 1 20001 UP ICMP/0/0/0/0 A 1
Name Type Service mode VRF
==========================================================================================
SRV_One_Arm_Fw-v6 L3 Full vrf-3000
Security-group Reverse security-group Threshold
=======================================================
220
Endpoint IP/Intf Track SLA Op-ST Probe Hold-down Role Weight
Reverse IP/Intf Track SLA Op-ST Probe
=================================================================================================
NXOS16# 22:22::1/ 2 20002 UP ICMP/0/0/0/0 A 1
NXOS16#
NXOS16# show logging ip access-list cache
Src IP Dst IP S-Port D-Port Src Intf Protocol Hits
------------------------------------------------------------------------------------------
21.21.21.1 20.20.20.1 0 0 nve-peer1 (1)ICMP 0
21.21.21.1 20.20.20.1 0 0 Ethernet1/51 (1)ICMP 0
2001:21:21:21::1 2001:20:20:20::1 0 0 Ethernet1/51 (58)IPv6-ICMP 0
2001:20:20:20::1 2001:21:21:21::1 0 0 Ethernet1/51 (58)IPv6-ICMP 24
2001:20:20:20::1 2001:21:21:21::1 0 0 nve-peer1 (58)IPv6-ICMP 24
2001:21:21:21::1 2001:20:20:20::1 0 0 nve-peer1 (58)IPv6-ICMP 0
20.20.20.1 21.21.21.1 0 0 Ethernet1/51 (1)ICMP 0
20.20.20.1 21.21.21.1 0 0 nve-peer1 (1)ICMP 0
NXOS16#
まとめ
今回は前回のGPOに続いて、ePBRとの組み合わせによるサービスチェイニングの動作を見てみました。
ワンアームFWを前提とした基本的な動作のみの確認でしたが、多段サービスチェイニングや、ツーアーム接続、サービスへのロード・バランシングなど多様なオプションがあるので、マルチテナント環境においてもサービスに応じた柔軟な構成が取れそうです。