VXLAN EVPN Fabric で仮想ネットワークの設定を Anaible を使って自動化する
VXLAN EVPN の環境で Overlay network の設定を Ansible で自動化してみました。VXLAN Fabric では、L2 ネットワークを 1つ追加するだけでも全ての Leaf スイッチに対して設定を行う必要があります。この設定を自動化する ansible playbook を書いてみました。
構成と作成する仮想ネットワークは次のようになっています。
スイッチは仮想 OS の NX-OS 9000v 9.3(1) です。
Ansible
Playbook
- name: Provisioning VXLAN Overlay Network hosts: leaf gather_facts: no connection: network_cli vars: asn: "65000" tasks: - name: Crete vlans nxos_vlan: vlan_id: "{{ item.vlan_id }}" name: "{{ item.name }} " mapped_vni: "{{ item.vni_id }}" with_items: - "{{ vlans }}" - name: Create VRF nxos_vrf: name: "{{ vrf.name }}" vni: "{{ vrf.vni_id }}" rd: auto - name: Manage VRF AF nxos_vrf_af: afi: ipv4 vrf: "{{ vrf.name }}" route_target_both_auto_evpn: True - name: Create SVI interface nxos_interface: name: "Vlan{{item.vlan_id}}" with_items: - "{{ vlans }}" - name: Associate VRF to SVI interface nxos_vrf_interface: vrf: "{{ vrf.name }}" interface: "Vlan{{item.vlan_id}}" with_items: - "{{ vlans }}" - name: Set IP address to SVI interface when: item.address is defined nxos_l3_interface: name: "Vlan{{item.vlan_id}}" ipv4: "{{item.address }}" with_items: - "{{ vlans }}" - name: Enable anycast gateway when: item.address is defined nxos_interface: fabric_forwarding_anycast_gateway: True name: "Vlan{{item.vlan_id}}" with_items: - "{{ vlans }}" - name: Enable ip forwarding when: item.address is not defined nxos_interface: ip_forward: enable name: "Vlan{{item.vlan_id}}" with_items: - "{{ vlans }}" - name: Manage BGP address family nxos_bgp_af: vrf: "{{vrf.name}}" asn: "{{asn}}" afi: ipv4 safi: unicast - name: Set l2 vni to nve when: item.address is defined nxos_vxlan_vtep_vni: ingress_replication: bgp interface: nve1 vni: "{{item.vni_id}}" with_items: - "{{ vlans }}" - name: Set l3 vni to nve when: item.address is not defined nxos_vxlan_vtep_vni: interface: nve1 vni: "{{item.vni_id}}" assoc_vrf: True with_items: - "{{ vlans }}"
Inventory
[leaf] 172.16.0.101 172.16.0.102 172.16.0.103 [leaf:vars] ansible_ssh_user = admin ansible_connection = local
Variables
vlans: - vlan_id: 100 name: VNI10000 vni_id: 10000 address: 192.168.0.254/24 mask: 24 - vlan_id: 101 name: VNI10001 vni_id: 10001 address: 192.168.1.254/24 - vlan_id: 900 name: VNI90000_vrf_blue vni_id: 90000 vrf: name: blue vni_id: 90000
気づいたこと
nxos_vlan は2回実行しても、ステータスが changed になる。
nxos_interface で ip forward、fabric forwarding anycast gateway を設定した後に、nxos_vrf_interface で Interface に VRF を割り当てると設定が消える。VRF に紐づく L3 の設定なので当たり前ですが、公式ドキュメントの説明に物理インターフェースの管理と記載があるので、注意が必要であると思いました。
Manages physical attributes of interfaces of NX-OS switches.
結果
[root@localhost ansible]# ansible-playbook -i inventories/hosts provision_overlay_nw.yaml -e @group_vars/vrf_blue.yaml --timeout=300
PLAY [Provisioning VXLAN Overlay Network] *****************************************************************************************************************************
TASK [Crete vlans] ****************************************************************************************************************************************************
changed: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
TASK [Create VRF] *****************************************************************************************************************************************************
changed: [172.16.0.103]
changed: [172.16.0.102]
changed: [172.16.0.101]
TASK [Manage VRF AF] **************************************************************************************************************************************************
changed: [172.16.0.103]
changed: [172.16.0.102]
changed: [172.16.0.101]
TASK [Create SVI interface] *******************************************************************************************************************************************
changed: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
TASK [Associate VRF to SVI interface] *********************************************************************************************************************************
changed: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
TASK [Set IP address to SVI interface] ********************************************************************************************************************************
changed: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
TASK [Enable anycast gateway] *****************************************************************************************************************************************
changed: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
TASK [Enable ip forwarding] *******************************************************************************************************************************************
skipping: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
skipping: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
TASK [Manage BGP address family] **************************************************************************************************************************************
changed: [172.16.0.103]
changed: [172.16.0.101]
changed: [172.16.0.102]
TASK [Set l2 vni to nve] **********************************************************************************************************************************************
changed: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
TASK [Set l3 vni to nve] **********************************************************************************************************************************************
skipping: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
PLAY RECAP ************************************************************************************************************************************************************
172.16.0.101 : ok=11 changed=11 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
172.16.0.102 : ok=11 changed=11 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
172.16.0.103 : ok=11 changed=11 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
設定された config
config の差分を確認してみます。
[root@localhost ~]# diff -y -W 80 before/Leaf1-running-config after/Leaf1-running-config
version 9.3(1) Bios:version version 9.3(1) Bios:version
hostname Leaf1 hostname Leaf1
--- snip ---
fabric forwarding anycast-gateway-mac fabric forwarding anycast-gateway-mac
vlan 1 | vlan 1,100-101,900
| vlan 100
> name VNI10000
> vn-segment 10000
> vlan 101
> name VNI10001
> vn-segment 10001
> vlan 900
> name VNI90000_vrf_blue
> vn-segment 90000
>
> vrf context blue
> vni 90000
> rd auto
> address-family ipv4 unicast
> route-target both auto evpn
--- snip ---
interface Vlan1 interface Vlan1
> interface Vlan100
> no shutdown
> vrf member blue
> ip address 192.168.0.254/24
> fabric forwarding mode anycast-gate
>
> interface Vlan101
> no shutdown
> vrf member blue
> ip address 192.168.1.254/24
> fabric forwarding mode anycast-gate
>
> interface Vlan900
> no shutdown
> vrf member blue
> ip forward
>
interface nve1 interface nve1
no shutdown no shutdown
host-reachability protocol bgp host-reachability protocol bgp
source-interface loopback1 source-interface loopback1
> member vni 10000
> ingress-replication protocol bgp
> member vni 10001
> ingress-replication protocol bgp
> member vni 90000 associate-vrf
router bgp 65000 router bgp 65000
address-family ipv4 unicast address-family ipv4 unicast
address-family l2vpn evpn address-family l2vpn evpn
neighbor 1.1.1.1 neighbor 1.1.1.1
remote-as 65000 remote-as 65000
update-source loopback0 update-source loopback0
address-family ipv4 unicast address-family ipv4 unicast
address-family l2vpn evpn address-family l2vpn evpn
send-community extended send-community extended
neighbor 1.1.1.2 neighbor 1.1.1.2
remote-as 65000 remote-as 65000
update-source loopback0 update-source loopback0
address-family ipv4 unicast address-family ipv4 unicast
address-family l2vpn evpn address-family l2vpn evpn
send-community extended send-community extended
> vrf blue
> address-family ipv4 unicast
Leaf2、Leaf3 も同様の変更があることを確認しました。
通信確認
L2 と L3 で通信できることを確認できました。
PC-1> ping 192.168.0.2 84 bytes from 192.168.0.2 icmp_seq=1 ttl=64 time=101.646 ms 84 bytes from 192.168.0.2 icmp_seq=2 ttl=64 time=40.088 ms 84 bytes from 192.168.0.2 icmp_seq=3 ttl=64 time=29.891 ms 84 bytes from 192.168.0.2 icmp_seq=4 ttl=64 time=22.204 ms ^C PC-1> ping 192.168.1.3 84 bytes from 192.168.1.3 icmp_seq=1 ttl=63 time=47.751 ms 84 bytes from 192.168.1.3 icmp_seq=2 ttl=63 time=38.885 ms 84 bytes from 192.168.1.3 icmp_seq=3 ttl=63 time=32.026 ms 84 bytes from 192.168.1.3 icmp_seq=4 ttl=63 time=53.960 ms ^C PC-1>
Leaf1が別のLeafに接続されている端末の情報を、BGP で受け取っていることがわかります。
Leaf1# show bgp l2vpn evpn vrf blue
Route Distinguisher: 1.1.1.3:7 (L3VNI 90000)
*>i[2]:[0]:[0]:[48]:[0000.aaaa.0001]:[32]:[192.168.0.254]/272
2.2.2.4 100 0 i
*>i[2]:[0]:[0]:[48]:[0000.aaaa.0001]:[32]:[192.168.1.254]/272
2.2.2.4 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[192.168.1.3]/272
2.2.2.5 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6802]:[32]:[192.168.0.2]/272
2.2.2.4 100 0 i
まとめ
VXLAN EVPN Fabric の環境に仮想ネットワークを作成する ansible playbook を作成しました。VXLAN Fabric の設定変更は複数のスイッチに対して同時に設定する必要があるので、Ansible を利用する効果が高いと思います。
