nxos and inter-vlan ACLs

---

Hello ITPro. I am pulling my hair out trying to figure out how ACLs work on the nxos platform with inter-vlan routing. Hoping someone with experience can explain to me why with the following configuration, packets from source 192.168.120.0/24 are not blocked when destined for the 192.168.119.0/24 network.

Note, this is an HSRP setup, so two switches sharing virtual IP for gateway on both vlans.

WILNXLAB-01:
interface Vlan1
  no shutdown
  ip address 192.168.119.3/24
  hsrp version 2
  hsrp 119
    preempt
    priority 90
    ip 192.168.119.2
    track 119
    track 120
    track 121
    track 122


interface Vlan120
  no shutdown
  ip access-group test in <-----------enable ACL
  ip address 192.168.120.3/24
  hsrp version 2
  hsrp 120
    preempt
    priority 90
    ip 192.168.120.2
    track 119
    track 120
    track 121
    track 122	WILNXLAB-02:

WILNXLAB-02:
interface Vlan1
  no shutdown
  ip address 192.168.119.4/24
  hsrp version 2
  hsrp 119
    preempt
    priority 80
    ip 192.168.119.2
    track 119
    track 120
    track 121
    track 122


interface Vlan120
  no shutdown
  ip access-group test in <-----------enable ACL
  ip address 192.168.120.4/24
  hsrp version 2
  hsrp 120
    preempt
    priority 80
    ip 192.168.120.2
    track 119
    track 120
    track 121
    track 122

---------------------Access List on each switch
IP access list test
        10 deny ip 192.168.120.0 0.0.0.255 any
        20 deny icmp 192.168.120.0 0.0.0.255

@Adam-Tyler said in nxos and inter-vlan ACLs:

IP access list test
10 deny ip 192.168.120.0 0.0.0.255 any
20 deny icmp 192.168.120.0 0.0.0.255

try the following to see what shows up:

show running-config aclmgr
show vlan filter
show vlan access-map

You may have to consider using VLAN ACLs (VACL) instead of traditional ACLs.

switch(config)# show running-config aclmgr

!Command: show running-config aclmgr
!Time: Tue Apr  9 20:23:04 2019

ip access-list DENY_ACL_MAP
  10 deny ip 192.168.120.0/24 any
  20 deny icmp 192.168.120.0/24 any
  30 permit ip any any
vlan access-map DENY_TEST 10
        match ip address DENY_ACL_MAP
        action drop
vlan filter DENY_TEST vlan-list 120

Now run the same verification commands now
show running-config aclmgr
show vlan filter
show vlan access-map

Let me know if I've not missed your point completely.

Thanks Ronnie, you are tracking right with me. I thought of using VACL's too, but I can't get it work for some reason. Here is the output from "show run aclmgr" after entering your commands. Both workstations can still connect via ping and SMB with this configuratoin, just tested.. The heck?

WILNXLAB-01(config)# show run aclmgr

!Command: show running-config aclmgr
!Running configuration last done at: Tue Apr  9 13:49:37 2019
!Time: Tue Apr  9 13:49:40 2019

version 7.0(3)I7(6) Bios:version
ip access-list DENY_ACL_MAP
  10 deny ip 192.168.120.0/24 any
  20 deny icmp 192.168.120.0/24 any
  30 permit ip any any
vlan access-map DENY_TEST 10
  match ip address DENY_ACL_MAP
  action drop
vlan filter DENY_TEST vlan-list 120

WILNXLAB-02(config)# show run aclmgr

!Command: show running-config aclmgr
!Running configuration last done at: Tue Apr  9 13:49:33 2019
!Time: Tue Apr  9 13:49:52 2019

version 7.0(3)I7(6) Bios:version
ip access-list DENY_ACL_MAP
  10 deny ip 192.168.120.0/24 any
  20 deny icmp 192.168.120.0/24 any
  30 permit ip any any
vlan access-map DENY_TEST 10
  match ip address DENY_ACL_MAP
  action drop
vlan filter DENY_TEST vlan-list 120

@Adam-Tyler,

try this...I think I messed up and forgot that when using match statements, the extended ACL to use must permit, then match can drop it (yeah, intuitive right?). Try this. I don't have a complete lab setup for testing only the NX-OS demo.

SW1(config)# show running-config aclmgr

!Command: show running-config aclmgr
!Time: Wed Apr 10 10:20:24 2019

version 7.3(0)D1(1)
ip access-list DENY2VLAN1
  10 permit ip 192.168.120.0/24 any
  20 permit icmp 192.168.120.0/24 any
vlan access-map DENY2VLAN1MAP 10
        match ip address DENY2VLAN1
        action drop
vlan filter DENY2VLAN1MAP vlan-list 120

hopefully that works...
or try ..and configure the filter on vlan 1 instead?
this is me is relatively confusing since these things don't have ingress or egress attached to them. sigh

Thanks Ronnie. I did actually try the permit statements on the VACL approach too. I've tried a bunch of combinations now and the only thing I can seem to get working reliably is as follows..

This allows anything on the 192.168.119.0/24 network to ping anything on the 192.168.120.0/24 network as well as the reverse. It ONLY allows hosts on the 192.168.120.0/24 network to access SMB shares hosted on the 192.168.119.0/24 network (VLAN1). Additionally all other traffic is blocked. I am able to enable "statistics per-entry" and see the hit counter of the "deny ip any any" entry increment. Again, it only ever seems to block traffic "egressing/out" to its final destination.

ip access-list 120_out
  statistics per-entry
  10 remark ------Allow file sharing return traffic to VLAN 120
  20 permit udp 192.168.119.0/24 eq netbios-ss 192.168.120.0/24
  30 permit udp 192.168.119.0/24 eq netbios-ns 192.168.120.0/24
  40 permit tcp 192.168.119.0/24 eq 137 192.168.120.0/24
  50 permit tcp 192.168.119.0/24 eq 135 192.168.120.0/24
  60 permit tcp 192.168.119.0/24 eq 139 192.168.120.0/24
  70 permit tcp 192.168.119.0/24 eq 445 192.168.120.0/24
  80 remark ------Allow echo to VLAN 120
  90 permit icmp 192.168.119.0/24 any echo
  100 remark ------Allow echo reply traffic to VLAN 120
  110 permit icmp any any echo-reply
  120 remark ------Allow ospf
  130 permit ospf any any
  140 remark ------Deny everthing else
  150 deny ip any any
ip access-list 1_out
  statistics per-entry
  10 remark ------Allow file sharing to VLAN 1
  20 permit udp 192.168.120.0/24 192.168.119.0/24 eq netbios-ss
  30 permit udp 192.168.120.0/24 192.168.119.0/24 eq netbios-ns
  40 permit tcp 192.168.120.0/24 192.168.119.0/24 eq 445
  50 permit tcp 192.168.120.0/24 192.168.119.0/24 eq 139
  60 permit tcp 192.168.120.0/24 192.168.119.0/24 eq 135
  70 permit tcp 192.168.120.0/24 192.168.119.0/24 eq 137
  80 remark ------Allow echo to VLAN 1
  90 permit icmp 192.168.120.0/24 any echo
  100 remark ------Allow echo reply traffic to VLAN1
  110 permit icmp any any echo-reply
  120 remark ------Allow ospf
  130 permit ospf any any
  140 remark ------Deny everything else
  150 deny ip any any

interface Vlan1
  ip access-group 1_out out

interface Vlan120
  ip access-group 120_out out

So it's a mystery as to why ACLs only seem to work when applied to each interface "out" instead of "in". Any ideas on that one?

Additionally I did a ton of testing with the "established" ACL option trying to get the switch to respect return flow at least for TCP traffic. For example: 40 permit tcp 192.168.120.0/24 192.168.119.0/24 eq 445 established. This unfortunately never worked. I am surprised the Nexus doesn't seem to support session aware ACLs?

Why didn't the VACL work? I was reading an article recently about VACLs and there was some confusion around the VACL applying to traffic within the same VLAN only vs routed/inter-vlan traffic..

Thanks for your help and thoughts.

Regards,
Adam Tyler

@Adam-Tyler said in nxos and inter-vlan ACLs:

Why does this work?

You must view this from the standpoint of the vlan interface: So don't look at it independently but something similar to the following e.g. :

interface Vlan1
ip access-group 1_out in

should be viewed as more as inside to outside

interface Vlan1
ip access-group 1_out out

should be viewed as more as outside to inside

according to https://community.cisco.com/t5/switching/acl-between-vlans-on-3560g/td-p/1539959

This is why this works.

Thanks Ronnie, I am following the logic. Based on the article and your comments, it sounds like perhaps it's suggested that the ACL is simply built incorrectly to successfully apply inbound? Just to 100% confirm, I went ahead and created a new ACL that shows as follows:

ip access-list 120_in
  statistics per-entry
  10 deny ip any any log
  20 deny icmp any any log

After applying this to interface vlan 120 "in", ICMP still happily flows between hosts on VLAN 120 and VLAN 1. Did I miss something, because this is still not making any sense to me... This rule should block inbound echo from VLAN 120...? It should not apply to echo-reply coming from 119.

I also am getting no statistic counters or log entries indicating the rule is being actively applied to packets.

Regards,
Adam Tyler

@Adam-Tyler said in nxos and inter-vlan ACLs:

I also am getting no statistic counters or log entries indicating the rule is being actively applied to packets.

You need at least one permit statement.

SW1# show access-list 120_in

IP access list 120_in
        statistics per-entry
        10 deny ip any any log [match=0]
        20 deny icmp any any log [match=0]
        30 permit ip any any [match=0]

I'm still working the answer for the other part... let me get some more information
are you doing this between vlans on the same switch? or between vlans on different switches?

Hmm... Still no dice.. Starting to wonder if I have buggy firmware or something. These are 9000v VMware virtual machines running in the lab. It was a real trick figuring out how to get a VM on the same VMware host to connect to a switchport on the "virtual" nexus switch. I ended up making a separate VMware vSwitch for each host that needed to connect.

IP access list 120_in<------------Still not counters..
        statistics per-entry
        8 deny icmp any any log
        9 deny ip any any log
        10 permit ip any any log
        20 permit icmp any any log
WILNXLAB-01(config)# show run int vlan 120

!Command: show running-config interface Vlan120
!Running configuration last done at: Fri Apr 12 14:51:50 2019
!Time: Fri Apr 12 14:52:07 2019

version 7.0(3)I7(6) Bios:version

interface Vlan120
  no shutdown
  ip access-group 120_in in
  ip access-group 120_out out

Outbound seems to work exactly as expected.... WEIRD!

WILNXLAB-01(config)# show ip access-lists 120_out
IP access list 120_out
        statistics per-entry
        10 remark ------Allow file sharing return traffic to VLAN 120
        20 permit udp 192.168.119.0/24 eq netbios-ss 192.168.120.0/24 [match=0]
        30 permit udp 192.168.119.0/24 eq netbios-ns 192.168.120.0/24 [match=0]
        40 permit tcp 192.168.119.0/24 eq 137 192.168.120.0/24 [match=0]
        50 permit tcp 192.168.119.0/24 eq 135 192.168.120.0/24 [match=0]
        60 permit tcp 192.168.119.0/24 eq 139 192.168.120.0/24 [match=0]
        70 permit tcp 192.168.119.0/24 eq 445 192.168.120.0/24 [match=9326]
        80 remark ------Allow echo to VLAN 120
        90 permit icmp 192.168.119.0/24 any echo [match=0]
        100 remark ------Allow echo reply traffic to VLAN 120
        110 permit icmp any any echo-reply [match=83322]
        120 remark ------Allow ospf
        130 permit ospf any any [match=0]
        140 remark ------Deny everthing else
        150 deny ip any any [match=18]

@Adam-Tyler,

remember on traffic between vlans. The inbound and outbound terminology are weird. We have to think of directionality for the processing. ip access-group 120_in in when applied to the vlan interface is (from within the VLAN 120 outbound to another) . And ip access-group 120_in out when applied to the vlan interface means (from outside the vlan 120 heading in to the vlan).

so it's a bit twisted in our syntax to the meaning of the ACL.

Hey Ronnie, sorry for the delay on this. I checked in with a few consultants we work with regularly and the consensus is that this Cisco device is not working as it should. It's a 9000v Nexus virtual appliance and apparently ACLs aren't officially supported. Super interested in testing inbound ACLs on a physical Nexus device, but I don't have a spare handy.

Regards,
Adam Tyler

@Adam-Tyler,
Yes...I only have access to the simulator

Just to close the loop here, I ended up buying a couple of nexus devices for the lab and could not re-create this problem. It seems the nexus virtual appliance simply doesn't support inbound ACLs. That was fun. You just can't get the same level of design experience from virtual labs in some cases.

Regards,
Adam Tyler

@Adam-Tyler,

Yes. Sometimes there are limitations to what they can do. sigh