Don’t Use NFS 4 for Your ESXi Storage (and What to Do If It Goes Wrong)

NFS 4.1 brought a lot new features comparing to its predecessor NFS 3: Multipath, “reliable” transmission based on TCP, and better authentication support. And ESXi comes with native support for NFS 4.1. But what if I tell you that, under a very specific circumstance, NFS 4.1 will result in all your VMs being forcefully powered off, and causing data loss?

TL; DR: Avoid NFS 4.1, stick with NFS 3 on your vSphere cluster, especially if your storage backend have relatively poor performance.

Continue reading

Flux Language Tutorial with Grafana

Flux Language is the shiny new query language that comes with InfluxDB 2.0. Since it is “the future”, and the OSS (free) version of InfluxDB 2.0 is not getting the old query language support in (at least) the first few versions, I’ve been using Flux Language for some new projects. Meanwhile, the (functional) documentation for both the Flux Language and its support in Grafana does not exist yet. In this post, I’d like to give some examples to quickly address common data processing needs, for both server monitoring and simple BI usages.

Continue reading

Interoperating Cisco DMVPN, Huawei DSVPN and OpenNHRP

Traditional point-to-point site-to-site VPN protocols require extensive setup in certain use cases. For example, if you want shortcuts between branch offices rather than let every packet go through the HQ, then you need to set up a cartesian product of tunnels by hand, which is time-consuming and error prone. So, people want something better, something easy to set up and maintain, and dynamic enough. While there are already a lot dynamic point-to-multipoint or full-mesh site-to-site VPN implementations (e.g. Tinc VPN, ZeroTier) on Linux, you don’t have many choices on these commercial black boxes.

Cisco DMVPN (Cisco Dynamic Multipoint VPN) is one solution to this. Huawei also had their DMVPN-compatible solution called DSVPN (Dynamic Smart VPN). Since the protocol is largely compatible, I’ll just reference it as DMVPN.

Continue reading

What the Flow: Reverse Route Modes on Juniper SRX

A SRX is a “security device”, or as we call it conventionally, a firewall. Modern layer-3 firewalls route packets just like a router, but unlike a router, a firewall can organize packets into connections (flows) and run ACLs on the entire flow. This unique functionality is the fundamental building block of every “advanced” security feature offered by a firewall: dynamic NAT (PAT/NPT), zone-based firewall (ZBFW), ACLs for in or out connections only, L7 filtering, etc. For the connection (flow) tracking to work, all the packets in a connection must go through the same device, and the 5-tuple of all the packets in a connection must be of expected values, which usually means:

  • The packets from A to B and the packets from B to A must all go through the firewall at some point
  • There shouldn’t be single-sided stateless NAT happening on the route

This was never an issue when everyone was single-homed and all the routers had only one routing table. But not today. SRXs now have built-in support for virtual routers which can create an asymmetric flow easily. Let’s look at this simplified topology:

Continue reading

Cisco Aironet 1800i: Hardware Detail

Cisco Aironet 1800i is a cute little device that is just a little smaller than my hand. They are light in weight, not very hot (not a good replacement of the old 3502i model if you also have a cat around your home) and require less power to operate. I recently got one 1800i in my room, so I’d like to write a little about this model since it is so different from the old PowerPC-based ones.

Continue reading

Multicast VXLAN for Routers

VXLAN has been around for a while, so how do router vendors support it? Well, let’s use a dead simple topology to test them out.

Our setup today:

  • All routers connected to the same dumb switch using IP range 169.254.0.0/24
  • Multicast signaling on address 239.0.0.1, No PIM
  • VXLAN UDP port 4789
  • Network 10.0.0.0/24 on VNI 5000 (layer 3 termination / inter-VXLAN routing)

Continue reading

Setting up an ESXi Cluster

So you have a handful of brand new ESXi servers, and want VMs to automagically move here and there based on host availability and resource usage; vCenter have you covered with the DRS and HA but obviously you need to put all the hosts into a cluster for these thing to work. What you might not know is that there are 3 ways of creating a cluster which differs in certain things, and you will regret it if you choose the wrong one. Trust me, I learned it the hard way.

Note: we are using ESXi 7.0 and vCenter 7.0 here.

Continue reading

MOP: The Unknown “MAC Telnet” Protocol on Cisco Routers

When I was replacing all my buggy little MikroTik RouterOS boxes and VMs with some new shiny (and also buggy) Cisco ISR1000s and CSR1000vs a few years ago, there were several things that I missed so much that existed on the former but not on the latter. One of them was the “MAC Winbox” and “MAC Telnet” capability with which you can plug your maintenance workstation into the router with an Ethernet cable, fire up a Winbox, and it will let you configure the router through a layer 2 connection. It require no valid IP configuration, so it would work as long as you doesn’t shut down the port and there is no wild switch ACL in place. Newer routers have USB console ports, and I do have a console cable in my EDC, but a router’s ability to be configured without a console cable is still its big advantage to me.

Imagine my face today when I learned that Cisco routers (IOS and IOS XE) do support a layer 2 protocol with remote console capability. And the protocol is not new. The protocol is from the 1980s and IOS has been quietly supporting it for years. It has even been enabled by default for years. It is still being supported (as of IOS XE 17.2).

Continue reading