KVM Macvtap vs bridging

2014-03-03

I’ve been using KVM based virtual machine as a web server for a while. The VM needs a public IP of course, so I used a bridged setup. Recently (2014-03-03), however, one of the other machines on my subnet generated some 3Mbps of traffic (to an external server). This caused my host machine CPU usage to rise to 40%!

I’m guessing this is because the bridged setup put eth0 in promiscuous mode, and the host kernel has to inspect each packet and decide whether to pass it to the VM guest or not. I switched my networking setup to macvtap, and got a huge performance improvement. The 3Mbps of traffic barely puts a dent on my CPU usage.

macvtap Setup

Run virt-manager and do the following:

Select Add Hardware / Network.
Set Network source to Host device ethXX : macvtap. (Here ethXX is your host machines outgoing interface.)
Set Source Mode to VEPA
Set Device Model to virtio.

Communication with the host machine.

The only caveat with VEPA is that your host machine CAN NOT communicate to the guest machine over this interface (unless you have some special hair pin switch.) I worked around this by creating a second NAT interface for the guest, and assigning static IP’s as follows:

Use virt-manager to create a second (NAT) interface for the guest. Call it default. Get the interface up and running in the guest.
Run virsh net-autostart default to make sure the interface comes up automatically after reboots.
Run virsh net-edit default. It will pop up an XML file with basic network configuration. Find the <dhcp> section, and below the <range start... /> line add the line
```
<host mac='guestmac' name='guestname' ip='192.168.122.2' />
```
Edit /etc/hosts on the host, and add the line
```
192.168.122.2   guest.host.name
```
Edit /etc/hosts on the guest, and add the line
```
192.168.122.1   host.host.name
```
Make sure that the private network doesn’t get used for internet access. (If it does, your VM will be able to access the internet, but will not be reachable from the internet.) Let’s assume your public (macvtap) interface is called eth0, and the private (NAT) interface is called eth1. Edit /etc/dhcp/dhclient.conf and add the following at the end:
```
# eth1 is only for communication with VM host, so don't request routers, etc.
interface "eth1" {
    request subnet-mask, broadcast-address, time-offset, host-name,
            interface-mtu, rfc3442-classless-static-routes;
}
```
When your interfaces come up type ip route show and confirm that your default route uses eth0 and not eth1.

Now you should have transparent host/guest access on all machines. (Accessing the guest via an IP address still won’t work though; you’ll have to use host-names.)

🗫 Comments

Traffic to guest extremely slow after upgrade to 1.1.2+dfsg-6+deb7u3

GI (2014-06-18 17:31:13)

After upgrading I found that the network on my host machine was dropping up to 20% of incoming packets. This had the effect of reducing the incoming speed to less than 100kbps, while the outgoing speed was a good 20mbps+.

I found that setting the device model to virtio fixed it. (The device model was Default earlier.)
One more reason to use this instead of a bridge

Francois (2014-07-16 22:49:51)

Thanks for the nice and simple post. Another reason to use macvtap is that if a bridge is added in a corporate network you are most likely to face a shutdown of the physical network port. This happens if the bridge announces itself by sending BPDUs.
Typo?

Vincenzo (2014-10-15 10:16:09)

I fear it is “VEPA” ( virtual ethernet port aggregator) and not “VPEA”.
Re: Typo?

GI (2014-10-15 13:59:16)

Oops. Indeed; I fixed it.
lost at the beginning

Stefan (2015-07-05 20:35:50)

I’d like to work through this manual, but at the very beinning I already get lost. Perhaps it is due to the translation of virt-manager’s interface (v1.0.1 running on Netrunner v16 distro) :

I can not find the place / menu to

Add Hardware –> Network

In the following I had to translate back - so please excuse the different terms:
1. So if i right - klick on localhost (QEMU) … I can go on with New VM - wrong place
2. Menu File - Add connection - wrong place
3. Menu File - New virtual machine - see 1. - wrong place
4. Edit Connection details seems promising due to this contains the tabs: “overview / Virtual Netzworks / Memory(or storage)/ LAN-Adapters (containing lo) 4a) when I add a LAN connection the only choices are: Bridge / Bond / Ethernet / VLAN Here I added eth0 and neither can make it a macvtap nor can I delete / remove this choice. Even not by calling sudo virt-manager
So where might I add the eth0 macvtap?
Thanks for Useful info

Himanshu (2017-10-22 14:52:21 EDT)

Thanks for quick tip for tuning my KVM machine, facing similar issue i will try the same. - Thanks Himanshu Blogger @
problem with setting the private NAT not default

hakayova (2018-03-24 14:54:08 EDT)

Thank you for a very useful and easy-to-understand, easy-to-follow guide. My only problem is with step 6, where we define private NAT to be not default for internet communication. I use Manjaro linux, which is Arch-based, and I don’t have a /etc/dhcp/dhclient.conf file in the VM (nor in the host for that matter). I believe these settings can be made by issuing an ip route change or replace command, but I am not technologically savvy enough to compose this command. Additionally, I am not sure if those setting would be saved after rebooting the VM. Your input and help troubleshoot this step will be greatly appreciated. I recently set up a Pi-Hole server in this VM and although I could get dmasq listen to the right interface (macvtap), I could not get it answer to the same interface, no matter how hard I tried. It would always answer to the private NAT for whatever reason. I eventually removed the private NAT interface from the VM via virt-manager to make it work. However, this effectively caused the loss of network communication between the host and VM as expected.
Re: problem with setting the private NAT not default

GI (2018-03-24 17:41:51 EDT)

The ip route command you want is
```
ip route change default via x.x.x.1 dev eth0
```
Here x.x.x.1 is the ip address of the gateway on eth0 (which I’m assuming is your public macvtap interface). Usually this is the IP address of the interface with the last digit replaced by 1.

This won’t persist across reboots. My suggestion to get something permanent is to figure out what DHCP client your system is using and configure it to not set the default route on your NAT interface. A few other common clients are dhcpcd, pump and udhcpc. Try seeing if one of these is installed and running on your system, and then look up the appropriate configuration instructions.
Re: problem with setting the private NAT not default

hakayova (2018-03-24 22:25:55 EDT)

Thank you so much for your prompt reply. While preparing to tackle the problem, I first added the private NAT again to the VM via virt-manager. When I booted up the VM, I happily noticed that this interface (enp0s3) was no longer marked as default (previously, both interfaces were marked as default). That was it! Everything is working as intended right now. Thank you again for this wonderful guide!
Re: problem with setting the private NAT not default

GI (2018-03-25 09:13:15 EDT)

On 2018-03-24, hakayova wrote:

While preparing to tackle the problem, I first added the private NAT again to the VM via virt-manager. When I booted up the VM, I happily noticed that this interface (enp0s3) was no longer marked as default (previously, both interfaces were marked as default). That was it!

Be careful! If when your machine boots, your network interfaces are initialized in a different order then it might stop working again. To get it working consistently, you will likely have to find a way to bring up your interfaces in the same order every boot. (I’m not sure how to do this, but I’m sure it’s possible…)

Good luck.

GI
No host communication in *any* mode

Bryan (2018-08-04 09:44:26 EDT)

When you singled out VEPA mode I had hope that other macvtap modes, there is also bridge mode and private mode, might allow communication with the host. None of them allow communication with the host. With private mode not even a hairpin switch will allow communication with the host. With bridge mode guests on the same machine will be able to communicate with each other, but not with the host.
Rant

Matt (2022-09-28 14:41:24 EDT)

I so HATE it when someone decides that because they’ve never heard of a given 3 letter acronym in their field that there mush to BE one. For 40 years KVM has been “keyboard, video, Mouse” and refers to a switcher which allows one to use multiple machines with the same desktop. If this other specialty is less than 40 years old it needs to find another name. If it is more than 40 years old it needs to find a new acronym for dealing with computers. You may notice there is no third option.
Re: Rant

Gautam Iyer (2022-09-28 16:18:54 EDT)

😄. I feel your pain. Perhaps you can rant a bit to the people who coined the name.

I am but pawn in the quest for 3 letter acronym dominion.