The Problem

Recently i had to migrate a bunch of files from one VmWare Hypervisor to another one. It turned out that there are multiple ways how to do this. The setup is as follows: 2 Computers - both of them are running VmWare ESXi vSphere Hypervisor. The are connected to the same LAN  via a 1 Gigabit connenction.

Solutions

As stated before - there are multiple ways how to copy data between those two machines. In this article i am focusing on techniques that do not require thrid party software to perform this task. The following sections are describing my attempts to copy an ISO image from one Host (192.168.30.3) to another host (192.168.30.79). All of these attempts are solving the problem - but the performance is different. All tests where performed on two similar machines with no additional load on them.

To be able to reproduce these steps you have to turn on the ESXi Shell on the hypervisor to get access via SSH. Then you can access the consoile via Putty in WIndows or ssh under Linux.

Copy via SSH / SCP

The most obvious solution is to copy the data via SCP from one machine to another. Even if you have turned on the ESXi shell you will discover taht a simple copy operation is not possible because the firewall blocks outgoing connections on port 22. So the forst step is to enable the SSH client in the firewall:

Once you have done this you can connect to the other machine via SSH/SCP and transfer some files:

So lets try to copy a 468 MB file from the source to the target:

We see that it takes 1:13 to copy 468 MB of data. Since this is not very fast i tried to copy it into the opposite direction - instead of sucking it from the source i wanted to push it onto the target:

This also wasn't as fast as i would have expected it to be - the machines are connected with a 1 GBit connection! Then i had the following idea: Lets send the data compressed via scp - this should speed up the effective datatransfer a little bit. This time t tried to send a 3GB file over the wire. The compression is done on-the-fly ba adding the -C flag to the scp command.

WHOW! The transfer rate dropped to 4.16 MB/sek. OK my next guess was that the system is some kind of busy but since there where no VMs running this could not be the case. So if the machine is busy then everything should be slow ... so lets copy the same file again but without compression.

... and again i was disappointed (or confused). After a cup of coffe and several attempts later I came to the following conclusion:

Copy via netcat

Some internet forums indicated that VmWare intentionally slows down scp operations on the admin-shell. Well - there are other possibilities how to get your data shipped. One of the most famous tools among hackers and system administrators is netcat - often reffered as the swiss army knife of networking. Fortunately netcat is installed out of the box on the ESXi hosts. So lets try to move the data by running netcat as a server on the destination machine. The tricky part was to find a port that could be used without the need to manipulate the firewall settings. I have coosen the port 8000 which is (according to a VMware Knowledgebase article) reserved for the vMotion service. Since i am not aware of the fact that my hosts are doing anything with vMotion this seemed to be a valuable choice.

On the receiving host i fired up netcat as a server that listens on port 8000 and writes everything it gets into the file acp_test_netcat.iso

On the sending host i fired up netcat as a client that connects to the server and transmits everything within the file acp_systemcd.iso

It took 213 seconds to copy 467 MB of data which means that netcat reaches 2.19 MB / sec. Since netcat utilizes neither a protocol to transfer the data nor a compression my (and the assertions of the internet forums) are confirmed: Some things are intentionally slowed down within the VMware ESXi Shell.

Copy via Python & wget

OK now it was time to figure out which programs are not crippled. Since VMware offers features to move data between two hosts i assumed that they slowed down everything in the standard shell that could be used for this purpose. To proove my theory I tried to to something that was not that usual: I searched for a way to write a server / client by myself. Then i stumbled over python. Python is a general purpose scripting language with a large standard library that usually enables you to run a webserver almost out of the box.

The other reasons to choose a python where:

• It is a general purpuse language - So it is not that easy to cripple a certain part of it (lets say limit network speed) because before you run a script the python interpreter has no idea what you are gonna do.
• It is used by VMware itself - So why would they taint their own tools.

Python 2.6 is installed out-of-the-box. So my idea was to use SimpleHTTPServer as a simple webserver on the senders side and thre preinstalled wget as http client on the receiving host. For all of you who are not aware of what SimpleHTTPServer does: This python class serves files from the current directory and below, directly mapping the directory structure to HTTP requests.

Unfortunately this class was not installed on my machines - so i had to download it befor i could start the server:

A first attempt to access the python webserver with a standard browser showed me this beautiful picture:

SUCCESS - Now i just had to download the files on the client:

BIG SUCCESS: The entirely uncompressed download of 2 files lead to the following results: