Fast Data Transfer between different hosts

I had some doubt about my OVM configuration running an Oracle Linux Server. The transfer rate from winscp (windows desktop machine – network card with 1 Gbps) to my dom-0 machine (100Mbs) was pretty bad and it’s a know problem, an average of 6Mbs. So it takes hours/day to transfer big file.

On my dom-0 machine, I have the following configuration about my network interface. So the maximum limit i can get is a Speed of 100Mb/s

[root@t610 ~]# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 100Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: g
        Wake-on: d
        Link detected: yes

Using cygwin, show a better rate transfer, but it is not using the full bandwith, with a rate of 11Mb/s

$ time scp vbox-oracle-1.2.1-disk1.vmdk root@xxx.xxx.xxx.xxx:/OVS/Repositories/0004fb00000300003527d62008dfea1f/VirtualDisks/
vbox-oracle-1.2.1-disk1.vmdk                                                                                                                                                            100% 6801MB  11.3MB/s   10:03

real    10m8.034s
user    1m45.361s
sys     0m29.498s

I discover a nice tools, written in Java to speed up rate transfer. It is explained and documented clearly on this web site : Fast Data Transfer. After running the process, on my destination host, and using the same JAR as a client, the transfer now use almost the full the bandwidth !

java -jar fdt.jar -ss 1M -P 10 -c <HOST-IP> vbox-oracle-1.2.1-disk2.vmdk.raw -d /OVS/Repositories/0004fb00000300003527d62008dfea1f/VirtualDisks/
$ time java -jar fdt.jar -ss 1M -P 10 -c xx.xx.xx.xx vbox-oracle-1.2.1-disk2.vmdk.raw -d /OVS/Repositories/0004fb00000300003527d62008dfea1f/VirtualDisks/
FDT [ 0.20.0-201407152021 ] STARTED ...
Sep 28, 2014 11:02:44 AM lia.util.net.common.Config <init>
INFO: Using lia.util.net.copy.PosixFSFileChannelProviderFactory as FileChannelProviderFactory
Sep 28, 2014 11:02:44 AM lia.util.net.common.Config <init>
INFO: FDT started in client mode
FDT uses *blocking* I/O mode.
Sep 28, 2014 11:02:45 AM lia.util.net.copy.transport.ControlChannel initStreams
INFO: NEW CONTROL stream for 4e5908a6-0633-4a7d-bf20-6a368398c7b1 initialized
Sep 28, 2014 11:02:45 AM lia.util.net.copy.transport.ControlChannel initStreams
INFO: App KeepAlive [ 120 second(s) ] enabled for control channel. Local FDTVersion [0.20.0-2014-07-15], Remote FDTVersion [0.20.0-2014-07-15]
Sep 28, 2014 11:02:45 AM lia.util.net.copy.transport.ControlChannel$ControlChannelPingerTask <init>
INFO: [ ControlChannelPingerTask ] initialized
Sep 28, 2014 11:02:45 AM lia.util.net.copy.FDTReaderSession internalInit
INFO: [ FDTReaderSession ] No pre processing filters defined/processed.
Sep 28, 2014 11:02:46 AM lia.util.net.copy.transport.SpeedLimitManager addLimiter
INFO:  Adding SpeedLimiterTask for lia.util.net.copy.transport.TCPSessionWriter@1f89644 delay: 300 ms
Sep 28, 2014 11:02:46 AM lia.util.net.copy.FDTReaderSession startReading
INFO:  Started DiskReaderTasks for the following partions [ 0 ] for FDTSession: 4e5908a6-0633-4a7d-bf20-6a368398c7b1
Sep 28, 2014 11:02:46 AM lia.util.net.copy.transport.TCPTransportProvider tryToConnect
INFO: Requested window size 1048576. Using window size: 1048576
28/09 11:02:54  Net Out: 90.602 Mb/s    Avg: 90.602 Mb/s
28/09 11:02:59  Net Out: 93.920 Mb/s    Avg: 92.252 Mb/s
28/09 11:03:04  Net Out: 93.939 Mb/s    Avg: 92.814 Mb/s
28/09 11:03:09  Net Out: 97.294 Mb/s    Avg: 93.934 Mb/s 01.28% ( 30m 05s )
28/09 11:03:14  Net Out: 93.939 Mb/s    Avg: 93.935 Mb/s 01.56% ( 30m 00s )
28/09 11:03:19  Net Out: 93.939 Mb/s    Avg: 93.936 Mb/s 01.83% ( 29m 55s )
28/09 11:03:24  Net Out: 95.635 Mb/s    Avg: 94.178 Mb/s 02.11% ( 29m 45s )
...
28/09 11:33:05  Net Out: 95.635 Mb/s    Avg: 93.969 Mb/s 99.48% ( 09s )
28/09 11:33:10  Net Out: 90.602 Mb/s    Avg: 93.960 Mb/s 99.74% ( 04s )
28/09 11:33:15  Net Out: 88.906 Mb/s    Avg: 93.946 Mb/s 100.00% ( 00s )
Sep 28, 2014 11:33:15 AM lia.util.net.copy.FDTReaderSession handleEndFDTSession
INFO: [ FDTReaderSession ] Remote FDTWriterSession for session [ 4e5908a6-0633-4a7d-bf20-6a368398c7b1 ] finished OK!


FDTReaderSession ( 4e5908a6-0633-4a7d-bf20-6a368398c7b1 ) final stats:
 Started: Sun Sep 28 11:02:45 CEST 2014
 Ended:   Sun Sep 28 11:33:18 CEST 2014
 Transfer period:   30m 33s
 TotalBytes: 21474836480
 TotalNetworkBytes: 21474836480
 Exit Status: OK

Sep 28, 2014 11:33:18 AM lia.util.net.copy.FDTReaderSession doPostProcessing
INFO: [ FDTReaderSession ] Post Processing started
Sep 28, 2014 11:33:18 AM lia.util.net.copy.FDTReaderSession doPostProcessing
INFO: [ FDTReaderSession ] No post processing filters defined/processed.
 [ Sun Sep 28 11:33:19 CEST 2014 ] - GracefulStopper hook started ... Waiting for the cleanup to finish
 [ Sun Sep 28 11:33:19 CEST 2014 ]  - GracefulStopper hook finished!

 [ Sun Sep 28 11:33:19 CEST 2014 ]  FDT Session finished OK.


real    30m37.482s
user    0m0.015s
sys     0m0.000s

Thanks for reading !

Advertisements

About Chenda Mok

19 years of hands on experience in software design and development with emphasis on Enterprise Application Integration (EAI), Services Oriented Architecture (SOA) and Identity Management (IDM) solutions. I’m a software engineer, member of the professional service delivery team working for Salesforce. Prior to this, I worked for Oracle as Solution Architect, through SeeBeyond(06/2005), then SUN’s acquisition (04/2009). After my master’s degree in computer science in 1997; I always delivered consulting on architecture, design, implementation on integration’s field. I’m interested in architecture using EAI/SOA/IDM/BPM/Cloud technologies, software development and Java’s related technologies. I may blog about my work/activities at Salesforce, but I do not speak for my employer, past, present or future.
This entry was posted in Oracle manager (ovm manager), Oracle server (ovm server) and tagged , , . Bookmark the permalink.