What's the best way to transfer files between pogoplugs

This forum is for all other ARMv5 devices

Re: What's the best way to transfer files between pogoplugs

Postby momentum » Tue Jan 10, 2012 4:45 am

Not yet. I used rsync -P with a single file for testing since I found it would show the transfer speed. Don't know how cp would show the speed information. still googling it...

$this->bbcode_second_pass_quote('WarheadsSE', 'H')ave you tried `cp -ar` ?
momentum
 
Posts: 22
Joined: Mon Jan 09, 2012 2:09 am

Re: What's the best way to transfer files between pogoplugs

Postby jkgm » Tue Jan 10, 2012 7:07 am

$this->bbcode_second_pass_quote('WarheadsSE', 'T')hat rsync method uses ssh, which isn't exactly the worlds most performant network transport.

Really? I was doing a network rsync over 1000Mbit today and getting sustained 65+MB/sec (not Mb/sec!). That was from an iMac to a Mac Server, but unless the ALARM ports are specifically broken I'd say there's absolutely nothing wrong with rsync/ssh for file transfers. I always use the -W flag for LAN or local transfers, and it helps a lot.

For reference, here are what the flags I normally use do (from the man page):
$this->bbcode_second_pass_code('', 'Flags I use for rsync:
-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
-v, --verbose increase verbosity
-x, --one-file-system don't cross filesystem boundaries
-P same as --partial --progress
-W, --whole-file copy files whole (w/o delta-xfer algorithm)
--inplace update destination files in-place

Individual flags from combined flags above:
-r, --recursive recurse into directories
-l, --links copy symlinks as symlinks
-p, --perms preserve permissions
-t, --times preserve modification times
-g, --group preserve group
-o, --owner preserve owner (super-user only)
-D same as --devices --specials
--progress show progress during transfer
--partial keep partially transferred files
--devices preserve device files (super-user only)
--specials preserve special files
')
If the problem is the Pogoplug being too slow to cope with standard SSH compression, you can also turn that off with --no-z
$this->bbcode_second_pass_code('', ' --no-OPTION turn off an implied OPTION (e.g. --no-z)
-z, --compress compress file data during the transfer
')
jkgm
 
Posts: 18
Joined: Thu Dec 15, 2011 10:51 am

Re: What's the best way to transfer files between pogoplugs

Postby WarheadsSE » Tue Jan 10, 2012 6:15 pm

*cough*

PLATFORM PERFORMANCE.

You're comparing a Volkswagon Beetle to a Factory V8 Mustang.
Core Developer
Remember: Arch Linux ARM is entirely community donation supported!
WarheadsSE
Developer
 
Posts: 6807
Joined: Mon Oct 18, 2010 2:12 pm

Re: What's the best way to transfer files between pogoplugs

Postby Geoff » Tue Jan 10, 2012 7:13 pm

I use rsync all the time, and I find it reliable but not very fast. I think what slows it down on the Pogo is where it computes a digest of each block of a file to compare with that of the file at the opposite end of the transfer. The PLX7820 chip in the Pogo Pro contains a specialized hardware module for doing digest and encryption. I think it might make a big difference if this module were used, instead of using the general-purpose CPU to digest the file blocks. However, I doubt that a kernel module exists to take advantage of this.
Geoff
 
Posts: 231
Joined: Wed Mar 09, 2011 5:14 pm

Re: What's the best way to transfer files between pogoplugs

Postby jkgm » Tue Jan 10, 2012 8:05 pm

$this->bbcode_second_pass_quote('WarheadsSE', '*')cough*

PLATFORM PERFORMANCE.

You're comparing a Volkswagon Beetle to a Factory V8 Mustang.

LOL fair point. :lol:
jkgm
 
Posts: 18
Joined: Thu Dec 15, 2011 10:51 am
Top

Re: What's the best way to transfer files between pogoplugs

Postby momentum » Wed Jan 11, 2012 4:21 am

I tried several tweaks around rsync. The best speed i got is around 10MB/s with my situation, using "rsync -WP --inplace" and mount cifs (rather than ssh). I guess that's it since the limitation of pogoplug (cpu, ram, etc)? not so bad for me for now since I could at least run those job on the background now without keep my pc open.

$this->bbcode_second_pass_quote('WarheadsSE', 'S')ure,
- copy directly, instead of from Windows (via samba)
- make sure the copy is over ethernet, not wireless
- use rsync, if possible
- depending on the speed of actual disks, that might be a hamper.

The best I have managed is about 22MB/s sustained, which is 170Mb/s over gigabit ethernet. Throw wireless in its place and it would have capped at ~15MB/s, maybe. The fact that there is not 1 GB of RAM makes a huge difference to what people are accustomed to as ""normal""
momentum
 
Posts: 22
Joined: Mon Jan 09, 2012 2:09 am
Top

Re: What's the best way to transfer files between pogoplugs

Postby WarheadsSE » Wed Jan 11, 2012 1:31 pm

Right, and I should say that my 22MB/s was to a SATA disk, with ext3/4. I will make note that with the iOmega HMCE, which has 256MB ram, I was able to sustain ~30MB/s so it's obvious RAM is a limiting factor.
Core Developer
Remember: Arch Linux ARM is entirely community donation supported!
WarheadsSE
Developer
 
Posts: 6807
Joined: Mon Oct 18, 2010 2:12 pm

Previous

Return to Community Supported

Who is online

Users browsing this forum: No registered users and 40 guests