Difference between revisions of "Software:Rclone"

From HandWiki
Jump to: navigation, search
(import)
 
 
Line 136: Line 136:
  
  
{{Sourceattribution|Rclone}}
+
{{Sourceattribution|Rclone|1}}
 +
 
 +
 
 +
[[Category:Cloud applications]]

Latest revision as of 07:44, 31 July 2020

Rclone
Original author(s)Nick Craig-Wood
Developer(s)Nick Craig-Wood
Initial releaseJune 19, 2012; 8 years ago (2012-06-19)[1]
Stable release
1.52.2[1] / 24 June 2020; 41 days ago (2020-06-24)[1]
Written inGo[2]
Operating systemLinux, Windows, macOS, FreeBSD, NetBSD, OpenBSD, Plan9, Solaris[2]
PlatformIntel/AMD-64, Intel/AMD-32, ARM-32, ARM-64, MIPS-Big-Endian, MIPS-Little-Endian[2]
LicenseMIT
Websiterclone.org

Rclone is a command line program to manage files on cloud storage. It is promoted as an alternative to individual cloud storage vendors' web interfaces. The rclone website lists fifty supported backends including major S3 services and consumer file sharing platforms.[2]

Descriptions of Rclone often carry the strapline Rclone syncs your files to cloud storage.[2] Those prior to 2020 include the alternative Rsync for Cloud Storage.[3]

Users have called Rclone The Swiss Army Knife of cloud storage,[4] and likened it to magic.[5] Whilst rclone is best known[6] for its rlone sync and rclone mount commands, it provides a broad suite of file management utilities analogous to those ordinarily used for local disks. These rclone commands can be applied directly to the full range of rclone backends. In many cases they operate on directories, not files.[7]

Official Ubuntu, Debian, Fedora, Brew and Chocolatey package managers include rclone.[8]

Rclone is widely used at hpc research sites to transfer large datasets. Examples include the University of Maryland,[9] Iowa State University,[10] University of Utah,[11] National Institutes of Health,[12] Wharton,[13] Yale[14] and Berkeley.[15]

History

Nick Craig-Wood was inspired by rsync.[16] His wife's concerns about the noise and power bills arising from their home computer servers prompted him to begin developing rclone[17] in 2012.

Rclone was promoted to stable version 1.00 in July 2014.

In May 2017 Amazon controversially barred new rclone users from its consumer Amazon Drive data storage product. Amazon Drive had been advertised as offering unlimited storage for £55 per year. Amazon blamed security concerns and also banned other upload services.[18] Amazon's more expensive AWS S3 service continues to support new rclone users.

The original rclone logo was retired to be replaced with the present one in September 2018.[19]

Amazon's AWS April 2020 public sector blog explained how the Fred Hutch Cancer Research Center were using rclone in their Motuz tool to migrate very large biomedical research datasets in and out of AWS S3 object stores.[20]

May 2020 reports stated rclone had been used by hackers that attacked Diebold Nixdorf ATMs.[21][22]

Backends

The following back ends are supported. There are others, built on standard protocols such as WebDAV or S3, that work out of the box.[2]

1Fichier | Alibaba (Aliyun) Object Storage System (OSS) | Amazon Drive (See note) | Amazon S3 | Backblaze B2 | Box | Ceph | Citrix ShareFile | C14 | DigitalOcean Spaces | Dreamhost | Dropbox | FTP | Google Cloud Storage | Google Drive | Google Photos | HTTP | Hubic | Jottacloud | IBM COS S3 | Koofr | Mail.ru Cloud | Memset Memstore | Mega | Memory | Microsoft Azure Blob Storage | Microsoft OneDrive | Minio | Nextcloud | OVH | OpenDrive | OpenStack Swift | Oracle Cloud Storage | ownCloud | pCloud | premiumize.me | put.io | QingStor | Rackspace Cloud Files | rsync.net | Scaleway | Seafile | SFTP | StackPath | SugarSync | Tardigrade | Wasabi | WebDAV | Yandex Disk | The local filesystem

One or more remotes are defined from backends with rclone config. Rclone can further layer those remotes through virtual, encrypted, chunked, cached, vfs-cached or union backends. Once defined they are referenced by other rclone commands.

Note that remote names are followed by a colon to distinguish them from local drives. For example, a remote example_remote containing a folder or pseudofolder my folder is referred to within a command as example_remote:/myfolder.[23]

Rclone directly addresses remotes or mounts them as if a conventional disk. It also serves them over SFTP, HTTP, WebDAV, FTP and DLNA.[2]

Commands

Usage:

 rclone [flags]
 rclone [command]

Available Commands:[24]

 about           Get quota information from the remote.
 authorize       Remote authorisation.
 cachestats      Print cache stats for a remote
 cat             Concatenates any files and sends them to stdout.
 check           Checks the files in the source and destination match.
 cleanup         Clean up the remote if possible
 config          Enter an interactive configuration session.
 copy            Copy files from source to dest, skipping already copied
 copyto          Copy files from source to dest, skipping already copied
 copyurl         Copy url content to dest.
 cryptcheck      Cryptcheck checks the integrity of a crypted remote.
 cryptdecode     Cryptdecode returns unencrypted file names.
 dbhashsum       Produces a Dropbox hash file for all the objects in the path.
 dedupe          Interactively find duplicate files and delete/rename them.
 delete          Remove the contents of path.
 deletefile      Remove a single file from remote.
 genautocomplete Output completion script for a given shell.
 gendocs         Output markdown docs for rclone to the directory supplied.
 hashsum         Produces an hashsum file for all the objects in the path.
 help            Show help for rclone commands, flags and backends.
 link            Generate public link to file/folder.
 listremotes     List all the remotes in the config file.
 ls              List the objects in the path with size and path.
 lsd             List all directories/containers/buckets in the path.
 lsf             List directories and objects in remote:path formatted for parsing
 lsjson          List directories and objects in the path in JSON format.
 lsl             List the objects in path with modification time, size and path.
 md5sum          Produces an md5sum file for all the objects in the path.
 mkdir           Make the path if it doesn't already exist.
 mount           Mount the remote as file system on a mountpoint.
 move            Move files from source to dest.
 moveto          Move file or directory from source to dest.
 ncdu            Explore a remote with a text based user interface.
 obscure         Obscure password for use in the rclone.conf
 purge           Remove the path and all of its contents.
 rc              Run a command against a running rclone.
 rcat            Copies standard input to file on remote.
 rcd             Run rclone listening to remote control commands only.
 rmdir           Remove the path if empty.
 rmdirs          Remove empty directories under the path.
 serve           Serve a remote over a protocol.
 settier         Changes storage class/tier of objects in remote.
 sha1sum         Produces an sha1sum file for all the objects in the path.
 size            Prints the total size and number of objects in remote:path.
 sync            Make source and dest identical, modifying destination only.
 touch           Create new file or change file modification time.
 tree            List the contents of the remote in a tree like fashion.
 version         Show the version number.

Future

Rclone can serve a web based graphical interface - rclone rc[25].

There have been persistent calls for a --magic flag to improve rclone.[26][27][28][29][30]

Rclone applications

The rclone website lists third party developers that have used rclone within their own applications.[31]

Rclone and rsync

Rsync transfers files between computers it is installed on. It operates at the block level and has a delta algorithm so that it only needs to transfer changes in files. Rclone has a wider range of file management capabilities, and types of backend it can address, but only works at a whole file/object level.[32] Rclone decides which of those whole files to transfer after obtaining a checksum, to compare, from the remote server. In extremis, where checksums are not available, it can be configured to use file size and time.

See also

References

  1. 1.0 1.1 1.2 "Changelog". https://rclone.org/changelog/. 
  2. 2.0 2.1 2.2 2.3 2.4 2.5 2.6 "Rclone". https://rclone.org/. 
  3. https://github.com/rclone/rclone
  4. "Issues with special characters in filenames". June 6, 2020. https://forum.rclone.org/t/issues-with-special-characters-in-filenames/16955/11. 
  5. "Stopping and starting rclone mid transfer". October 11, 2016. https://forum.rclone.org/t/stopping-and-starting-rclone-mid-transfer/60. 
  6. https://itsfoss.com/use-onedrive-linux-rclone/
  7. https://rclone.org/commands/rclone_copy/
  8. https://repology.org/project/rclone/versions
  9. https://hpcc.umd.edu/hpcc/help/software/rclone.html
  10. https://www.hpc.iastate.edu/guides/rclone
  11. https://www.chpc.utah.edu/documentation/software/rclone.php
  12. https://hpc.nih.gov/apps/rclone.html
  13. https://research-it.wharton.upenn.edu/tools/rclone/
  14. https://docs.ycrc.yale.edu/clusters-at-yale/guides/rclone/
  15. https://research-it.berkeley.edu/services/high-performance-computing/transferring-data-between-savio-and-your-bdrive-google-drive
  16. https://github.com/rclone/rclone
  17. "Is rclone more efficient than an nfs mount". June 10, 2020. https://forum.rclone.org/t/is-rclone-more-efficient-than-an-nfs-mount/16974/6. 
  18. https://www.theregister.com/2017/05/23/amazon_drive_bans_rclone_storage_client/
  19. https://forum.rclone.org/t/a-new-logo-for-rclone/7019
  20. https://aws.amazon.com/blogs/publicsector/how-fred-hutch-unlocks-siloed-data-open-source-software/
  21. https://www.scmagazineuk.com/diebold-nixdorf-atm-attack-prolock-ransomware-used-qakbot-trojan-access-networks/article/1683611
  22. https://www.bleepingcomputer.com/news/security/prolock-ransomware-teams-up-with-qakbot-trojan-for-network-access/
  23. https://www.howtogeek.com/451262/how-to-use-rclone-to-back-up-to-google-drive-on-linux/
  24. https://rclone.org/commands/
  25. https://rclone.org/gui/
  26. https://forum.rclone.org/t/rclone-1-52-release/16718/7
  27. https://forum.rclone.org/t/mkdir-on-b2-consider-adding-some-output/17689/6
  28. https://forum.rclone.org/t/copy-local-files-to-google-drive-td-with-drive-server-side-across-configs/16653/8
  29. https://forum.rclone.org/t/google-drive-very-slow-and-way-below-quota/18004/8
  30. https://forum.rclone.org/t/total-newbie-question-about-rclone-cloud-to-cloud-copy-speeds/17603/8
  31. "rclone / rclone". https://github.com/rclone/rclone. 
  32. https://forum.rclone.org/t/difference-between-rclone-and-rsync-when-syncing-to-the-local-filesystem/3088/2

External links