Part
01
of one
Part
01
What protocol do popular file storage companies, like Google Drive / Dropbox, use to transfer data from your computer to their servers? Is there a faster protocol that could be used in different circumstances? Can someone download a file directly from your computer (using it's IP) and is that faster/slower than using an intermediary cloud server?
Hello! Thanks for your question about the protocol popular file storage companies such as Dropbox and Google Drive use to transfer data from a user's computer to their servers and whether there is a faster protocol. Also you asked whether there is a way to download a file directly from one computer using its IP address to another computer and whether that method would be faster than using an intermediary cloud server.
For the first part of your question, the most useful sources my colleague found are a blog from Dropbox that explains how file synching works, an explanation of the difference between Dropbox and Google systems, and the synching mechanism of BitTorrent. In addition, we found two other protocols, Resilio (which is based on BitTorrent) and μTP2, as well as direct computer-to-computer transfer software.
The short answer is, the slowdown in file transfers in the cloud occurs because the downloading computer must wait for the uploading computer to finish, and because transfers are bound by bandwidth size. Dropbox has protocols to speed up modified files, but not original uploads and downloads. BitTorrent spreads the transfer over many computers, which can offer faster download times. The quickest method is direct transfer from computer to computer, over a LAN, but this is not always possible.
DROPBOX
Dropbox uses three systems for file transfers: Delta Sync, LAN Sync, and Streaming Sync.
- Delta Sync - This part of the Dropbox system transfers only the modified portions of files, instead of reloading the entire file. This greatly speeds up the process once you have already uploaded the initial file and are just making minor changes. This is much faster than Google Drive, which uploads the entire file every time you make changes.
- LAN Sync - As the name suggests, LAN Sync looks through a local area network for the key information prior to checking in cloud storage. LAN Sync only checks computers on the same local area network, which saves time if computers sharing a common router have also shared files.
- Streaming Sync - Streaming Sync is said to greatly decrease latency by beginning the download to a second device before the upload from the original device is complete. This is only useful for large files (such as when you want to share a video between your phone and your TV) because each file is separated into blocks of 4 MB each. Normally when a file is uploaded, the database must check to see whether the file already exists, then whether it must be updated. Next, the second device checks for updates, which are then downloaded. Streaming Sync allows the second device to "pre-fetch" the 4 MB blocks even before authentication, which means that, by the time Dropbox says to the second device, "Okay, you need an update and have the authority to receive it," the blocks are already on their way. Dropbox found speeds up to 1/4 faster using Streaming Sync than previous methods.
In all, Dropbox states that the improvements they have instituted make their service 5 times faster than Google Drive, Box and Microsoft OneDrive.
GOOGLE DRIVE
Google Drive operates in much the same way as Dropbox, though there is room for improvement across all protocols. Inefficiencies in Google Drive's protocol are a result of lack of adoption of deduplication (making sure that file contents are not re-transmitted) or bundling (transmitting numerous small "chunks" of data as one large one). From my research, I found that "Google Drive establishes a new connection for one file without bundling (suffering TCP slow start)". As stated above, Google Drive's protocol includes re-transmission of entire files rather than just the modified portions (known as Delta encoding), resulting in a slower process. However, Google Drive may be better than Dropbox when e-mail attachments are concerned.
PEER-TO-PEER COMMUNICATION
Peer-to-peer communication is the name for the protocol that allows computers to share information directly with each other, without accessing a cloud storage server.
- BitTorrent - Like Dropbox' Streaming Sync, BitTorrent protocol separates the original file (called the "seed") into chunks. Anyone who has downloaded the BitTorrent client program has given BitTorrent authorization to store bits of separated files on their computer (collectively, these are known as the "swarm"). When someone wants to download a file, BitTorrent uses a tracker to seek out each part of that file on the most convenient computer and sends it back to the requesting computer. This system evens out traffic on the internet and makes transfer of files generally faster. Two cautionary notes: BitTorrent relies on users of the protocol to make the files on their computers available for others by staying online even when their download is complete, and ISP's may disallow large amounts of data being transferred through a single computer at peak times--or ever.
- Resilio - This file transfer app is based on BitTorrent's protocol. Resilio can handle any size file, and, when transferring files over a LAN, claims speeds up to 16 times faster than Dropbox (for new files only).
- Resilio Connect - This system is used for enterprise environments, which can require the transfer of large amounts of data. It is based on the μTP2 protocol, which is able to adjust the rate of transfer of packet speed based on internet traffic and congestion on WAN, satellite and mobile networks, known for their high latency. The rate of packet transfer can be slowed down to resist loss of data, then sped back up when the congestion has eased.
DESKTOP SHARING SOFTWARE
For synching files between your own devices, TeamViewer and UltraVNC offer non-cloud options. Windows even has its own remote desktop option. However, each of these protocols requires that the remote computer you are accessing must be on.
VPN
A final option is to set up a Virtual Private Network, either between your own devices, or between your device and others'. Many workplaces set up VPN's for the benefit of their employees who must share files.
In conclusion, the major determiner of how quickly a file can be transferred from one computer to another is the bandwidth size allowed by the ISP. The fastest form of transfer is over a LAN, where a router is shared, but the internet is not accessed. However, this only works when the devices are physically near to one another. When accessing files from the internet, some form of BitTorrent will generally work faster than will cloud storage, because the blocks of data are all coming from different sources. A VPN can also allow direct transfer between computers of users who have agreed to share files. For known slow ISPs, such as satellite and mobile networks, the μTP2 protocol, with its ability to prevent loss of packets by changing transfer rate, is a good option--but right now it is only available for enterprise clients.
Thanks for using Wonder! Please let us know if we can help with anything else!