Downloading source tarballs with zsync

Another interesting application for zsync: downloading updated source tarballs. For regular software updaters, this has to be a big win. I wrote a small wrapper script for the fetch operation in the FreeBSD ports system, and was able to cut the total download for updating XFCE by roughly a third.

% make FETCH_CMD="/home/cph/src/zsync-fetch -ARr"
===>  Vulnerability check disabled, database not found
===>  Found saved configuration for xfce4-wm-4.2.1
=> xfwm4-4.2.1.tar.gz doesn't seem to exist in /usr/ports/distfiles/xfce4.
=> Attempting to fetch from http://www.us.xfce.org/archive/xfce-4.2.1/src/.
Downloading xfwm4-4.2.1.tar.gz; old versions ./xfce4/xfwm4-4.0.6.tar.gz found.
#################### 100.0% 21.2 kBps DONE    
Read /usr/ports/distfiles/./xfce4/xfwm4-4.0.6.tar.gz. Target 27.4% complete.
downloading from http://www.us.xfce.org/archive/xfce-4.2.1/src/xfwm4-4.2.1.tar.g
z:
#################### 100.0% 47.3 kBps DONE    
verifying download...checksum matches OK
used 1681408 local, fetched 857709
===>  Extracting for xfce4-wm-4.2.1
=> Checksum OK for xfce4/xfwm4-4.2.1.tar.gz.
…
=> libxfcegui4-4.2.1.tar.gz doesn't seem to exist in /usr/ports/distfiles/xfce4.
=> Attempting to fetch from http://www.us.xfce.org/archive/xfce-4.2.1/src/.
Downloading libxfcegui4-4.2.1.tar.gz; old versions ./xfce4/libxfcegui4-4.0.6.tar
.gz found.
#################### 100.0% 15.2 kBps DONE
Read /usr/ports/distfiles/./xfce4/libxfcegui4-4.0.6.tar.gz. Target 33.8% complete. 
downloading from http://www.us.xfce.org/archive/xfce-4.2.1/src/libxfcegui4-4.2.1.tar.gz:
#################### 100.0% 48.8 kBps DONE
verifying download...checksum matches OK
used 1582080 local, fetched 548199
===>  Extracting for libxfce4gui-4.2.1
=> Checksum OK for xfce4/libxfcegui4-4.2.1.tar.gz.
…

In total, a 6.1MB download was reduced to 4.0MB. You can see the benefits of the recompression support added in 0.3.2 now — it reconstructs the original .gz file, allowing it to pass the checksum applied by the ports framework. This was on a fairly large update, from 4.0.x to 4.2.x, so a smaller update would probably see a larger benefit; this has to be a big application of zsync, I think.


For the moment I am running a CGI on moria.org.uk which will automatically generate .zsync files for FreeBSD package sources. And I am providing the zsync-fetch wrapper script which you see in use above. Feel free to try it out (note that you should upgrade to zsync-0.3.3 at least before using it, though, as 0.3.2 had some glitches in the recompression support) — it obviously only kicks in if you have an older version of a file to update from, and it only works for ports with gzip-compressed source tarballs (it just falls back to fetch for .tbz, .tar.bz, and files that moria.org.uk rejects or is unable to download). This is highly experimental, but I hope gives a good idea of the potential that zsync has.