Data Transport Between Different Machines (Synchronization)

I don't have experience with this topic yet. So just a survey about some means of data transport and maintaining data consistency between different machines.

Useful Hardware

  1. external harddisks

  2. ZIP drive

Wade Hampton wrote: "You may use MS-DOS formatted ZIP and floppy discs for data transfer. You may be able to also use LS120. If you have SCSI, you could use JAZ, MO or possibly DVD-RAM (any SCSI disc that you could write to). I have the internal ZIP for my Toshiba 700CT. It works great (I use automount to mount it). I use VFAT on the ZIP disks so I can move them to Windows boxes, Linux boxes, NT, give them to coworkers, etc. One problem, I must SHUTDOWN to swap the internal CD with the ZIP."

Useful Software

Version Management Software

Although it is certainly not their main aim, version management software like CVS (Concurrent Version System) are a perfect tool when you work on several machines and you have trouble keeping them in sync (something which is often called disconnected filesystems in the computer science literature). Unlike programs like rsync, which are asymmetric (one side is the master and its files override those of the slave), CVS accept that you make changes on several machines, and try afterwards to merge them. Asymmetric tools are good only when you can respect a strict discipline, when you switch from one machine to another. On the contrary, tools like CVS are more forgetful.

To synchronize two or more machines (typically a desktop and a laptop), just choose a CVS repository somewhere on the network. It can be on one of the machines you want to synchronize or on a third host. Anyway, this machine should be easily reachable via the network and have good disks.

Then, cvs co the module you want to work on, edit it, and cvs commit when you reached a synch point and are connected. If you made changes on both hosts, CVS will try to merge them (it typically succeeds automatically) or give in and ask you to resolve it by hand.

The typical limits of this solution: CVS does not deal well with binary files, so this solution is more for users of vi or emacs than for GIMP fans. CVS has trouble with some UNIX goodies like symbolic links.

For more information on CVS, see the Web page . The CVS documentation is excellent (in info format).

CODA Filesystem

The CODA File System is a descendant of the Andrew File System. Like AFS, Coda offers location-transparent access to a shared UNIX file name-space that is mapped on to a collection of dedicated file servers. But Coda represents a substantial improvement over AFS because it offers considerably higher availability in the face of server and network failures. The improvement in availability is achieved using the complementary techniques of server replication and disconnected operation. Disconnected operation proven especially valuable in supporting portable computers .

unison

unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other. Unison was written by researchers with an eye for well-defined replication semantics: they were very fussy about safety, and made sure to handle gracefully things like premature termination etc. Unison is symmetric/bidirectional (unlike rsync), works fine with binaries (unlike cvs), and is a user-level program (unlike most distributed filesystems). It also makes a reasonable attempt to synchronize transparently between Unix/Linux and Windows filesystems, which is no small feat. Drawbacks: it does not do version control, and does not handle synchronization among more than 2 file trees. unison shares a number of features with tools such as configuration management packages (CVS, PRCS, etc.) distributed filesystems ( CODA , etc.) uni-directional mirroring utilities (rsync, etc.) and other synchronizers ( Intellisync, Reconcile, etc). However, there are a number of points where it differs:

  • unison runs on both MicroSoft-Windows (95, 98, NT, and 2k) and Unix (Solaris, Linux, etc.) systems ( for ARM based Linux PDAs see the TuxMobil IPK feed . Moreover, unison works across platforms, allowing you to synchronize a Microsoft-Windows laptop with a Unix server, for example.

  • Unlike a distributed filesystem, unison is a user-level program: there is no need to hack (or own!) the kernel, or to have superuser privileges on either host.

  • Unlike simple mirroring or backup utilities, unison can deal with updates to both replicas of a distributed directory structure. Updates that do not conflict are propagated automatically. Conflicting updates are detected and displayed.

  • unison works between any pair of machines connected to the internet, communicating over either a direct socket link or tunneling over an rsh or an encrypted ssh connection. It is careful with network bandwidth, and runs well over slow links such as PPP connections.

  • unison has a clear and precise specification.

  • unison is resilient to failure. It is careful to leave the replicas and its own private structures in a sensible state at all times, even in case of abnormal termination or communication failures.

  • unison is free; full source code is available under the GNU Public License.

OpenSync, MultiSync

OpenSync is the successor of KitchenSync and MultiSync. OpenSync is a synchronization framework that is platform and distribution independent. It consists of a powerful sync-engine and several plugins that can be used to connect to devices. OpenSync is very flexible and capable of synchronizing any type of data, including contacts, calendar, tasks, notes and files.

MultiSync is a free modular program to synchronize calendars, address books, and other PIM data between programs on your computer and other computers, mobile devices, PDAs or cell phones. Currently MultiSync has plugins for Ximian Evolution calendars and IrMC Mobile Client calendars (supported by the Sony/Ericsson T68i) via Bluetooth, IrDA, or a cable connection.

Funambol

Funambol is an open source mobile application server software that provides push email, address book and calendar (PIM) data synchronization, application provisioning, and device management for wireless devices and PCs, leveraging standard protocols. For users, this means BlackBerry-like capabilities on commodity handsets. Funambol is also a software development platform for mobile applications. It provides client and server side Java APIs, and facilitates the development, deployment and management of any mobile project. Funambol is the de facto standard implementation of the Open Mobile Alliance Data Synchronization and Device Management protocols (OMA DS and DM, formerly known as SyncML). Funambol is replaces the former sync4J tools.

Tsync

Tsync (Transparent) Synchronization is a user-level daemon that provides transparent synchronization amongst a set of computers. Tsync uses a peer-to-peer architecture for scalability, efficiency, and robustness.

InterMezzo

InterMezzo is a new distributed file system with a focus on high availability. InterMezzo is an Open Source project, currently on Linux (2.2 and 2.3). A primary target of development is to provide support for flexible replication of directories, with disconnected operation and a persistent cache. For example, we want to make it easy to manage copies of home directories on multiple computers, and solve the laptop/desktop synchronization problems. On a larger scale we aim to provide replication of large file repositories, for example to support high availability for servers. InterMezzo was deeply inspired by the Coda File System, but totally re-designed and re-engineered.

WWWsync

WWWsync/ is a program written in Perl that will update your web pages by ftp from your local pages. This was originally written for updating Demon home-pages, but will work with other providers which provide direct FTP access to your web pages. I didn't check this for laptop purposes yet.

rsync

rsync is a program that allows files to be copied to and from remote machines in much the same way as rcp. It has many more options than rcp, and uses the rsync remote-update protocol to greatly speedup file transfers when the destination file already exists. The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network link.

Xfiles - file tree synchronization and cross-validation

Xfiles is an interactive utility for comparing and merging one file tree with another over a network. It supports freeform work on several machines (no need to keep track of what files are changed on which machine). Xfiles can also be used as a cross-validating disk <-> disk backup strategy (portions of a disk may go bad at any time, with no simple indication of which files were affected. Cross-validate against a second disk before backup to make sure you aren't backing up bad data).

A client/server program (GUI on the client) traverses a file tree and reports any files that are missing on the server machine, missing on the client machine, or different. For each such file, the file size/sizes and modification date(s) are shown, and a comparison (using UNIX diff) can be obtained. For files that are missing from one tree, similarly named files in that tree are reported. Inconsistent files can then be copied in either direction or deleted on either machine. The file trees do not need to be accessible via nfs. Files checksums are computed in parallel, so largely similar trees can be compared over a slow network link. The client and server processes can also be run on the same machine. File selection and interaction with a revision control system such as RCS can be handled by scripting using jpython. Requirements Java1.1 or later and JFC/Swing1.1 are needed. Xfiles.

sitecopy

Sitecopy is for copying locally stored websites to remote web servers. The program will upload files to the server which have changed locally, and delete files from the server which have been removed locally, to keep the remote site synchronized with the local site, with a single command. The aim is to remove the hassle of uploading and deleting individual files using an FTP client. sitecopy.

DataConversion: AddressBooks, BookMarks, Todo-Lists, LDAP, Webpages

Transfering user data from one mobile device to another one, often requires some tools to extract the data from the source device before importing them into the target device, for example if you want to change your favorite mobile phone. Or if you want to use the addressbook from your mobile with your PDA, too. Here are some tools for bookmark conversion, addressbook migration, vCard extraction, LDAP merging and data conversion for PDAs and HandHeld PCs.