Directory synchronization utility

Release: 0.31

→ preface → usage → download → history further information → contact

Preface

This utility is designed for the synchronization of directory trees on different machines. It is especially useful if these machines do not have any network connection and removables have to be used. It will synchronize in both directions using incremental updates to conserve space and time.

There is a REXX version and a Perl version of this utility available, whichever you prefer. They perform (mainly) identical. The Perl script has the advantage that it can run under other operating systems like WinNT, too. However, there are some constraints if you want to synchronize cross platform trees.

Usage

Requirements

REXX (built-in to OS/2) or
Perl version 5.6 or higher.

Installation

Copy the script somewhere to your path.
In case of the Perl version you may want to create a CMD file instead, calling Perl with the path to the script as argument:
perl -S dsync.pl %1 %2 %3 %4 %5 %6 %7 %8 %9

Parameters

dsync dirspec1 [dirspec2] [options] [@optionfile]

dirspec1, dirspec2:

Path to the the first and second directory to compare. The path may be absolute, relative, UNC (\\server\share\folder\) or an FTP(S) server (ftp://user:passwd@server:port/folder/, ftps://user:passwd@server:port/folder/). Note that the FTP protocol is restricted to the Perl version only, since rxftp.dll does not support the required size and mdtm commands. dirspec may also contain a file mask (e.g. *.c). By default a file mask of * is assumed. If the second path specifier is dropped, the current directory (.\) is used.

options and dirspecs may be placed in an option file. This file must contain one argument per line. It may call another option file, but ensure not to create a recursion.

If no options are given the directories are only compared. The compare result is written to stdout; one line for each different file.
The first two characters of each line signaling the comparison result followed by a space character and the file name of the relating file. The file name includes the relative path counting from the directories given at the command line.

*O

File in directory 1 is newer.

O*

File in directory 2 is newer.

*-

File not found in directory 2.

-*

File not found in directory 1.

>>

File has the same time stamp, but in directory 1 it is longer.

<<

File has the same time stamp, but in directory 2 it is longer.

<>

Files do not compare equal.

EE

Error during file compare.

FF

Error during file synchronization.

Options

Command line
Description

-s Include sub directories.
If a file mask is used it applies to all files in all sub directories but not to the directory names.

-f Compare file contents, too.
This will use XCOMP if two files do not differ in any other aspect. XCOMP is an replacement for COMP and must reside in your path environment.
You cannot use -f together with file synchronization.

-e REXX only! - Synchronize (empty) directories, too.
dsync -e will include folders into synchronization. This has two effects:
- first: empty folders will be synchronized too,
- secondly: the extended attributes of the folder will be synchronized. This works only with the REXX version, since there is not portable way so far to access extended attributes from Perl.

-v Write compare results as text messages instead of the default format.

-q Be quiet.

-y Synchronize directories automatically.

-y1 Synchronize only the first directory.
For files which are newer in the first tree a warning is reported.

-y2 Synchronize only the second directory.
For files which are newer in the second tree a warning is reported.

-fy Synchronize files and directories even if the direction is ambiguous.
When two files or directories of the same name are different only in size (or content) the file content or directory attributes are copied anyway. This works only with /y1 or /y2.

-i Ignore downgrades.
If -y1 or -y2 is specified (but not -y) the case when the destination file is newer than the source file generates a warning. This may be unintensional, especially if the source or destination is an FTP server, since FTP transfers generally do not preserve the file's modification times.

-u[file] Use update file instead of the content of the first directory tree.
This is designed for remote updates using temporary storage. If the file name is omitted, the file dirspec1\UPDATE.DIR is used. -u is currently unsupported on FTP servers.

-d Delete files which are not found in the other directory when using an unidirectional synchronization mode (-y1 or -y2).
Be very careful with this option, because it may result in massive data destruction if one of the source directories is wrong. In this case it is most likely that all of the files are not found in the other directory.

-m Move files from the update directory instead of copying.
This is affects only the copy operations from the update directory to the first tree. In fact this will be ignored unless you used the option -u.

-zi or -z0 Create initial update file. (Shortcut for -s -q -u)
This does nothing but writing the directory content of dirspec1 to the update file. The second tree is ignored in this case.

-zp or -z1 Pack files for remote tree using the update file as directory content of the second (remote) tree. (Shortcut for -s -q -u -y2)
All files which are newer in the first directory tree are copied to the second tree, which is a temporary location in this case. Files which are older in the first tree cannot be synchronized this way and a warning is shown. Finally the directory content of dirspec1 is written to the update file as -zi does. In fact this is an incremental update.

-zu or -z2 Unpack update packet previously packed with -zp on another machine. (Shortcut for -s -q -u -y1)
In contrast to simply copying the packed files over the content of the local directory, the timestamps are reconsidered to avoid accidentally downgrading and the removal of files can be handled, too, using the -d option.

-za or -z3 Synchronize the remote update tree fully automatically. (Shortcut for -s -q -u -y -m)
This is an automatic replacement for -zu and -zp.^[1]
It will automatically detect the action to be taken for each file. Files which are newer in the update tree are moved to the first tree. Files which are newer in the first tree are copied to the update directory. After all the update file is renewed.

-cs Enable case sensitive file names. By default, the all operations are case-insensitive but case-preserving. - Perl version only!

-tthreshold Consider timestamps as equal even when they differ by less than or equal threshold seconds.
This is useful for different reasons: firstly, some file systems use a quantization coarser than one second, e.g. FAT: 2 seconds. Secondly, The time zones are treated different by different operating systems and compression utilities. OS/2 normally ignores the time zone and handle the time stamps as local time. Unix stores them as GMT. This can make cross platform synchronization very complicated, especially with daylight saving. However, if you are sure, that the time stamps cannot differ less than the time zone shift reasonably, than you may specify this amount as threshold.

-t Use timezone correction. - REXX version only!
This shifts all time stamps to GMT according to the setting of the environment variable TZ. This affects also the time stamps in the update file (see -u). You must use this option always or never to keep the update file compatible.
Perl always report the time stamps as GMT. So you need this option to be compatible with the Perl version. See Cross platform considerations

-tdelta Consider time stamps which differ by delta or less seconds as equal.
This is helpful if the underlying file systems of tree 1 and tree 2 use a different precision for time stamps (e.g. FAT vs. any other).

-l Use PMShell for copy and move operations and .LONGNAME EAs. - REXX version only!
This is useful if at least one of the directory trees resides on a FAT volume. You must use this option always or never if you perform remote updates with -u or -z. Otherwise the file names in the index file may not be compatible.

-_ Debug mode (verbose and read-only).
This is useful if you have problems with dsync or if you want to test special actions.

Notes:

There are some considerations to take into account when using -za.
First, -d cannot be used to track the deletion of files also, because adding a file to one tree is ambiguous to deleting it at the other tree. In doubt the first case is assumed. However, you will get error messages about missing files in the second case.
Secondly, if a file in the update tree is not needed anymore, it will remain there until it is overwritten randomly or deleted manually. This is most likely to happen if the file is modified on both sides, which is of course always a potential trouble through tree synchronizations.
Thirdly, the automatic update may be executed twice (or even more) from one site. This is not that trivial, because the update file already contains the index of the current tree instead of that of the remote tree. But since the other files in the update tree remain there and the synchronization now will copy only files which are modified since the last call, the result is nevertheless a complete set of files for the remote tree.

Examples

dsync tree1 -s -f

Compare tree1\* and .\* including sub directories and file contents. The differences are printed on the screen.

dsync tree1 tree2 -s -y

Synchronize two directory trees. In fact this will copy all files to the other tree which are newer or do not exist in the other tree. After that the trees should be identical. (Except if two files differ only by the length or by the content).

dsync upddir -za

Synchronize the current directory (including subdirs) with an offline remote tree using upddir for temporary storage. upddir may be on a removable media or may be zipped manually.

dsync upddir -zp -t3610

Create update packet for remote directory tree in upddir. The index file of the remote tree (upddir\UPDATE.DIR) must exist. The comparison with the remote tree index is done fuzzy. Only time stamps that differ by more than 1 hour and 10 seconds are treated unequal.

dsync upddir -zu -d

Merge previously created update packet in upddir with .\. Files that does not exist in the index file of the remote tree are removed in the local directory.

Download

This program is distributed under the terms of the GNU GENERAL PUBLIC LICENSE.

ZIP file with the REXX and Perl version and this guide: Release 0.21

Attention: Please keep in mind that this utility is still under construction. It may cause massive damage to your data. Make sure to have a backup of your data.

History

Version 0.31 [Jul. 4th, 2014]

Added FTPS support.

Version 0.3 [Feb. 2, 2008]

Added ignore downgrade option.
Fixed bugs with folder synchronization on FTP.

Version 0.21 [Jan. 21, 2005]

Added synchronization of folders.
Fixed bug with empty source trees.
Fixed bug: used root directory if one of the filespecs did not exist.

Version 0.2 [Aug. 1, 2004]

limited FTP support in the Perl version

Version 0.13 [12.03.2003]

-cs (case-sensitive) option for Unix file systems.
The Tool is now case-preserving.
Bug fixes in command line parsing.
Fixed hangs after successful synchronization.

Version 0.12 [24.02.2003]

Improved compatibility. Switched from platform dependent shell commands to File::* Perl modules.
Some minor fixes.

Version 0.11 [20.11.2001]

Fully automatic remote synchronization.
Some bug fixes.

Version 0.1 (first release)

Further information

Cross platform considerations

Time zones

This utility exists as REXX and as Perl implementation. So you can use it on different platforms. Unfortunately they are not 100% compatible.
Perl (or more exactly fstat) reports time stamps as GMT whereas REXX uses localtime. In fact the Unix file systems store the time stamps as GMT whereas OS/2 (and DOS, WinXX) does not. Because of this difference the OS/2 and WinXX Perl implementations convert the FS time stamps from local time to GMT. To keep the REXX script compatible with this I added the -t switch. I forces to convert all time stamps from local time to GMT, according to the TZ environment variable, before processing. Unfortunately this conversion may be ambiguous when the current timezone uses daylight saving. This may result in deviations of usually one hour during conversion.
However, there is another issue with OS/2 Perl: I never got Perl's locale working properly within OS/2. So I recommend using the REXX version with OS/2, with the -t switch if required.
If you are really sure that all modified files differ at least by e.g. 2 hours, you may suppress the synchronization of files with less than 2 hours difference by -t7200. This may be helpful with time zones close to GMT.

Case sensitivity

DOS like platforms (WinXX, OS/2) usually have case insensitive file systems. Unfortunately they are not always case preserving.
Special care should be taken when synchronizing between file systems with different case-sensitivity. In general the default, case-insensitive mode should be used. But wired things may happen if the filenames are not unique in this context.

Resolution of time stamps

Some file systems cannot represent the time stamps on per second basis (e.g. FAT*). In this case two trees may never compare equal, if the other file system is capable of higher precision. The option -t3 will avoid this.

Contact

Suggestions, help, complaints (but not too much:-):

Original homepage: http://www.maazl.de/project/misc/dsync.html

Command line	Description
`-s`	Include sub directories. If a file mask is used it applies to all files in all sub directories but not to the directory names.
`-f`	Compare file contents, too. This will use `XCOMP` if two files do not differ in any other aspect. `XCOMP` is an replacement for `COMP` and must reside in your path environment. You cannot use `-f` together with file synchronization.
`-e`	REXX only! - Synchronize (empty) directories, too. `dsync -e` will include folders into synchronization. This has two effects: - first: empty folders will be synchronized too, - secondly: the extended attributes of the folder will be synchronized. This works only with the REXX version, since there is not portable way so far to access extended attributes from Perl.
`-v`	Write compare results as text messages instead of the default format.
`-q`	Be quiet.
`-y`	Synchronize directories automatically.
`-y1`	Synchronize only the first directory. For files which are newer in the first tree a warning is reported.
`-y2`	Synchronize only the second directory. For files which are newer in the second tree a warning is reported.
`-fy`	Synchronize files and directories even if the direction is ambiguous. When two files or directories of the same name are different only in size (or content) the file content or directory attributes are copied anyway. This works only with `/y1` or `/y2`.
`-i`	Ignore downgrades. If `-y1` or `-y2` is specified (but not `-y`) the case when the destination file is newer than the source file generates a warning. This may be unintensional, especially if the source or destination is an FTP server, since FTP transfers generally do not preserve the file's modification times.
`-u`[`file`]	Use update file instead of the content of the first directory tree. This is designed for remote updates using temporary storage. If the file name is omitted, the file `dirspec1\UPDATE.DIR` is used. -u is currently unsupported on FTP servers.
`-d`	Delete files which are not found in the other directory when using an unidirectional synchronization mode (`-y1` or `-y2`). Be very careful with this option, because it may result in massive data destruction if one of the source directories is wrong. In this case it is most likely that all of the files are not found in the other directory.
`-m`	Move files from the update directory instead of copying. This is affects only the copy operations from the update directory to the first tree. In fact this will be ignored unless you used the option `-u`.
`-zi` or `-z0`	Create initial update file. (Shortcut for `-s -q -u`) This does nothing but writing the directory content of `dirspec1` to the update file. The second tree is ignored in this case.
`-zp` or `-z1`	Pack files for remote tree using the update file as directory content of the second (remote) tree. (Shortcut for `-s -q -u -y2`) All files which are newer in the first directory tree are copied to the second tree, which is a temporary location in this case. Files which are older in the first tree cannot be synchronized this way and a warning is shown. Finally the directory content of `dirspec1` is written to the update file as `-zi` does. In fact this is an incremental update.
`-zu` or `-z2`	Unpack update packet previously packed with `-zp` on another machine. (Shortcut for `-s -q -u -y1`) In contrast to simply copying the packed files over the content of the local directory, the timestamps are reconsidered to avoid accidentally downgrading and the removal of files can be handled, too, using the `-d` option.
`-za` or `-z3`	Synchronize the remote update tree fully automatically. (Shortcut for `-s -q -u -y -m`) This is an automatic replacement for `-zu` and `-zp`.^[1] It will automatically detect the action to be taken for each file. Files which are newer in the update tree are moved to the first tree. Files which are newer in the first tree are copied to the update directory. After all the update file is renewed.
`-cs`	Enable case sensitive file names. By default, the all operations are case-insensitive but case-preserving. - Perl version only!
`-tthreshold`	Consider timestamps as equal even when they differ by less than or equal `threshold` seconds. This is useful for different reasons: firstly, some file systems use a quantization coarser than one second, e.g. FAT: 2 seconds. Secondly, The time zones are treated different by different operating systems and compression utilities. OS/2 normally ignores the time zone and handle the time stamps as local time. Unix stores them as GMT. This can make cross platform synchronization very complicated, especially with daylight saving. However, if you are sure, that the time stamps cannot differ less than the time zone shift reasonably, than you may specify this amount as threshold.
`-t`	Use timezone correction. - REXX version only! This shifts all time stamps to GMT according to the setting of the environment variable `TZ`. This affects also the time stamps in the update file (see `-u`). You must use this option always or never to keep the update file compatible. Perl always report the time stamps as GMT. So you need this option to be compatible with the Perl version. See Cross platform considerations
`-tdelta`	Consider time stamps which differ by `delta` or less seconds as equal. This is helpful if the underlying file systems of tree 1 and tree 2 use a different precision for time stamps (e.g. FAT vs. any other).
`-l`	Use PMShell for copy and move operations and `.LONGNAME` EAs. - REXX version only! This is useful if at least one of the directory trees resides on a FAT volume. You must use this option always or never if you perform remote updates with `-u` or `-z`. Otherwise the file names in the index file may not be compatible.
`-_`	Debug mode (verbose and read-only). This is useful if you have problems with `dsync` or if you want to test special actions.