NAME
Algorithm::Diff::Fast - Efficiently compute differences between files / lists
SYNOPSIS
use Algorithm::Diff::Fast qw(
LCS LCS_length LCSidx
diff sdiff compact_diff
traverse_sequences traverse_balanced );
@lcs = LCS( \@seq1, \@seq2 );
$lcsref = LCS( \@seq1, \@seq2 );
$count = LCS_length( \@seq1, \@seq2 );
( $seq1idxref, $seq2idxref ) = LCSidx( \@seq1, \@seq2 );
# Complicated interfaces:
@diffs = diff( \@seq1, \@seq2 );
@sdiffs = sdiff( \@seq1, \@seq2 );
@cdiffs = compact_diff( \@seq1, \@seq2 );
traverse_sequences(
\@seq1,
\@seq2,
{ MATCH => \&callback1,
DISCARD_A => \&callback2,
DISCARD_B => \&callback3,
},
\&key_generator,
@extra_args,
);
traverse_balanced(
\@seq1,
\@seq2,
{ MATCH => \&callback1,
DISCARD_A => \&callback2,
DISCARD_B => \&callback3,
CHANGE => \&callback4,
},
\&key_generator,
@extra_args,
);
# Object oriented interface
my $diff = Algorithm::Diff->new( \@seq1, \@seq2 );
$diff->Base( 1 );
while( $diff->Next() ) {
next if $diff->Same();
}DESCRIPTION
This module is a drop-in replacement for Algorithm::Diff, but uses C code internally to make the computations more efficient. It also uses an updated algorithm, essentially the same algorithm used by GNU's diff command-line utility. This is far more efficient, especially for large data sets.
The underlying C code came from libmba (see: http://www.ioplex.com/~miallen/libmba/), although this module only includes the parts of this library needed to support the diff implementation.
There is already a C-based implementation in Algorithm::Diff::XS, itself drawn from Algorithm::Diff::LCS. This also cannot handle large data sets, due to the algorithm's requirements for large amounts of memory in these cases, and it only uses C to enhance parts of the Algorithm::Diff API. By contrast, this module uses the underlying C code across the whole of the Algorithm::Diff API.
DIFFERENCES FROM Algorithm::Diff
The principal difference is internal, and should generally not matter to module users. This module uses a different algorithm - essentially the same algorithm used by GNU diff.
All the tests for Algorithm::Diff have been included and are applied, in a slightly updated form.
The prepare function is defined, but has no significant performance advantage unless you have a particularly slow key generation function.
OBJECT INTERFACE
Algorithm::Diff::Fast has the same object interface as Algorithm::Diff, and indeed the code is drawn from Algorithm::Diff. It uses the underlying diff engine as Algorithm::Diff::Fast, so the performance is greater.
FUNCTIONS
The functions compare elements using string eq. Where a function in Algorithm::Diff permits and uses a key generation function, the same function also allows a key generation function in Algorithm::Diff::Fast.
LCS
Given references to two lists of items, LCS returns an array containing their longest common subsequence. In scalar context, it returns a reference to such a list.
@lcs = LCS( \@seq1, \@seq2 );
$lcsref = LCS( \@seq1, \@seq2 );LCS may be passed an optional third parameter; this is a CODE reference to a key generation function. See "KEY GENERATION FUNCTIONS".
@lcs = LCS( \@seq1, \@seq2, \&keyGen, @args );
$lcsref = LCS( \@seq1, \@seq2, \&keyGen, @args );Additional parameters, if any, will be passed to the key generation routine.
LCSidx
Like LCS except it returns references to two arrays. The first array contains the indices into @seq1 where the LCS items are located. The second array contains the indices into @seq2 where the LCS items are located.
Therefore, the following three lists will contain the same values:
my( $idx1, $idx2 ) = LCSidx( \@seq1, \@seq2 );
my @list1 = @seq1[ @$idx1 ];
my @list2 = @seq2[ @$idx2 ];
my @list3 = LCS( \@seq1, \@seq2 );LCS_length
This is just like LCS except it only returns the length of the longest common subsequence. This may provide a small performance gain compared to LCS.
compact_diff
compact_diff is much like sdiff except it returns a much more compact description consisting of just one flat list of indices. An example helps explain the format:
my @a = qw( a b c e h j l m n p );
my @b = qw( b c d e f j k l m r s t );
@cdiff = compact_diff( \@a, \@b );
# Returns:
# @a @b @a @b
# start start values values
( 0, 0, # =
0, 0, # a !
1, 0, # b c = b c
3, 2, # ! d
3, 3, # e = e
4, 4, # f ! h
5, 5, # j = j
6, 6, # ! k
6, 7, # l m = l m
8, 9, # n p ! r s t
10, 12, #
);The 0th, 2nd, 4th, etc. entries are all indices into @seq1 (@a in the above example) indicating where a hunk begins. The 1st, 3rd, 5th, etc. entries are all indices into @seq2 (@b in the above example) indicating where the same hunk begins.
So each pair of indices (except the last pair) describes where a hunk begins (in each sequence). Since each hunk must end at the item just before the item that starts the next hunk, the next pair of indices can be used to determine where the hunk ends.
So, the first 4 entries (0..3) describe the first hunk. Entries 0 and 1 describe where the first hunk begins (and so are always both 0). Entries 2 and 3 describe where the next hunk begins, so subtracting 1 from each tells us where the first hunk ends. That is, the first hunk contains items $diff[0] through $diff[2] - 1 of the first sequence and contains items $diff[1] through $diff[3] - 1 of the second sequence.
In other words, the first hunk consists of the following two lists of items:
# 1st pair 2nd pair
# of indices of indices
@list1 = @a[ $cdiff[0] .. $cdiff[2]-1 ];
@list2 = @b[ $cdiff[1] .. $cdiff[3]-1 ];
# Hunk start Hunk endNote that the hunks will always alternate between those that are part of the LCS (those that contain unchanged items) and those that contain changes. This means that all we need to be told is whether the first hunk is a 'same' or 'diff' hunk and we can determine which of the other hunks contain 'same' items or 'diff' items.
By convention, we always make the first hunk contain unchanged items. So the 1st, 3rd, 5th, etc. hunks (all odd-numbered hunks if you start counting from 1) all contain unchanged items. And the 2nd, 4th, 6th, etc. hunks (all even-numbered hunks if you start counting from 1) all contain changed items.
Since @a and @b don't begin with the same value, the first hunk in our example is empty (otherwise we'd violate the above convention). Note that the first 4 index values in our example are all zero. Plug these values into our previous code block and we get:
@hunk1a = @a[ 0 .. 0-1 ];
@hunk1b = @b[ 0 .. 0-1 ];And 0..-1 returns the empty list.
Move down one pair of indices (2..5) and we get the offset ranges for the second hunk, which contains changed items.
Since @diff[2..5] contains (0,0,1,0) in our example, the second hunk consists of these two lists of items:
@hunk2a = @a[ $cdiff[2] .. $cdiff[4]-1 ];
@hunk2b = @b[ $cdiff[3] .. $cdiff[5]-1 ];
# or
@hunk2a = @a[ 0 .. 1-1 ];
@hunk2b = @b[ 0 .. 0-1 ];
# or
@hunk2a = @a[ 0 .. 0 ];
@hunk2b = @b[ 0 .. -1 ];
# or
@hunk2a = ( 'a' );
@hunk2b = ( );That is, we would delete item 0 ('a') from @a.
Since @diff[4..7] contains (1,0,3,2) in our example, the third hunk consists of these two lists of items:
@hunk3a = @a[ $cdiff[4] .. $cdiff[6]-1 ];
@hunk3a = @b[ $cdiff[5] .. $cdiff[7]-1 ];
# or
@hunk3a = @a[ 1 .. 3-1 ];
@hunk3a = @b[ 0 .. 2-1 ];
# or
@hunk3a = @a[ 1 .. 2 ];
@hunk3a = @b[ 0 .. 1 ];
# or
@hunk3a = qw( b c );
@hunk3a = qw( b c );Note that this third hunk contains unchanged items as our convention demands.
You can continue this process until you reach the last two indices, which will always be the number of items in each sequence. This is required so that subtracting one from each will give you the indices to the last items in each sequence.
traverse_sequences
traverse_sequences used to be the most general facility provided by this module (the new OO interface is more powerful and much easier to use).
Imagine that there are two arrows. Arrow A points to an element of sequence A, and arrow B points to an element of the sequence B. Initially, the arrows point to the first elements of the respective sequences. traverse_sequences will advance the arrows through the sequences one element at a time, calling an appropriate user-specified callback function before each advance. It willadvance the arrows in such a way that if there are equal elements $A[$i] and $B[$j] which are equal and which are part of the LCS, there will be some moment during the execution of traverse_sequences when arrow A is pointing to $A[$i] and arrow B is pointing to $B[$j]. When this happens, traverse_sequences will call the MATCH callback function and then it will advance both arrows.
Otherwise, one of the arrows is pointing to an element of its sequence that is not part of the LCS. traverse_sequences will advance that arrow and will call the DISCARD_A or the DISCARD_B callback, depending on which arrow it advanced. If both arrows point to elements that are not part of the LCS, then traverse_sequences will advance one of them and call the appropriate callback, but it is not specified which it will call.
The arguments to traverse_sequences are the two sequences to traverse, and a hash which specifies the callback functions, like this:
traverse_sequences(
\@seq1, \@seq2,
{ MATCH => $callback_1,
DISCARD_A => $callback_2,
DISCARD_B => $callback_3,
}
);Callbacks for MATCH, DISCARD_A, and DISCARD_B are invoked with at least the indices of the two arrows as their arguments. They are not expected to return any values. If a callback is omitted from the table, it is not called.
Callbacks for A_FINISHED and B_FINISHED are invoked with at least the corresponding index in A or B.
If arrow A reaches the end of its sequence, before arrow B does, traverse_sequences will call the A_FINISHED callback when it advances arrow B, if there is such a function; if not it will call DISCARD_B instead. Similarly if arrow B finishes first. traverse_sequences returns when both arrows are at the ends of their respective sequences. It returns true on success and false on failure. At present there is no way to fail.
traverse_sequences may be passed an optional fourth parameter; this is a CODE reference to a key generation function. See "KEY GENERATION FUNCTIONS".
Additional parameters, if any, will be passed to the key generation function.
If you want to pass additional parameters to your callbacks, but don't need a custom key generation function, you can get the default by passing undef:
traverse_sequences(
\@seq1, \@seq2,
{ MATCH => $callback_1,
DISCARD_A => $callback_2,
DISCARD_B => $callback_3,
},
undef, # default key-gen
$myArgument1,
$myArgument2,
$myArgument3,
);traverse_sequences does not have a useful return value; you are expected to plug in the appropriate behavior with the callback functions.
traverse_balanced
traverse_balanced is an alternative to traverse_sequences. It uses a different algorithm to iterate through the entries in the computed LCS. Instead of sticking to one side and showing element changes as insertions and deletions only, it will jump back and forth between the two sequences and report changes occurring as deletions on one side followed immediatly by an insertion on the other side.
In addition to the DISCARD_A, DISCARD_B, and MATCH callbacks supported by traverse_sequences, traverse_balanced supports a CHANGE callback indicating that one element got replaced by another:
traverse_balanced(
\@seq1, \@seq2,
{ MATCH => $callback_1,
DISCARD_A => $callback_2,
DISCARD_B => $callback_3,
CHANGE => $callback_4,
}
);If no CHANGE callback is specified, traverse_balanced will map CHANGE events to DISCARD_A and DISCARD_B actions, therefore resulting in a similar behaviour as traverse_sequences with different order of events.
The sdiff function of this module is implemented as call to traverse_balanced.
traverse_balanced does not have a useful return value; you are expected to plug in the appropriate behavior with the callback functions.
TODO
Proper leak testing is needed. I have run the module repeatedly on large test sets, and there is no evidence of leaking, but I would feel happier if I'd really checked this with leak detecting tools.
AUTHOR
Stuart Watt, stuart@morungos.com.
LICENSE
Includes portions of libmba, copyright (c) 2004 Michael B. Allen, <mba2000@ioplex.com>, and used under the MIT license.
Perl module code copyright (c) 2010 Stuart Watt. Perl tests and documentation adapted from Algorithm::Diff contributors including Mark-Jason Dominus, Ned Konz, Mike Schilli, and Tye McQueen.
This program is free software; you can redistribute it and/or modify it under the MIT license.