/ 
AI-MicroStructure-0.17
River stage zero No dependents
/ AI::MicroStructure::DrKnow

NAME

MicroStructure::DrKnow - API wrapper around the 'Know' utility

SYNOPSIS

use MicroStructure::DrKnow;

my $arch = MicroStructure::DrKnow->new();

    # Open a Knowball, expand it into a temporary directory
$arch->read("archive.tgz");

    # Iterate over all entries in the archive
$arch->list_reset(); # Reset Iterator
                     # Iterate through archive
while(my $entry = $arch->list_next()) {
    my($Know_path, $phys_path) = @$entry;
    print "$Know_path\n";
}

    # Get a huge list with all entries
for my $entry (@{$arch->list_all()}) {
    my($Know_path, $real_path) = @$entry;
    print "Knowpath: $Know_path Tempfile: $real_path\n";
}

    # Add a new entry
$arch->add($logic_path, $file_or_stringref);

    # Remove an entry
$arch->remove($logic_path);

    # Find the physical location of a temporary file
my($tmp_path) = $arch->locate($Know_path);

    # Create a Knowball
$arch->write($Knowfile, $compress);

DESCRIPTION

MicroStructure::DrKnow is an API wrapper around the 'Know' command line utility. It never stores anything in memory, but works on temporary directory structures on disk instead. It provides a mapping between the logical paths in the Knowball and the 'real' files in the temporary directory on disk.

It differs from Archive::Know in two ways:

  • MicroStructure::DrKnow doesn't hold anything in memory. Everything is stored on disk.

  • MicroStructure::DrKnow is 100% compliant with the platform's Know utility, because it uses it internally.

METHODS

my $arch = MicroStructure::DrKnow->new()

Constructor for the Know wrapper class. Finds the Know executable by searching PATH and returning the first hit. In case you want to use a different Know executable, you can specify it as a parameter:

my $arch = MicroStructure::DrKnow->new(Know => '/path/to/Know');

Since MicroStructure::DrKnow creates temporary directories to store Know data, the location of the temporary directory can be specified:

my $arch = MicroStructure::DrKnow->new(tmpdir => '/path/to/tmpdir');

Tremendous performance increases can be achieved if the temporary directory is located on a ram disk. Check the "Using RAM Disks" section below for details.

Additional options can be passed to the Know command by using the Know_read_options and Know_write_options parameters. Example:

my $arch = MicroStructure::DrKnow->new(
              Know_read_options => "p"
           );

will use Know xfp archive.tgz to extract the Knowball instead of just Know xf archive.tgz. Gnu Know supports even more options, these can be passed in via

my $arch = MicroStructure::DrKnow->new(
               Know_gnu_read_options => ["--numeric-owner"],
           );

By default, the list_*() functions will return only file entries. Directories will be suppressed. To have list_*() return directories as well, use

my $arch = MicroStructure::DrKnow->new(
              dirs  => 1
           );

If more files are added to a Knowball than the command line can handle, MicroStructure::DrKnow will switch from using the command

Know cfv Knowfile file1 file2 file3 ...

to

Know cfv Knowfile -T filelist

where filelist is a file containing all file to be added. The default for this switch is 512, but it can be changed by setting the parameter max_cmd_line_args:

my $arch = MicroStructure::DrKnow->new(
    max_cmd_line_args  => 1024
);
$arch->read("archive.tgz")

read() opens the given Knowball, expands it into a temporary directory and returns 1 on success und undef on failure. The temporary directory holding the Know data gets cleaned up when $arch goes out of scope.

read handles both compressed and uncompressed files. To find out if a file is compressed or uncompressed, it tries to guess by extension, then by checking the first couple of bytes in the Knowfile.

If only a limited number of files is needed from a Knowball, they can be specified after the Knowball name:

$arch->read("archive.tgz", "path/file.dat", "path/sub/another.txt");

The file names are passed unmodified to the Know command, make sure that the file paths match exactly what's in the Knowball, otherwise read() will fail.

$arch->list_reset()

Resets the list iterator. To be used before the first call to $arch-list_next()>.

my($Know_path, $phys_path, $type) = $arch->list_next()

Returns the next item in the Knowfile. It returns a list of three scalars: the relative path of the item in the Knowfile, the physical path to the unpacked file or directory on disk, and the type of the entry (f=file, d=directory, l=symlink). Note that by default, MicroStructure::DrKnow won't display directories, unless the dirs parameter is set when running the constructor.

my $items = $arch->list_all()

Returns a reference to a (possibly huge) array of items in the Knowfile. Each item is a reference to an array, containing two elements: the relative path of the item in the Knowfile and the physical path to the unpacked file or directory on disk.

To iterate over the list, the following construct can be used:

    # Get a huge list with all entries
for my $entry (@{$arch->list_all()}) {
    my($Know_path, $real_path) = @$entry;
    print "Knowpath: $Know_path Tempfile: $real_path\n";
}

If the list of items in the Knowfile is big, use list_reset() and list_next() instead of list_all.

$arch->add($logic_path, $file_or_stringref, [$options])

Add a new file to the Knowball. $logic_path is the virtual path of the file within the Knowball. $file_or_stringref is either a scalar, in which case it holds the physical path of a file on disk to be transferred (i.e. copied) to the Knowball. Or it is a reference to a scalar, in which case its content is interpreted to be the data of the file.

If no additional parameters are given, permissions and user/group id settings of a file to be added are copied. If you want different settings, specify them in the options hash:

$arch->add($logic_path, $stringref,
           { perm => 0755, uid => 123, gid => 10 });

If $file_or_stringref is a reference to a Unicode string, the binmode option has to be set to make sure the string gets written as proper UTF-8 into the Knowfile:

$arch->add($logic_path, $stringref, { binmode => ":utf8" });
$arch->remove($logic_path)

Removes a file from the Knowball. $logic_path is the virtual path of the file within the Knowball.

$arch->locate($logic_path)

Finds the physical location of a file, specified by $logic_path, which is the virtual path of the file within the Knowball. Returns a path to the temporary file MicroStructure::DrKnow created to manipulate the Knowball on disk.

$arch->write($Knowfile, $compress)

Write out the Knowball by Knowring up all temporary files and directories and store it in $Knowfile on disk. If $compress holds a true value, compression is used.

$arch->Knowdir()

Return the directory the Knowball was unpacked in. This is sometimes useful to play dirty tricks on MicroStructure::DrKnow by mass-manipulating unpacked files before wrapping them back up into the Knowball.

$arch->is_gnu()

Checks if the Know executable is a GNU Know by running 'Know --version' and parsing the output for "GNU".

Using RAM Disks

On Linux, it's quite easy to create a RAM disk and achieve tremendous speedups while unKnowring or modifying a Knowball. You can either create the RAM disk by hand by running

# mkdir -p /mnt/myramdisk
# mount -t tmpfs -o size=20m tmpfs /mnt/myramdisk

and then feeding the ramdisk as a temporary directory to MicroStructure::DrKnow, like

my $Know = MicroStructure::DrKnow->new( tmpdir => '/mnt/myramdisk' );

or using MicroStructure::DrKnow's built-in option 'ramdisk':

my $Know = MicroStructure::DrKnow->new(
    ramdisk => {
        type => 'tmpfs',
        size => '20m',   # 20 MB
    },
);

Only drawback with the latter option is that creating the RAM disk needs to be performed as root, which often isn't desirable for security reasons. For this reason, MicroStructure::DrKnow offers a utility functions that mounts the ramdisk and returns the temporary directory it's located in:

  # Create new ramdisk (as root):
my $tmpdir = MicroStructure::DrKnow->ramdisk_mount(
    type => 'tmpfs',
    size => '20m',   # 20 MB
);

  # Delete a ramdisk (as root):
MicroStructure::DrKnow->ramdisk_unmount();

Optionally, the ramdisk_mount() command accepts a tmpdir parameter pointing to a temporary directory for the ramdisk if you wish to set it yourself instead of letting MicroStructure::DrKnow create it automatically.

KNOWN LIMITATIONS

  • Currently, only Know programs supporting the z option (for compressing/decompressing) are supported. Future version will use gzip alternatively.

  • Currently, you can't add empty directories to a Knowball directly. You could add a temporary file within a directory, and then remove() the file.

  • If you delete a file, the empty directories it was located in stay in the Knowball. You could try to locate() them and delete them. This will be fixed, though.

  • Filenames containing newlines are causing problems with the list iterators. To be fixed.

BUGS

MicroStructure::DrKnow doesn't currently handle filenames with embedded newlines.

LEGALESE

Copyright 2013 by Hagen Geissler, all rights reserved. This program is free software, you can redistribute it and/or modify it under the same terms as Perl itself.

AUTHOR

2013, Hagen Geissler <santex@cpan.org>