Maltego Importer GUI

Recently I’ve received some requests about the Maltego importer I wrote a year back. That motivated me to write a simple GUI client for the importer, as building and running the project isn’t a trivial manner. I’ve also added a new CSV format where the node types are defined in the first row:

GangMember, Male, Female, LawOfficer
Jon Doe,Some random guy,Some lady,Policeman

You need Java and maven to build the source, but you can also download an executable from the project page. After you’ve downloaded maltego-importer.jar, double click on it and you’re supposed to see a window. With the new format you’ll have to define the relations among the columns manually, but the mappings will be remembered! Using the old example, the client window can look something like this:

Now click “Convert to Maltego”. It generates a Maltego .mtgx-file which corresponds to the following graph:

The importer tries to guess the right node type based on the column header using min edit distance, however, you can change the type when it misses. Mappings are remembered for later. Please let me know if you find any bugs or have any feature requests!

A simple world heat map using HTML and JavaScript

This is a simple geographic heat map using HTML and JavaScript. It is a mashup of http://raphaeljs.com/world/ and http://www.designchemical.com/blog/index.php/jquery/jquery-tutorial-create-a-flexible-data-heat-map/.

Get the source at github.

 


Doing face recognition with JavaCV

I couldn’t find any tutorial on how to perform face recognition using OpenCV and Java, so I decided to share a viable solution here. The solution is very inefficient in its current form as the training model is built at each run, however it shows what’s needed to make it work.

The class below takes two arguments: The path to the directory containing the training faces and the path to the image you want to classify. Not that all images has to be of the same size and that the faces already has to be cropped out of their original images (Take a look here if you haven’t done the face detection yet).

For the simplicity of this post, the class also requires that the training images have filename format: <label>-rest_of_filename.png. For example:

1-jon_doe_1.png
1-jon_doe_2.png
2-jane_doe_1.png
2-jane_doe_2.png
…and so on.

The code:

import com.googlecode.javacv.cpp.opencv_core;
import static com.googlecode.javacv.cpp.opencv_highgui.*;
import static com.googlecode.javacv.cpp.opencv_core.*;
import static com.googlecode.javacv.cpp.opencv_imgproc.*;
import static com.googlecode.javacv.cpp.opencv_contrib.*;
import java.io.File;
import java.io.FilenameFilter;

public class OpenCVFaceRecognizer {
    public static void main(String[] args) {
        String trainingDir = args[0];
        IplImage testImage = cvLoadImage(args[1]);

        File root = new File(trainingDir);

        FilenameFilter pngFilter = new FilenameFilter() {
            public boolean accept(File dir, String name) {
                return name.toLowerCase().endsWith(".png");
            }
        };

        File[] imageFiles = root.listFiles(pngFilter);

        MatVector images = new MatVector(imageFiles.length);

        int[] labels = new int[imageFiles.length];

        int counter = 0;
        int label;

        IplImage img;
        IplImage grayImg;

        for (File image : imageFiles) {
            img = cvLoadImage(image.getAbsolutePath());

            label = Integer.parseInt(image.getName().split("\\-")[0]);

            grayImg = IplImage.create(img.width(), img.height(), IPL_DEPTH_8U, 1);

            cvCvtColor(img, grayImg, CV_BGR2GRAY);

            images.put(counter, grayImg);

            labels[counter] = label;

            counter++;
        }

        IplImage greyTestImage = IplImage.create(testImage.width(), testImage.height(), IPL_DEPTH_8U, 1);

        FaceRecognizer faceRecognizer = createFisherFaceRecognizer();
        // FaceRecognizer faceRecognizer = createEigenFaceRecognizer();
        // FaceRecognizer faceRecognizer = createLBPHFaceRecognizer()

        faceRecognizer.train(images, labels);

        cvCvtColor(testImage, greyTestImage, CV_BGR2GRAY);

        int predictedLabel = faceRecognizer.predict(greyTestImage);

        System.out.println("Predicted label: " + predictedLabel);
    }
}

The class requires the OpenCV Java interface. If you’re using Maven, you can retrieve the required libraries with the following pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.pcbje</groupId>
    <artifactId>opencvfacerecognizer</artifactId>
    <version>0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>opencvfacerecognizer</name>
    <url>http://pcbje.com</url>

    <dependencies>
        <dependency>
            <groupId>com.googlecode.javacv</groupId>
            <artifactId>javacv</artifactId>
            <version>0.3</version>
        </dependency>

        <!-- For Linux x64 environments -->
        <dependency>
            <groupId>com.googlecode.javacv</groupId>
            <artifactId>javacv</artifactId>
            <classifier>linux-x86_64</classifier>
            <version>0.3</version>
        </dependency>        

        <!-- For OSX environments -->
        <dependency>
            <groupId>com.googlecode.javacv</groupId>
            <artifactId>javacv</artifactId>
            <classifier>macosx-x86_64</classifier>
            <version>0.3</version>
        </dependency>
    </dependencies>

    <repositories>
        <repository>
            <id>javacv</id>
            <name>JavaCV</name>
            <url>http://maven2.javacv.googlecode.com/git/</url>
        </repository>
    </repositories>
</project>

 

Creating a Hadoop MapFile from a disk image using Sleuthkit 4

There are two advantages with using the MapFile data structure on Apache Hadoop and HDFS when working with files like Word documents, PDF-files, etc.: First, you may efficiently search and retrieve the contents of these files by searching on their index. Second, you can easily implement MapReduce jobs where each mapping receives the file index and its content.

The code below reads the SQLite database generated by Sleuthkit 4.0.0 and places the extracted files on HDFS as a MapFile.

import java.io.IOException;
import java.util.List;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.MapFile;
import org.apache.solr.client.solrj.SolrServerException;
import org.sleuthkit.datamodel.Content;
import org.sleuthkit.datamodel.ContentVisitor;
import org.sleuthkit.datamodel.File;
import org.sleuthkit.datamodel.SleuthkitCase;
import org.sleuthkit.datamodel.TskCoreException;

public class SleuthkitToMapFile {

    private void process(ExampleVisitor visitor,
            MapFile.Writer hdfsWriter, List lc)
            throws IOException, SolrServerException, TskCoreException {

        for (Content c : lc) {
            boolean append = c.accept(visitor);

            if (append) {
                byte[] content = readContent(c);

                hdfsWriter.append(new LongWritable(c.getId()),
                        new BytesWritable(content));
            }

            process(visitor, hdfsWriter, c.getChildren());
        }
    }

    private byte[] readContent(Content c, int bufferSize) throws IOException {
        ByteArrayOutputStream baos = new ByteArrayOutputStream(bufferSize);

        byte[] readBuffer = new byte[bufferSize];

        for (long pos = 0; pos < c.getSize(); pos = pos + bufferSize) {
            int remaining = (int) Math.min(c.getSize() - pos, bufferSize);

            try {
                int read = c.read(readBuffer, pos, remaining);

                baos.write(readBuffer, 0, read);

            } catch (Exception ex) {
                logger.log(Level.WARN, null, ex);
                break;
            }
        }

        baos.close();

        return baos.toByteArray();
    }

    private static class ExampleVisitor extends ContentVisitor.Default {

        @Override
        public Boolean visit(File f) {
            return true;
        }

        @Override
        protected Boolean defaultVisit(Content c) {
            return false;
        }
    }

    public static void main(String[] args) throws Exception {
        SleuthkitToMapFile pfs = new SleuthkitToMapFile();

        Configuration hadoopConfig = new Configuration();

        hadoopConfig.set("fs.default.name", "hdfs://localhost:9000");

        FileSystem fs = FileSystem.get(hadoopConfig);

        MapFile.Writer hdfsWriter = new MapFile.Writer(hadoopConfig, fs,
                "/target/path/on/hdfs", LongWritable.class, BytesWritable.class);

        SleuthkitCase sk = SleuthkitCase.openCase("/tmp/sleuthkit.db");

        ExampleVisitor visitor = new ExampleVisitor();

        pfs.process(visitor, hdfsWriter, sk.getRootObjects());

        hdfsWriter.close();
    }
}

The code is inspired by the Java binding test classes at the Sleuthkit project page at github.

To run the Java program above, you need to have The Sleuth Kit 4.0.0 and its Java bindings installed. Here's a description of how you can do this on a UNIX system (I've only tested this on my OSX with NetBeans 7.2, but this should work on Linux as well):

-- Download TSK4 from: http://www.sleuthkit.org/sleuthkit/download.php
$ tar -xvf sleuthkit-4.0.0.tar.gz
$ cd sleuthkit-4.0.0
$ ./configure
$ make
$ make install
$ cd bindings/java
$ ant

The jar-file under bindings/java/dist/ has to be added to the Java program's classpath. You'll also need to point the java.library.path property to the TSK_JNI library (Typically "-Djava.library.path=/usr/local/lib").

As previously stated does the code above assume that there exist a Sleuthkit database containing disk image metadata at /tmp/sleuthkit.db. This database can be created using Autopsy 3.0 (The SQLite file is then stored in some other directory), or you can create it by using the Java bindings:

import org.sleuthkit.datamodel.SleuthkitCase;
import org.sleuthkit.datamodel.SleuthkitJNI;
import org.sleuthkit.datamodel.TskCoreException;
import org.sleuthkit.datamodel.TskDataException;

public class SleuthkitJNIRip {
    public static void main(String[] args) throws TskCoreException, TskDataException {
        SleuthkitCase sk = SleuthkitCase.newCase("/tmp/sleuthkit.db");

        SleuthkitJNI.CaseDbHandle.AddImageProcess process =
                sk.makeAddImageProcess("", false, true);

        process.run(new String[]{"/path/to/image"});

        process.commit();
    }
}

Now you can do all sorts of exiting things with your data.