Introduction to the Morpheus DataFrame

The Morpheus library is designed to facilitate the development of high-performance analytical software involving large datasets for both offline and real-time analysis on the Java Virtual Machine (JVM). The library is written in Java 8 with extensive use of lambdas but is accessible to all JVM languages.

Motivation

At its core, Morpheus provides a versatile two-dimensional memory-efficient tabular data structure called a DataFrame, similar to that first popularized in R. While dynamically typed scientific computing languages like R, Python, and Matlab are great for doing research, they are not well-suited for large-scale production systems, as they become extremely difficult to maintain and dangerous to refactor. The Morpheus library attempts to retain the power and versatility of the DataFrame concept while providing a much more type safe and self-describing set of interfaces, which should make developing, maintaining, and scaling code complexity much easier.

Another advantage of the Morpheus library is that it's extremely good at scaling on multi-core processor architectures given the powerful threading capabilities of the Java Virtual Machine. Many operations on a Morpheus DataFrame can seamlessly be run in parallel by simply calling parallel() on the entity you wish to operate on, much like with Java 8 Streams. Internally, these parallel implementations are based on the fork and join framework, and near-linear improvements in performance are observed for certain types of operations as CPU cores are added.

Capabilities

A Morpheus DataFrame is a column store structure where each column is represented by a Morpheus array, of which there are many implementations, including dense, sparse, and memory-mapped versions. Morpheus arrays are optimized and, wherever possible, are backed by primitive native Java arrays (even for types such as LocalDate, LocalDateTime, etc.) as these are far more efficient from a storage, access, and garbage collection perspective. Memory-mapped Morpheus aArrays, while still experimental, allow very large DataFrames to be created using off-heap storage backed by files.

While the complete feature set of the Morpheus DataFrame is still evolving, there are already many powerful APIs to affect complex transformations and analytical operations with ease. There are standard functions to compute summary statistics, perform various types of Linear Regressions and apply Principal Component Analysis (PCA), just to mention just a few. The DataFrame is indexed in both the row and column dimension, allowing data to be efficiently sorted, sliced, grouped, and aggregated along either axis.

Morpheus at a Glance

Let's look at a simple example, a regression example, and a use case.

A Simple Example

Consider a dataset of motor vehicle characteristics accessible here. The code below loads this CSV data into a Morpheus DataFrame, filters the rows to only include those vehicles that have a power to weight ratio > 0.1 (where weight is converted into kilograms), then adds a column to record the relative efficiency between highway and city mileage (MPG), sorts the rows by this newly added column in descending order, and finally records this transformed result to a CSV file.

DataFrame.read().csv(options -> {
    options.setResource("http://zavtech.com/data/samples/cars93.csv");
    options.setExcludeColumnIndexes(0);
}).rows().select(row -> {
    double weightKG = row.getDouble("Weight") * 0.453592d;
    double horsepower = row.getDouble("Horsepower");
    return horsepower / weightKG > 0.1d;
}).cols().add("MPG(Highway/City)", Double.class, v -> {
    double cityMpg = v.row().getDouble("MPG.city");
    double highwayMpg = v.row().getDouble("MPG.highway");
    return highwayMpg / cityMpg;
}).rows().sort(false, "MPG(Highway/City)").write().csv(options -> {
    options.setFile("/Users/witdxav/cars93m.csv");
    options.setTitle("DataFrame");
});

This example demonstrates the functional nature of the Morpheus API, where many method return types are in fact a DataFrame and therefore allow this form of method chaining. In this example, the methods csv(), select(), add(), and sort() all return a frame. In some cases, the same frame that the method operates on; in other cases, it's a filter or shallow copy of the frame being operated on. The first 10 rows of the transformed dataset in this example look as follows, with the newly added column appearing on the far right of the frame.

A Regression Example

The Morpheus API includes a regression interface in order to fit data to a linear model using either OLS, WLS, or GLS. The code below uses the same car dataset introduced in the previous example and regresses Horsepower on EngineSize. The code example prints the model results to standard out, which is shown below, and then creates a scatter chart with the regression line clearly displayed.

//Load the data
DataFrame<Integer,String> data = DataFrame.read().csv(options -> {
    options.setResource("http://zavtech.com/data/samples/cars93.csv");
    options.setExcludeColumnIndexes(0);
});

//Run OLS regression and plot 
String regressand = "Horsepower";
String regressor = "EngineSize";
data.regress().ols(regressand, regressor, true, model -> {
    System.out.println(model);
    DataFrame<Integer,String> xy = data.cols().select(regressand, regressor);
    Chart.create().withScatterPlot(xy, false, regressor, chart -> {
        chart.title().withText(regressand + " regressed on " + regressor);
        chart.subtitle().withText("Single Variable Linear Regression");
        chart.plot().style(regressand).withColor(Color.RED).withPointsVisible(true);
        chart.plot().trend(regressand).withColor(Color.BLACK);
        chart.plot().axes().domain().label().withText(regressor);
        chart.plot().axes().domain().format().withPattern("0.00;-0.00");
        chart.plot().axes().range(0).label().withText(regressand);
        chart.plot().axes().range(0).format().withPattern("0;-0");
        chart.show();
    });
    return Optional.empty();
});

==============================================================================================
                                   Linear Regression Results                                                            
==============================================================================================
Model:                                   OLS    R-Squared:                            0.5360
Observations:                             93    R-Squared(adjusted):                  0.5309
DF Model:                                  1    F-Statistic:                        105.1204
DF Residuals:                             91    F-Statistic(Prob):                  1.11E-16
Standard Error:                      35.8717    Runtime(millis)                           52
Durbin-Watson:                        1.9591                                                
==============================================================================================
   Index     |  PARAMETER  |  STD_ERROR  |  T_STAT   |   P_VALUE   |  CI_LOWER  |  CI_UPPER  |
----------------------------------------------------------------------------------------------
  Intercept  |    45.2195  |    10.3119  |   4.3852  |   3.107E-5  |    24.736  |   65.7029  |
 EngineSize  |    36.9633  |     3.6052  |  10.2528  |  7.573E-17  |    29.802  |   44.1245  |
==============================================================================================

UK House Price Trends

It is possible to access all UK residential real-estate transaction records from 1995 through to current day via the UK Government Open Data initiative. The data is presented in CSV format, and contains numerous columns including such information as the transaction date, price paid, fully qualified address (including postal code), property type, lease type, and so on.

Let's begin by writing a function to load these CSV files from Amazon S3 buckets, and since they are stored one file per year, we provide a parameterized function accordingly. Given the requirements of our analysis, there is no need to load all the columns in the file, so below we only choose to read columns at index 1, 2, 4, and 11. In addition, since the files do not include a header, we re-name columns to something more meaningful to make subsequent access a little clearer.

/**
 * Loads UK house price from the Land Registry stored in an Amazon S3 bucket
 * Note the data does not have a header, so columns will be named Column-0, Column-1 etc...
 * @param year      the year for which to load prices
 * @return          the resulting DataFrame, with some columns renamed
 */
private DataFrame<Integer,String> loadHousePrices(Year year) {
    String resource = "http://prod.publicdata.landregistry.gov.uk.s3-website-eu-west-1.amazonaws.com/pp-%s.csv";
    return DataFrame.read().csv(options -> {
        options.setResource(String.format(resource, year.getValue()));
        options.setHeader(false);
        options.setCharset(StandardCharsets.UTF_8);
        options.setIncludeColumnIndexes(1, 2, 4, 11);
        options.getFormats().setParser("TransactDate", Parser.ofLocalDate("yyyy-MM-dd HH:mm"));
        options.setColumnNameMapping((colName, colOrdinal) -> {
            switch (colOrdinal) {
                case 0:     return "PricePaid";
                case 1:     return "TransactDate";
                case 2:     return "PropertyType";
                case 3:     return "City";
                default:    return colName;
            }
        });
    });
}

Below, we use this data in order to compute the median nominal price (not inflation adjusted) of an apartment for each year between 1995 through 2014 for a subset of the largest cities in the UK. There are about 20 million records in the unfiltered dataset between 1993 and 2014, and while it takes a fairly long time to load and parse (approximately 3.5GB of data), Morpheus executes the analytical portion of the code in about five seconds (not including load time) on a standard Apple MacBook Pro purchased in late 2013. Note how we use parallel processing to load and process the data by calling results.rows().keys().parallel().

//Create a data frame to capture the median prices of Apartments in the UK'a largest cities
DataFrame<Year,String> results = DataFrame.ofDoubles(
    Range.of(1995, 2015).map(Year::of),
    Array.of("LONDON", "BIRMINGHAM", "SHEFFIELD", "LEEDS", "LIVERPOOL", "MANCHESTER")
);

//Process yearly data in parallel to leverage all CPU cores
results.rows().keys().parallel().forEach(year -> {
    System.out.printf("Loading UK house prices for %s...\n", year);
    DataFrame<Integer,String> prices = loadHousePrices(year);
    prices.rows().select(row -> {
        //Filter rows to include only apartments in the relevant cities
        final String propType = row.getValue("PropertyType");
        final String city = row.getValue("City");
        final String cityUpperCase = city != null ? city.toUpperCase() : null;
        return propType != null && propType.equals("F") && results.cols().contains(cityUpperCase);
    }).rows().groupBy("City").forEach(0, (groupKey, group) -> {
        //Group row filtered frame so we can compute median prices in selected cities
        final String city = groupKey.item(0);
        final double priceStat = group.colAt("PricePaid").stats().median();
        results.data().setDouble(year, city, priceStat);
    });
});

//Map row keys to LocalDates, and map values to be percentage changes from start date
final DataFrame<LocalDate,String> plotFrame = results.mapToDoubles(v -> {
    final double firstValue = v.col().getDouble(0);
    final double currentValue = v.getDouble();
    return (currentValue / firstValue - 1d) * 100d;
}).rows().mapKeys(row -> {
    final Year year = row.key();
    return LocalDate.of(year.getValue(), 12, 31);
});

//Create a plot, and display it
Chart.create().withLinePlot(plotFrame, chart -> {
    chart.title().withText("Median Nominal House Price Changes");
    chart.title().withFont(new Font("Arial", Font.BOLD, 14));
    chart.subtitle().withText("Date Range: 1995 - 2014");
    chart.plot().axes().domain().label().withText("Year");
    chart.plot().axes().range(0).label().withText("Percent Change from 1995");
    chart.plot().axes().range(0).format().withPattern("0.##'%';-0.##'%'");
    chart.plot().style("LONDON").withColor(Color.BLACK);
    chart.legend().on().bottom();
    chart.show();
});

The percent change in nominal median prices for apartments in the subset of chosen cities is shown in the plot below. It shows that London did not suffer any nominal house price decline as a result of the Global Financial Crisis (GFC); however, not all cities in the UK proved as resilient. What is slightly surprising is that some of the less affluent northern cities saw a higher rate of appreciation in the 2003 to 2006 period compared to London. One thing to note is that while London did not see any nominal price reduction, there was certainly a fairly severe correction in terms of EUR and USD since Pound Sterling depreciated heavily against these currencies during the GFC.

Visualization

Visualizing data in Morpheus DataFrames is made easy via a simple chart abstraction API with adapters supporting both JFreeChart as well as Google Charts (with others to follow by popular demand). This design makes it possible to generate interactive Java Swing charts as well as HTML5 browser-based charts via the same programmatic interface. For more details on how to use this API, see the section on visualization here and the code here. There are just a few charts below.

Maven Artifacts

Morpheus is published to Maven Central, so it can be easily added as a dependency in your build tool of choice. The codebase is currently divided into five repositories to allow each module to be evolved independently. The core module, which is aptly named morpheus-core, is the foundational library on which all other modules depend. The various Maven artifacts are as follows:

Morpheus Core

This is the foundational library that contains Morpheus arrays, DataFrames, and other key interfaces and implementations.

<dependency>
    <groupId>com.zavtech</groupId>
    <artifactId>morpheus-core</artifactId>
    <version>${VERSION}</version>
</dependency>

Morpheus Visualization

The visualization components to display DataFrames in charts and tables:

<dependency>
    <groupId>com.zavtech</groupId>
    <artifactId>morpheus-viz</artifactId>
    <version>${VERSION}</version>
</dependency>

Morpheus Quandl

The adapter to load data from Quandl:

<dependency>
    <groupId>com.zavtech</groupId>
    <artifactId>morpheus-quandl</artifactId>
    <version>${VERSION}</version>
</dependency>

Morpheus Google

The adapter to load data from Google Finance:

<dependency>
    <groupId>com.zavtech</groupId>
    <artifactId>morpheus-google</artifactId>
    <version>${VERSION}</version>
</dependency>

Morpheus Yahoo

The adapter to load data from Yahoo! Finance:

<dependency>
    <groupId>com.zavtech</groupId>
    <artifactId>morpheus-yahoo</artifactId>
    <version>${VERSION}</version>
</dependency>

And that's it!