Using Machine Learning for Static Analysis

In some ways, machine learning and AI systems are becoming a victim of their own success. While they are genuinely useful in many fields, particularly when it comes to marketing analysis and for cybersecurity, their utility in these fields means that some people have tried to use them for everything. That includes using machine learning systems to create static code analyzers for locating security vulnerabilities.

Some of these attempts have met with a modicum of success: Facebook, Amazon, and Mozilla all now offer some form of ML-driven static code analyzers. However, and as anyone who understands the fundamentals of machine learning will appreciate, these approaches also come with some inherent limitations. 

Machine Learning Static Analyzers

In the last few years, we’ve seen a plethora of ML-driven static analyzers appear on the market. Some of these have been developed by enthusiastic amateurs; others by tech giants, with Facebook, Amazon, and Mozilla all now offering tools of this kind. Each new release generates a huge amount of hype, and each promises to revolutionize the development process.

In reality, these tools offer developers efficiency savings when it comes to searching for vulnerabilities and bugs, but no static code analyzer can yet catch everything that is wrong with a particular piece of code. Let’s take a look at the most popular options.

DeepCode

DeepCode is perhaps the best-known vulnerability-searching program for Java, JavaScript, and Python. DeepCode also features a machine learning module that the developers claim works as a “Grammarly for developers”.

The machine learning module for DeepCode works by looking at a huge repo of changes from developers as they work on a huge variety of projects. By learning from this repo, DeepCode is able to offer developers a suggested solution to the problem they are working on, as well as catching errors that have arisen before.

DeepCode still contains some limitations, however. The developers of the system have claimed that C++ support will be in place since 2018, but to date this is still not the case. That said, it’s possible for developers to work with C++ and DeepCode via an add-on described here.

Infer

Infer has been around for almost a decade now, and was acquired by Facebook way back in 2013 as a basis for a static code analyzer based upon ML principles. Infer has become extremely popular as a static analyzer for a number of reasons: it supports a wide range of languages, and can be used in conjunction with Amazon Web Services and Oculus. Best of all, source code for the project was made open back in 2015, spurring rapid development.

Despite its popularity, even developers who use Infer accept that even within Facebook projects only 80% of the warnings it generates are useful. It will find pointer dereference and memory leak errors, but there still remain whole classes of errors that Infer cannot detect, including type casting exceptions and unverified data leaks.

Source{d}

Source{d} is an open source static code analyzer and development manager that is more focused on providing information to managers on the progress of particular software projects, but that also provides many tools for developers. As a part of static application security testing protocols, it can perform many useful SAST functions including analyzing byte codes and binaries and application source codes for vulnerabilities. 

One of the major advantages of this software is that its source code is completely transparent, and available in a GitHub repo, allowing developers to essentially build their own ML-enhanced code analyzers.

This said, Source{d} is pretty limited when it comes to isolating errors in code. It uses the Babelfish service to convert specific code instances into universal syntax trees, and from there is able to simplify and suggest edits to code that make it much easier to work with. This can offer developers significant efficiency savings when it comes to working with code, but falls short of being a full static analyzer. 

The Rest

In addition to these three solutions, there are a few emerging analyzers that hold promise. Clever-commit is a project by Mozilla, but one that remains frustratingly vague when it comes to details; CodeGuru is Amazon’s ML-enhanced static code analyzer, but currently only works with Java; and Embold is a start-up platform for error analysis that offers visualizations of code dependencies but is also limited in the languages it can be used with.

All of these systems can be extremely useful for developers, but only where they are used with particular languages, and only where they are trained on achieving particular outcomes. In other words, the promise of ML-enabled static code analyzers – a multi-purpose tool that will catch errors and glitches in a wide variety of languages and environments – has not yet been realized. 

Enthusiasts will say that this is merely because ML-driven static analyzers still require development. However, a reconsideration of how ML systems actually work suggests that these problems may persist for some time to come.

Conclusion

In conclusion, let’s be clear. ML technologies are still in their infancy, and are likely to find many useful applications in many areas of development. However, as is clear from the above, it remains difficult to imagine this technology changing the way in which static analysis tools are built.

At the most fundamental level, this is because static analysis is inherently different to machine learning platforms. Instead of the hazy, probabilistic logic that is used to run AI and ML systems, error isolation and remediation requires pin-point, repeatable methods. Ultimately, you don’t want your static analyzer to suggest that your code is “likely” to be wrong, based on other examples; you want to know, definitively, whether it will work.

Sadly, that certainty is only achievable with explicitly coded analyzers. At least for now.