FAQ: How does ClearGov forecasting work?

ClearGov utilizes an LSTM Recurrent Neural Network to create valid budget forecasts based on limited initial data sets. This methodology has been shown to be far more reliable, especially when subject to data-set limitations.

Overview

For starters, while there are multiple methods to approach data forecasting, most forecasting solutions are unreliable and/or extremely difficult to implement, and therefore not much better than a stab in the dark. Simple methods - like regression analysis - are usually linear, whereas non-linear algorithms are typically heavily data-dependent, so they require massive data sets to produce meaningful results.

When it comes to the ClearGov platform, our complete database actually includes trillions of dollars of spend across hundreds of millions of line items. So, in aggregate, there is enough data to leverage a pretty straight-forward non-linear algorithm to create an accurate forecast on the overall database. However, that would not be valuable or meaningful for an individual Client.

ClearGov Methodology

Relatively speaking, creating forecasts for individual lines items, or even for budgets as a whole for a specific ClearGov customer, involves a much, much smaller data set. Therefore, to overcome this data-set limitation, ClearGov has leveraged its expertise in AI and machine learning and has implemented something called a Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN). That is a long, complex term for what is, in fact, a fairly long and complex technology. If you're really interested in geeking out, the ClearGov solution is based on Jason’s Brownlee research on the topic, which can be found here.

There are a few key concepts to highlight from this article:

• Time series prediction problems are a difficult type of predictive modeling problem.
• Unlike regression predictive modeling, time series also adds to the complexity of a sequence dependence among the input variables. A recurrent neural network is designed to handle such sequence dependencies.
• The Long Short-Term Memory network or LSTM network is a type of recurrent neural network used in deep learning because very large architectures can be successfully trained with relatively small number of inputs.

Methodology Explanation

If you're looking for a layman's description, the key differentiator of an LSTM RNN is that it is essentially a neural network that learns and remembers specific data points. By learning and remember those items, the network can then make inferences by stringing those data points together.

As a very simple analogy, think about how humans learn to read. You can read and understand these sentences because you first learned and remembered the meaning of the individual words. And, you can read and understand other sentences even if the words are strung together differently (or use different words altogether) provided that A) you have already learned all of the individual words and B) the words are strung together in a logical order that makes sense. It is an over-simplification, but an LSTM Recurrent Neural Network kinda, sorta does the same thing. The key point is that you can take a relatively small number of data points (words) and create a large amount of meaningful content (sentences) from them.

For this same reason, the LSTM RNN that we built enables ClearGov to take a relatively small number of data points from a Client's historical budget data and create meaningful forecasts. And, if a Client provides additional historical budget data (i.e. more years), then ClearGov can create longer and more accurate forecasts. Back to the reading analogy, the more words (data) you have learned, the better you are able to understand longer and more complex sentences (forecasts). It's not a perfect analogy, but that's the general gist of it.