ml4debugging

I want to propose a three point Manifesto that Governs the Communication between a Computer and its Programmer.

The Manifesto reads as follows:

  • A Computer will be polite and respectful to its programmer.
  • A Computer will speak to its programmer in the natural language of its programmer, and not in its own internal language of cryptic error messages and stack traces.
  • A Computer will respond to the intent of its programmer, and not strictly to the sequence of characters entered by the programmer.
The first project in realizing this Manifesto is ml4debugging. The idea behind ml4debugging is to use a Machine Learning approach to translating the error messages of an interpreter or compiler into natural language. Specifically, within the field of Machine Learning, there is a subfield called Machine Translation, which uses computers to translate from one language another, such as from English to French or French to English. The technology used is called a Recurrent Neural Network.

We can use the same technology to think of the error message generated by a computer language interpreter, along with the line of code containing the error, as the "source" language, and the natural language description of the error message as the "target" language.

An example:

This is the present error message of a Python interpreter in response to a fragment of Python code:

in rnn_forward(x, a0, parameters)
     47     # Append "cache" to "caches" (?1 line)
     48     caches = np.append( caches, cache )
    ===> 49     print("cache.shape: " + str(cache.shape))
     50     print("caches.shape: " + str(t) + " "+ str(caches.shape))
     51     ### END CODE HERE ###
AttributeError: 'tuple' object has no attribute 'shape'

And this is the correct message as it should be:
Sandeep, I'm having a problem interpreting your code. I believe the problem is that in Line 49 of your code, the variable "cache" is of type "tuple", and a tuple cannot be referenced by a "shape" attribute.

The idea is not necessarily to replace the error message or stack trace of the interpreter, but possibly to supplement it with a message in natural language.

You can see a more complete list of such message pairs at this link: Present and Correct Message Pairs. And the underlying input-output encoding that constitutes the technology of the neural network is given at this link: The Input and Output Encodings of the Neural Network.

This project is a work in progress. In the first iteration, we will only use the information present in the line in which the error occurs to translate the error message. This, I believe, can be handled by a vanilla Recurrent Neural Network. Later on, we may use information from the surrounding lines of code, which, in a more advanced form, would be handled by a variation on a vanilla Recurrent Neural Network called a "Bidirectional LSTM Network".

The primary challenge in this project is to put together the training data that will be used to train the neural network, and I am still searching for a clever way to do it which does not make it necessary to hand-write all the message pairs.