Training Analysis Page: Directional Accuracy

The Modify Field Dialog allows you to analyze and modify the properties of a field. The Training Analysis page allows you to view an analysis of the prediction results versus the desired values.

&  For help with predictions, see Predicting and Modeling Financial Data.

Directional Accuracy Sub-Page Data

This sub-page displays an analysis of the directional accuracy of the predicted values. Specifically, it reports whether the predicted values increase or decrease at the same time as the desired values. This is useful for determining if the direction of change in the predicted values is useful even when the error or correlation values are marginal.

If the desired output is not pre-differenced or if Change or Percent Change preprocessing is used, directional information is based on the sign of the change in the predicted and desired valued from the last known value. If the desired output is pre-differenced and no preprocessing is used, the sign of the prediction and desired output will be used.

The values are presented as a percentage of samples in which the direction was predicted correctly. A directional accuracy of 100% would indicate that the predicted changes for every sample were in the correct direction. A directional accuracy of 50% would indicate that the direction was correct half the time, which would be equivalent to flipping a coin and choosing a random direction.

accuracy = 100% perfect accuracy

accuracy = 50% no directional accuracy

accuracy = 0% perfect inverse

The analysis is performed for the training, cross validation, and accuracy testing sets used for the prediction. Each type of analysis is performed on the entire subset. In addition, each type of analysis is also performed for when the value is predicted to increase or decrease. This is useful for determining if predictions in a particular direction are more accurate than others.

Ä    Note: If a value associated with a predicted direction is listed as "n/a", no predictions were made in that direction.

Since directional accuracy is based on the change from the previous value, only one report is displayed.

Directional Accuracy Sub-Page Analysis

The analysis presented on this page is based on the directional accuracy values. It detects common characteristics to look for in the data and is intended only as a starting point for evaluating the model. Some common results include:

·      This might be a good / reasonable / weak directional model.

This is an analysis of the directional accuracy in the testing set. If no testing set is used, the cross validation or training set directional accuracy is used. The following table is used:

accuracy > 70% excellent model

accuracy > 65% very good model

accuracy > 60% good model

accuracy > 55% reasonable model

accuracy <= 55% weak model

The evaluation of models in this way is very abstract. Models that are classified as "excellent" or "good" may be good at mirroring the desired data, but may not predict the values in a way that is useful in the way that was intended. Similarly, models that are classified as "weak" may produce values that are still useful. As stated above, this is intended only as a starting point for evaluating the model.

·      It predicted the direction of changes better / worse than simply predicting a random direction.

This is an explanation of the rationale for the analysis text. As explained above, a directional accuracy of 50% is equivalent to predicting a random direction. Therefore, the effectiveness of the model is based on it improving over selecting a random direction.

·      It is specialized on the training data.

This is an analysis of the directional accuracy for the testing set as compared to the directional accuracy for the training set. If no testing set is used, the directional accuracy for the cross validation set is used.

This is significant when determining whether the training has produced a model that is good at making generalizations outside of the training set. If a model over-trains or simply memorizes the training data, it may perform very well at predicting values in the training set; however, it will tend to perform poorly at predicting the values outside of the training set. A good predictive model will perform equally well on data in the training, cross validation, and testing sets.

This message may appear if the samples to weights ratio is not adequate for the type of data being used. A good rule of thumb is to have about ten training samples for each weight in the network. Significantly lower ratios may allow the neural network to simply use the weights to memorize the data, rather than make effective generalizations about its characteristics. See the Prediction Model page for a report of this ratio and help for improving it.

·      It is specialized on upward / downward trends.

This is an analysis of the directional accuracy across all data sets, comparing the directional accuracy when upward changes are predicted to the directional accuracy when downward changes are predicted. If the directional accuracy is significantly better for predictions in a given direction, the model may be specialized on predicting changes in that direction. This is significant when determining whether directional information from the prediction is useful.

This message may appear when the training set contains data that trends mostly in the reported direction. In this case, the network may determine that the average change is in a given direction and use that average change when it does not have enough information to correctly predict a value. A report of the actual number of changes up and down is on the Overview sub-page.

·      It is only producing upward / downward predictions for this data.

This is an analysis of the directional accuracy across all data sets. It indicates that a model is predicting changes in only one direction. If the training set does not contain predominantly changes in one direction, this is typically an indication that the model did not have enough relevant information to produce an effective model.

If a problem occurred during the training or calculation phase, the analysis will be replaced with a description of the error. A summary of these error messages is displayed on the help for the Modify Field Dialog: Training Analysis page.

How Did I Get Here?

This is a sub-page of the Modify Field Dialog: Training Analysis page.