Once the model has been constructed, its forecast quality needs to be tested (validated) before using the model for actual forecast. If the model shows good validation results, then one can feel more confident about the actual forecast that one makes with it eventually. To do this, 2 methods are available in CPT. The first is called Retroactive Forecast Validation method, while the second is called the Cross-Validation method.

Retroactive Forecast Validation

This validation method is the simpler between the two. The forecasts make use of only the earlier parts of the historical records to build the prediction model, and then make forecasts on the remaining later years. This method therefore requires enough cases in history that holding out later years leaves enough early years to develop or train the model. Going back to our earlier example of having SST-rainfall data pairs for June from 1970 to 2009, the first iteration could involve training the model between 1970 to 1979, and then making forecast for the year 1980 and after. The second iteration could involve training the model from 1970 to 1980, and then making forecast for 1981, and so on. By the end of the iterations, one would have pairs of observed and forecasted rainfall rate data from 1980 to 2009. With these values, the Pearson's Correlation Coefficients of the models and other model performance measures such as mean-squared error, categorical hit skill scores, etc can be calculated.

Cross Validation

When there are too few years in the data history for retroactive forecasts validation (less than 25 years could be considered few), the method of cross-validation is preferred. In this method, only a few (or even one) cases are excluded from the model training. The number of cases excluded from the model training is termed the Cross-validation Window parameter of the method, and these cases are being used for independent forecast tests. So rather than being done just once, this exercise is repeated for many different choices of which years are withheld, and the withheld years do not necessarily occur later than the years used for development of the model. The following diagram shows the actual observations (red) and the hindcasts (green) produced by the model for a particular grid in the region. For this case, you can see that the model performs quite well for the hindcasts (the green line follows the red line quite closely), and in fact the pearson's correlation score calculated is 0.68. The higher the value of the pearson's correlation score the better, and the perfect hindcast score would be exactly 1. But for the the application of PCR to seasonal climate forecasting, pearson's correlation coefficient of more than 0.6 would be good enough.

And if we attempt to forecast not for just a grid just over Singapore, but for the whole of the ASEAN region, we would have the pearson's correlation scores for all the grids in the region separately. And by putting them together, we get a pearson's correlation score map like the one shown below.

From this particular map (e.g. taken from Aug, Sept and Oct 2010 seasonal rainfall rate forecast), we could tell that the forecast quality can be expected to be better over places close to the Indonesian Archipelago, especially over the Islands of Sulawesi and New Guinea.