View source for Training, validation, and test data sets

{{short description|Tasks in machine learning}}
{{Machine learning}}
{{anchor|Parameter}}<!-- Parked anchor for redirects -->
In [[Machine learning|machine learning]], a common task is the study and construction of [[Algorithm|algorithm]]s that can learn from and make predictions on [[Data|data]].<ref>{{cite journal |title=Glossary of terms |author1=Ron Kohavi |author2=Foster Provost |journal=Machine Learning |volume=30 |pages=271–274 |year=1998 |url=https://ai.stanford.edu/~ronnyk/glossary.html|doi=10.1023/A:1007411609915 |doi-access=free }}</ref> Such algorithms function by making data-driven predictions or decisions,<ref name="bishop">{{cite book |first=Christopher M. |last=Bishop |title=Pattern Recognition and Machine Learning |location=New York |publisher=Springer |year=2006 |isbn=0-387-31073-8 |page=vii |quote=Pattern recognition has its origins in engineering, whereas machine learning grew out of computer science. However, these activities can be viewed as two facets of the same field, and together they have undergone substantial development over the past ten years. }}</ref> through building a [[Mathematical model|mathematical model]] from input data. These input data used to build the model are usually divided into multiple [[Data set|data set]]s. In particular, three data sets are commonly used in different stages of the creation of the model: training, validation, and test sets.

The model is initially fit on a '''training data set''',<ref name="James 2013 176">{{cite book|last=James|first=Gareth|title=An Introduction to Statistical Learning: with Applications in R|date=2013|publisher=Springer|isbn=978-1461471370|page=176|url=http://www-bcf.usc.edu/~gareth/ISL/}}</ref> which is a set of examples used to fit the parameters (e.g. weights of connections between neurons in artificial neural networks) of the model.<ref name="Ripley 1996 354">{{cite book|last=Ripley|first=Brian|title=Pattern Recognition and Neural Networks|url=https://archive.org/details/patternrecogniti00ripl|url-access=limited|date=1996|publisher=Cambridge University Press|isbn=978-0521717700|page=[https://archive.org/details/patternrecogniti00ripl/page/n365 354]}}</ref> The model (e.g. a [[Naive Bayes classifier|naive Bayes classifier]]) is trained on the training data set using a [[Supervised learning|supervised learning]] method, for example using optimization methods such as [[Gradient descent|gradient descent]] or [[Stochastic gradient descent|stochastic gradient descent]]. In practice, the training data set often consists of pairs of an input [[Array data structure|vector]] (or scalar) and the corresponding output vector (or scalar), where the answer key is commonly denoted as the ''target'' (or ''label''). The current model is run with the training data set and produces a result, which is then compared with the ''target'', for each input vector in the training data set. Based on the result of the comparison and the specific learning algorithm being used, the parameters of the model are adjusted. The model fitting can include both [[Feature selection|variable selection]] and parameter [[Estimation theory|estimation]].

Successively, the fitted model is used to predict the responses for the observations in a second data set called the '''validation data set'''.<ref name="James 2013 176"/> The validation data set provides an unbiased evaluation of a model fit on the training data set while tuning the model's [[Hyperparameter (machine learning)|hyperparameters]]<ref name="Brownlee">{{cite web |url=https://machinelearningmastery.com/difference-test-validation-datasets/|title=What is the Difference Between Test and Validation Datasets?|last=Brownlee|first=Jason|access-date=12 October 2017|date=2017-07-13}}</ref> (e.g. the number of hidden units—layers and layer widths—in a neural network<ref name="Ripley 1996 354"/>). Validation data sets can be used for [[Regularization (mathematics)|regularization]] by [[Early stopping|early stopping]] (stopping training when the error on the validation data set increases, as this is a sign of [[Overfitting|over-fitting]] to the training data set).<ref name="prechelt_early_2012">{{Cite book
| publisher = Springer Berlin Heidelberg
| isbn = 978-3-642-35289-8
| pages = [https://archive.org/details/neuralnetworkstr00mlle/page/n60 53]–67
| editor = Grégoire Montavon
| editor2 = Klaus-Robert Müller
| last = Prechelt
| first = Lutz
|author2=Geneviève B. Orr
| title = Neural Networks: Tricks of the Trade
| url = https://archive.org/details/neuralnetworkstr00mlle
| url-access = limited
| chapter = Early Stopping — But When?
| series = Lecture Notes in Computer Science
| date = 2012-01-01
| doi = 10.1007/978-3-642-35289-8_5
}}</ref>
This simple procedure is complicated in practice by the fact that the validation data set's error may fluctuate during training, producing multiple local minima. This complication has led to the creation of many ad-hoc rules for deciding when over-fitting has truly begun.<ref name="prechelt_early_2012"/>

Finally, the '''test data set''' is a data set used to provide an unbiased evaluation of a ''final'' model fit on the training data set.<ref name="Brownlee"/> If the data in the test data set has never been used in training (for example in [[Cross-validation (statistics)|cross-validation]]), the test data set is also called a '''holdout data set'''. The term "validation set" is sometimes used instead of "test set" in some literature (e.g., if the original data set was partitioned into only two subsets, the test set might be referred to as the validation set).<ref name="Brownlee"/>

Deciding the sizes and strategies for data set division in training, test and validation sets is very dependent on the problem and data available.<ref>{{Cite web|title=Machine learning - Is there a rule-of-thumb for how to divide a dataset into training and validation sets?|url=https://stackoverflow.com/questions/13610074/is-there-a-rule-of-thumb-for-how-to-divide-a-dataset-into-training-and-validatio|access-date=2021-08-12|website=Stack Overflow}}</ref>

== Training data set ==
{{multiple image
   | direction = horizontal
   | total_width = 400
   | footer    = 
   | image1    = Simplified neural network training example.svg
   | alt1      = 
   | caption1  = Simplified example of training a neural network in object detection: The network is trained by multiple images that are known to depict [[Biology:Starfish|starfish]] and [[Biology:Sea urchin|sea urchin]]s, which are correlated with "nodes" that represent visual [[Feature (computer vision)|features]]. The starfish match with a ringed texture and a star outline, whereas most sea urchins match with a striped texture and oval shape. However, the instance of a ring textured sea urchin creates a weakly weighted association between them.
   | image2    = Simplified neural network example.svg
   | alt2      = 
   | caption2  = Subsequent run of the network on an input image (left):<ref>{{cite book|author=Ferrie, C., & Kaiser, S.|year=2019|title=Neural Networks for Babies|publisher=Sourcebooks|isbn=1492671207}}</ref> The network correctly detects the starfish. However, the weakly weighted association between ringed texture and sea urchin also confers a weak signal to the latter from one of two intermediate nodes. In addition, a shell that was not included in the training gives a weak signal for the oval shape, also resulting in a weak signal for the sea urchin output. These weak signals may result in a false positive result for sea urchin.<br>In reality, textures and outlines would not be represented by single nodes, but rather by associated weight patterns of multiple nodes.}}
A training data set is a data set of examples used during the learning process and is used to fit the parameters (e.g., weights) of, for example, a classifier.<ref name="Ripley, B.D. 1996 p. 354">Ripley, B.D. (1996) ''Pattern Recognition and Neural Networks'', Cambridge: Cambridge University Press, p. 354</ref><ref name="cann-faq">"[ftp://ftp.sas.com/pub/neural/FAQ.html#A_data Subject: What are the population, sample, training set, design set, validation set, and test set?]", [ftp://ftp.sas.com/pub/neural/FAQ.html Neural Network FAQ, part 1 of 7: Introduction] ([ftp://ftp.sas.com/pub/neural/FAQ1.txt txt]), comp.ai.neural-nets, Sarle, W.S., ed. (1997, last modified 2002-05-17)</ref> 

For classification tasks, a supervised learning algorithm looks at the training data set to determine, or learn, the optimal combinations of variables that will generate a good [[Predictive modelling|predictive model]].<ref name="Larose2014">{{cite book | last1=Larose | first1=D. T. | last2=Larose | first2=C. D. | title=Discovering knowledge in data : an introduction to data mining | publisher=Wiley | publication-place=Hoboken | year=2014 | isbn=978-0-470-90874-7 | oclc=869460667 | doi=10.1002/9781118874059}}</ref> The goal is to produce a trained (fitted) model that generalizes well to new, unknown data.<ref name="Xu Goodacre 2018">{{cite journal | last1=Xu | first1=Yun | last2=Goodacre | first2=Royston | title=On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling for Estimating the Generalization Performance of Supervised Learning | journal=Journal of Analysis and Testing | publisher=Springer Science and Business Media LLC | volume=2 | issue=3 | year=2018 | issn=2096-241X | doi=10.1007/s41664-018-0068-2 | pages=249–262| pmid=30842888 | pmc=6373628 | doi-access=free }}</ref> The fitted model is evaluated using “new” examples from the held-out data sets (validation and test data sets) to estimate the model’s accuracy in classifying new data.<ref name="Brownlee"/> To reduce the risk of issues such as over-fitting, the examples in the validation and test data sets should not be used to train the model.<ref name="Brownlee"/>

Most approaches that search through training data for empirical relationships tend to overfit the data, meaning that they can identify and exploit apparent relationships in the training data that do not hold in general.

== Validation data set ==
A validation data set is a data set of examples used to tune the [[Hyperparameter (machine learning)|hyperparameter]]s (i.e. the architecture) of a classifier. It is sometimes also called the development set or the "dev set".<ref>{{Cite web|title=Deep Learning|url=https://www.coursera.org/specializations/deep-learning|access-date=2021-05-18|website=Coursera|language=en}}</ref> An example of a hyperparameter for artificial neural networks includes the number of hidden units in each layer.<ref name="Ripley, B.D. 1996 p. 354" /><ref name="cann-faq" /> It, as well as the testing set (as mentioned below), should follow the same probability distribution as the training data set.

In order to avoid overfitting, when any [[Statistical classification|classification]] parameter needs to be adjusted, it is necessary to have a validation data set in addition to the training and test data sets. For example, if the most suitable classifier for the problem is sought, the training data set is used to train the different candidate classifiers, the validation data set is used to compare their performances and decide which one to take and, finally, the test data set is used to obtain the performance characteristics such as accuracy, [[Sensitivity and specificity|sensitivity]], [[Sensitivity and specificity|specificity]], [[Precision and recall#F-measure|F-measure]], and so on. The validation data set functions as a hybrid: it is training data used for testing, but neither as part of the low-level training nor as part of the final testing.

The basic process of using a validation data set for [[Model selection|model selection]] (as part of training data set, validation data set, and test data set) is:<ref name="cann-faq" /><ref>Bishop, C.M. (1995), ''[https://books.google.com/books?id=T0S0BgAAQBAJ Neural Networks for Pattern Recognition]'', Oxford: Oxford University Press, p. 372</ref>

{{quote|
Since our goal is to find the network having the best performance on new data, the simplest approach to the comparison of different networks is to evaluate the error function using data which is independent of that used for training. Various networks are trained by minimization of an appropriate error function defined with respect to a training data set. The performance of the networks is then compared by evaluating the error function using an independent validation set, and the network having the smallest error with respect to the validation set is selected. This approach is called the ''hold out'' method. Since this procedure can itself lead to some overfitting to the validation set, the performance of the selected network should be confirmed by measuring its performance on a third independent set of data called a test set.}}

An application of this process is in [[Early stopping|early stopping]], where the candidate models are successive iterations of the same network, and training stops when the error on the validation set grows, choosing the previous model (the one with minimum error).

== Test data set ==
A test data set is a data set that is [[Independence (probability theory)|independent]] of the training data set, but that follows the same [[Probability distribution|probability distribution]] as the training data set. If a model fit to the training data set also fits the test data set well, minimal [[Overfitting|overfitting]] has taken place (see figure below). A better fitting of the training data set as opposed to the test data set usually points to over-fitting.

A test set is therefore a set of examples used only to assess the performance (i.e. generalization) of a fully specified classifier.<ref name="Ripley, B.D. 1996 p. 354"/><ref name="cann-faq"/> To do this, the final model is used to predict classifications of examples in the test set. Those predictions are compared to the examples' true classifications to assess the model's accuracy.<ref name="Larose2014" />

In a scenario where both validation and test data sets are used, the test data set is typically used to assess the final model that is selected during the validation process. In the case where the original data set is partitioned into two subsets (training and test data sets), the test data set might assess the model only once (e.g., in the holdout method).<ref name="Kohavi2001">{{Cite journal | url=https://www.researchgate.net/publication/2352264 | title=A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection| volume=14| date=2001-03-03| last1=Kohavi| first1=Ron}}</ref> Note that some sources advise against such a method.<ref name="Xu Goodacre 2018" /> However, when using a method such as [[Cross-validation (statistics)|cross-validation]], two partitions can be sufficient and effective since results are averaged after repeated rounds of model training and testing to help reduce bias and variability.<ref name="Brownlee"/><ref name="Xu Goodacre 2018" />


[[File:traintest.svg|center|700px|thumb|A training set (left) and a test set (right) from the same statistical population are shown as blue points.  Two predictive models are fit to the training data.  Both fitted models are plotted with both the training and test sets.  In the training set, the [[Mean squared error|MSE]] of the fit shown in orange is 4 whereas the MSE for the fit shown in green is 9.  In the test set, the MSE for the fit shown in orange is 15 and the MSE for the fit shown in green is 13.  The orange curve severely overfits the training data, since its MSE increases by almost a factor of four when comparing the test set to the training set.  The green curve overfits the training data much less, as its MSE increases by less than a factor of 2.]]

== Confusion in terminology ==

Testing is trying something to find out about it ("To put to the proof; to prove the truth, genuineness, or quality of by experiment" according to the Collaborative International Dictionary of English) and 
to validate is to prove that something is valid ("To confirm; to render valid" Collaborative International Dictionary of English). With this perspective, the most common use of the terms '''test set''' and '''validation set''' is the one here described. However, in both industry and academia, they are sometimes used interchanged, by considering that the internal process is testing different models to improve (test set as a development set) and the final model is the one that needs to be validated before real use with an unseen data (validation set). "The literature on machine learning often reverses the meaning of 'validation' and 'test' sets. This is the most blatant example of the terminological confusion that pervades artificial intelligence research."<ref>{{Cite book|title=Pattern recognition and neural networks|last=Ripley, Brian D.|date=2009|publisher=Cambridge Univ. Press|isbn=9780521717700|pages=Glossary|oclc=601063414}}</ref> Nevertheless, the important concept that must be kept is that the final set, whether called test or validation, should only be used in the final experiment.

== Cross-validation ==
In order to get more stable results and use all valuable data for training, a data set can be repeatedly split into several training and a validation data sets. This is known as [[Cross-validation (statistics)|cross-validation]]. To confirm the model's performance, an additional test data set held out from cross-validation is normally used.

==Causes of error==
[[File:Computer definition of extra hot.png|thumb|260px|Comic strip demonstrating a fictional erroneous computer output (making a coffee 5 million degrees [Celsius or Fahrenheit not specified], from a previous definition of "extra hot"). This can be classified as both a failure in logic and a failure to include various relevant environmental conditions.<ref name=ChandaBanerjee2022/>]]
Omissions in the training of algorithms are a major cause of erroneous outputs.<ref name=ChandaBanerjee2022>{{cite journal| author=Chanda SS, Banerjee DN| title=Omission and commission errors underlying AI failures. | journal=AI Soc | year= 2022 | volume=  | issue=  | pages= 1-24 | pmid=36415822 | doi=10.1007/s00146-022-01585-x | pmc=9669536 | url=https://www.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&tool=sumsearch.org/cite&retmode=ref&cmd=prlinks&id=36415822  }} </ref> Types of such omissions include:<ref name=ChandaBanerjee2022/>
*Particular circumstances or variations were not included. 
*Obsolete data
*Ambiguous input information
*Inability to change to new environments
*Inability to request help from a human or another AI system when needed
An example of an omission of particular circumstances is a case where a boy was able to unlock the phone because his mother registered her face under indoor, nighttime lighting, a condition which was not appropriately included in the training of the system.<ref name=ChandaBanerjee2022/><ref>{{cite web|author=Greenberg A|date=2017-11-14|url=https://www.wired.com/story/10-year-old-face-id-unlocks-mothers-iphone-x/?mbid=social_fb|title=Watch a 10-Year-Old's Face Unlock His Mom's iPhone X|website=Wired}}</ref> 

Usage of relatively irrelevant input can include situations where algorithms use the background rather than the object of interest for [[Object detection|object detection]], such as being trained by pictures of sheep on grasslands, leading to a risk that a different object will be interpreted as a sheep if located on a grassland.<ref name=ChandaBanerjee2022/>

== See also ==
* [[Statistical classification]]
* [[List of datasets for machine learning research]]
* [[Hierarchical classification]]

== References ==
{{Reflist}}

{{Differentiable computing}}
[[Category:Datasets in machine learning]]
[[Category:Validity (statistics)]]

{{Sourceattribution|Training, validation, and test data sets}}