Truth discovery
Truth discovery (also known as truth finding) is the process of choosing the actual true value for a data item when different data sources provide conflicting information on it.
Several algorithms have been proposed to tackle this problem, ranging from simple methods like majority voting to more complex ones able to estimate the trustworthiness of data sources.[1]
Truth discovery problems can be divided into two sub-classes: single-truth and multi-truth. In the first case only one true value is allowed for a data item (e.g birthday of a person, capital city of a country). While in the second case multiple true values are allowed (e.g. cast of a movie, authors of a book).[2][3]
Typically, truth discovery is the last step of a data integration pipeline, when the schemas of different data sources have been unified and the records referring to the same data item have been detected.[4]
General principles
The abundance of data available on the web makes more and more probable to find that different sources provide (partially or completely) different values for the same data item. This, together with the fact that we are increasing our reliance on data to derive important decisions, motivates the need of developing good truth discovery algorithms.[5]
Many currently available methods rely on a voting strategy to define the true value of a data item. Nevertheless, recent studies, have shown that, if we rely only on majority voting, we could get wrong results even in 30% of the data items.[5]
The solution to this problem is to assess the trustworthiness of the sources and give more importance to votes coming from trusted sources.[4][5]
Ideally, supervised learning techniques could be exploited to assign a reliability score to sources after hand-crafted labeling of the provided values; unfortunately, this is not feasible since the number of needed labeled examples should be proportional to the number of sources, and in many applications the number of sources can be prohibitive.[2][6]
Single-truth vs multi-truth discovery
Single-truth and multi-truth discovery are two very different problems.[2]
Single-truth discovery is characterized by the following properties:
- only one true value is allowed for each data item;
- different values provided for a given data item oppose to each other;
- values and sources can either be correct or erroneous.
While in the multi-truth case the following properties hold:
- the truth is composed by a set of values;
- different values could provide a partial truth;
- claiming one value for a given data item does not imply opposing to all the other values;
- the number of true values for each data item is not known a priori.
Multi-truth discovery has unique features that make the problem more complex and should be taken into consideration when developing truth-discovery solutions.[2]
The examples below point out the main differences of the two methods. Knowing that in both examples the truth is provided by source 1, in the single truth case (first table) we can say that sources 2 and 3 oppose to the truth and as a result provide wrong values. On the other hand, in the second case (second table), sources 2 and 3 are neither correct nor erroneous, they instead provide a subset of the true values and at the same time they do not oppose the truth.
| Source | Name | Birthdate | |
|---|---|---|---|
| S1 | George Washington | 1732-02-22 | Correct |
| S2 | George Washington | 1738-09-17 | Erroneous |
| S3 | George Washington | 1734-10-23 | Erroneous |
| Source | Title | Authors | |
|---|---|---|---|
| S1 | The nature of space and time | Stephen Hawking, Roger Penrose | Correct |
| S2 | The nature of space and time | Stephen Hawking | Partial truth |
| S3 | The nature of space and time | Roger Penrose | Partial truth |
| S4 | The nature of space and time | J. K. Rowling | Erroneous |
Source trustworthiness
The vast majority of truth discovery methods are based on a voting approach: each source votes for a value of a certain data item and, at the end, the value with the highest vote is select as the true one. In the more sophisticated methods, votes do not have the same weight for all the data sources, more importance is indeed given to votes coming from trusted sources.[5]
Source trustworthiness usually is not known a priori but estimated with an iterative approach. At each step of the truth discovery algorithm the trustworthiness score of each data source is refined, improving the assessment of the true values that in turn leads to a better estimation of the trustworthiness of the sources. This process usually ends when all the values reach a convergence state.[5]
Source trustworthiness can be based on different metrics, such as accuracy of provided values, copying values from other sources and domain coverage.[1]
Detecting copying behaviors is very important, in fact, copy allows to spread false values easily making truth discovery very hard, since many sources would vote for the wrong values. Usually systems decrease the weight of votes associated to copied values or even don’t count them at all.[7]
Single-truth methods
Most of the currently available truth discovery methods have been designed to work well only in the single-truth case.[1][3]
Below are reported some of the characteristics of the most relevant typologies of single-truth methods and how different systems model source trustworthiness.[5]
Majority voting
Majority voting is the simplest method, the most popular value is selected as the true one. Majority voting is commonly used as a baseline when assessing the performances of more complex methods.
Web-link based
These methods estimate source trustworthiness exploiting a similar technique to the one used to measure authority of web pages based on web links. The vote assigned to a value is computed as the sum of the trustworthiness of the sources that provide that particular value, while the trustworthiness of a source is computed as the sum of the votes assigned to the values that the source provides.[5][8]
Information-retrieval based
These methods estimate source trustworthiness using similarity measures typically used in information retrieval. Source trustworthiness is computed as the cosine similarity (or other similarity measures) between the set of values provided by the source and the set of values considered true (either selected in a probabilistic way or obtained from a ground truth).[5][9]
Bayesian based
These methods use Bayesian inference to define the probability of a value being true conditioned on the values provided by all the sources.
[math]\displaystyle{ P(v \mid \psi(o)) = \frac{P(\psi(o) \mid v) \cdot P(v)}{P(\psi(o))} }[/math]
where [math]\displaystyle{ \textstyle v }[/math] is a value provided for a data item [math]\displaystyle{ \textstyle o }[/math] and [math]\displaystyle{ \textstyle \psi(o) }[/math] is the set of the observed values provided by all the sources for that specific data item.
The trustworthiness of a source is then computed based on the accuracy of the values that provides.[7][10] Other more complex methods exploit Bayesian inference to detect copying behaviors and use these insights to better assess source trustworthiness.[7]
Multi-truth methods
Due to its complexity, less attention has been devoted to the study of the multi-truth discovery[2][3]
Below are reported two typologies of multi-truth methods and their characteristics.
Bayesian based
These methods use Bayesian inference to define the probability of a group of values being true conditioned on the values provided by all the data sources. In this case, since there could be multiple true values for each data item, and sources can provide multiple values for a single data item, it is not possible to consider values individually. An alternative is to consider mappings and relations between set of provided values and sources providing them. The trustworthiness of a source is then computed based on the accuracy of the values that provides.[2]
More sophisticated methods also consider domain coverage and copying behaviors to better estimate source trustworthiness.[2][3]
Probabilistic Graphical Models based
These methods use probabilistic graphical models to automatically define the set of true values of given data item and also to assess source quality without need of any supervision.[11]
Applications
Many real-world applications can benefit from the use of truth discovery algorithms. Typical domains of application include: healthcare, crowd/social sensing, crowdsourcing aggregation, information extraction and knowledge base construction.[1]
Truth discovery algorithms could be also used to revolutionize the way in which web pages are ranked in search engines, going from current methods based on link analysis like PageRank, to procedures that rank web pages based on the accuracy of the information they provide.[12]
See also
- Data Integration
- Information Integration
- Data Fusion (data integration)
- Data Quality
References
- ↑ 1.0 1.1 1.2 1.3 Li, Yaliang; Gao, Jing; Meng, Chuishi; Li, Qi; Su, Lu; Zhao, Bo; Fan, Wei; Han, Jiawei (2016-02-25). "A Survey on Truth Discovery" (in en). ACM SIGKDD Explorations Newsletter 17 (2): 1–16. doi:10.1145/2897350.2897352.
- ↑ 2.0 2.1 2.2 2.3 2.4 2.5 2.6 Wang, Xianzhi; Sheng, Quan Z.; Fang, Xiu Susie; Yao, Lina; Xu, Xiaofei; Li, Xue (2015). "An Integrated Bayesian Approach for Effective Multi-Truth Discovery" (in en). Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. Melbourne, Australia: ACM Press. pp. 493–502. doi:10.1145/2806416.2806443. ISBN 9781450337946. http://dl.acm.org/citation.cfm?doid=2806416.2806443.
- ↑ 3.0 3.1 3.2 3.3 Lin, Xueling; Chen, Lei (2018). "Domain-aware Multi-truth Discovery from Conflicting Sources". VLDB Endowment 11 (5): 635–647. doi:10.1145/3187009.3177739.
- ↑ 4.0 4.1 Dong, Xin Luna; Srivastava, Divesh (2015-02-15). "Big Data Integration" (in en). Synthesis Lectures on Data Management 7 (1): 1–198. doi:10.2200/S00578ED1V01Y201404DTM040. ISSN 2153-5418.
- ↑ 5.0 5.1 5.2 5.3 5.4 5.5 5.6 5.7 Li, Xian; Dong, Xin Luna; Lyons, Kenneth; Meng, Weiyi; Srivastava, Divesh (2012-12-01). "Truth finding on the deep web: is the problem solved?" (in en). Proceedings of the VLDB Endowment 6 (2): 97–108. doi:10.14778/2535568.2448943.
- ↑ Ng, Andrew Y; Jordan, Michael I. (2001). "On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes". Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic: 841–848. http://dl.acm.org/citation.cfm?id=2980539.2980648.
- ↑ 7.0 7.1 7.2 Dong, Xin Luna; Berti-Equille, Laure; Srivastava, Divesh (2009-08-01). "Integrating conflicting data: the role of source dependence" (in en). Proceedings of the VLDB Endowment 2 (1): 550–561. doi:10.14778/1687627.1687690.
- ↑ Kleinberg, Jon M. (1999-09-01). "Authoritative sources in a hyperlinked environment". Journal of the ACM 46 (5): 604–632. doi:10.1145/324133.324140.
- ↑ Galland, Alban; Abiteboul, Serge; Marian, Amélie; Senellart, Pierre (2010). "Corroborating information from disagreeing views" (in en). Proceedings of the third ACM international conference on Web search and data mining. New York, New York, USA: ACM Press. pp. 131–140. doi:10.1145/1718487.1718504. ISBN 9781605588896. http://portal.acm.org/citation.cfm?doid=1718487.1718504.
- ↑ Xiaoxin Yin; Jiawei Han; Yu, P.S. (2008). "Truth Discovery with Multiple Conflicting Information Providers on the Web". IEEE Transactions on Knowledge and Data Engineering 20 (6): 796–808. doi:10.1109/TKDE.2007.190745. ISSN 1041-4347.
- ↑ Zhao, Bo; Rubinstein, Benjamin I. P.; Gemmell, Jim; Han, Jiawei (2012-02-01). "A Bayesian approach to discovering truth from conflicting sources for data integration" (in en). Proceedings of the VLDB Endowment 5 (6): 550–561. doi:10.14778/2168651.2168656.
- ↑ "The huge implications of Google's idea to rank sites based on their accuracy". 2015. https://www.washingtonpost.com/news/energy-environment/wp/2015/03/11/the-huge-implications-of-googles-idea-to-rank-sites-based-on-their-accuracy/.
 |  |
