Cluster hypothesis

In machine learning and information retrieval, the cluster hypothesis is an assumption about the nature of the data handled in those fields, which takes various forms. In information retrieval, it states that documents that are clustered together "behave similarly with respect to relevance to information needs".^[1] In terms of classification, it states that if points are in the same cluster, they are likely to be of the same class.^[2] There may be multiple clusters forming a single class.

Information retrieval

The cluster hypothesis was formulated first by van Rijsbergen:^[3] "closely associated documents tend to be relevant to the same requests". Thus, theoretically, a search engine could try to locate only the appropriate cluster for a query, and then allow users to browse through this cluster. Although experiments showed that the cluster hypothesis as such holds, exploiting it for retrieval did not lead to satisfying results.^[4]

Machine learning

The cluster assumption is assumed in many machine learning algorithms such as the k-nearest neighbor classification algorithm and the k-means clustering algorithm. As the word "likely" appears in the definition, there is no clear border differentiating whether the assumption does hold or does not hold. In contrast the amount of adherence of data to this assumption can be quantitatively measured.

Properties

The cluster assumption is equivalent to the Low density separation assumption which states that the decision boundary should lie on a low-density region. To prove this, suppose the decision boundary crosses one of the clusters. Then this cluster will contain points from two different classes, therefore it is violated on this cluster.

Notes

^ Manning, Christopher (2008). "16. Flat clustering". Introduction to information retrieval. New York: Cambridge University Press. ISBN 0-521-86571-9. OCLC 190786122.
^ Chapelle, Olivier; Scholkopf, Bernhard; Zien, Alexander, eds. (2006-09-22). Semi-Supervised Learning. The MIT Press. doi:10.7551/mitpress/9780262033589.001.0001. ISBN 978-0-262-03358-9.
^ van Rijsbergen, C. J. (1979). Information Retrieval (PDF) (2nd ed.). Butterworths. p. 30 ff. Retrieved 11 March 2022.
^ Voorhees, Ellen M. (1985). The cluster hypothesis revisited.

[irbook-1] Manning, Christopher (2008). "16. Flat clustering". Introduction to information retrieval. New York: Cambridge University Press. ISBN 0-521-86571-9. OCLC 190786122.

[Semi-Supervised_Learning_p.-2] Chapelle, Olivier; Scholkopf, Bernhard; Zien, Alexander, eds. (2006-09-22). Semi-Supervised Learning. The MIT Press. doi:10.7551/mitpress/9780262033589.001.0001. ISBN 978-0-262-03358-9.

[Rijsbergen79-3] van Rijsbergen, C. J. (1979). Information Retrieval (PDF) (2nd ed.). Butterworths. p. 30 ff. Retrieved 11 March 2022.

[4] Voorhees, Ellen M. (1985). The cluster hypothesis revisited.

[1]

[2]

[3]

[4]