Software:YaCy

From HandWiki
YaCy
YaCy logo.png
Yacy-buscador.png
Original author(s)Michael Christen
Developer(s)YaCy community
Initial release2003; 21 years ago (2003)[1]
Stable release
1.924 / 14 December 2020; 3 years ago (2020-12-14)
Repositorygithub.com/yacy/yacy_search_server
Written inJava
Operating systemCross-platform
TypeOverlay network, Search engine
LicenseGPL-2.0-or-later
Websiteyacy.net/en/

YaCy (pronounced “ya see”) is a free distributed search engine, built on the principles of peer-to-peer (P2P) networks created by Michael Christen in 2003.[2][3] The engine is written in Java and distributed on several hundred computers, (As of September 2006), so-called YaCy-peers. Each YaCy-peer independently crawls through the Internet, analyzes and indexes found web pages, and stores indexing results in a common database (so-called index) which is shared with other YaCy-peers using principles of peer-to-peer. It is a search engine that everyone can use to build a search portal for their intranet and to help search the public internet.

Compared to semi-distributed search engines, the YaCy network has a distributed architecture. All YaCy-peers are equal and no central server exists. It can be run either in a crawling mode or as a local proxy server, indexing web pages visited by the person running YaCy on their computer. Several mechanisms are provided to protect the user's privacy. Access to the search functions is made by a locally run web server which provides a search box to enter search terms, and returns search results in a similar format to other popular search engines.

System components

YaCy search engine is based on four elements:[4]

Crawler
A search robot that traverses between web pages, analyzing their content.[5]
Indexer
It creates a reverse word index (RWI), i.e., each word from the RWI has its own list of relevant URLs and ranking information. Words are saved in the form of word hashes.[6]
Search and administration interface
Made as a web interface provided by a local HTTP servlet with a servlet engine.[7]
Data storage
Used to store the reverse word index database utilizing a distributed hash table.
Homepage of YaCy

Search-engine technology

YaCy network
  • YaCy is a complete search appliance with user interface, index, administration, and monitoring.
  • YaCy harvests web pages with a web crawler. Documents are then parsed, and indexed and the search index is stored locally. If your peer is part of a peer network, then your local search index is also merged into the shared index for that network.
  • A search is started, then the local index contributes with a global search index from peers in the YaCy search network.
  • The YaCy Grid is a second-generation implementation of the YaCy peer-to-peer search. A YaCy Grid installation comprises microservices that communicate using the Master Connect Program (MCP).
  • The YaCy Parser is a microservice that can be deployed using Docker. When the Parser Component is started, it searches for and connects to an MCP. By default, the local host is searched for an MCP, but you can configure one yourself.

YaCy platform architecture

Web search showing results of the different components YaCy uses

YaCy uses a combination of techniques for the networking, administration, and maintenance of indexing the search engine, including blacklisting, moderation, and communication with the community. Here is how YaCy performs these operations:

  • Community components
    1. Web forum[8]
    2. Statistics
    3. XML API
  • Maintenance
    1. Web Server
    2. Indexing
    3. Crawler with Balancer
    4. Peer-to-Peer Server Communication
  • Content organization
    1. Blacklisting and Filtering
    2. Search interface
    3. Bookmarks
    4. Monitoring search results

Distribution

YaCy is available in packages for Linux, Windows, and Macintosh, and also as a Docker image. YaCy can also be installed on other operating systems either by manually compiling it, or using a tarball.[9] YaCy requires Java 8, OpenJDK 8 is recommended.

The Debian package can be installed from a repository available at the subdomain of the project's website.[10][11] The package is not maintained in the official Debian package repository yet.[12]

See also

  • Dooble – an open-source web browser with an integrated YaCy Search Engine Tool Widget

References

  1. ↑ "Ich entwickle eine P2P-basierende Suchmaschine. Wer macht mit?" (in de). Heise Online. 2003-12-15. http://www.heise.de/newsticker/foren/go.shtml?read=1&msg_id=4744034&forum_id=50682. 
  2. ↑ "YaCy takes on Google with open source search engine". The Register. 2011-11-29. https://www.theregister.co.uk/2011/11/29/yacy_google_open_source_engine/. 
  3. ↑ "YaCy: It's About Freedom, Not Beating Google". PC World. 2011-12-03. https://www.pcworld.com/article/245414/yacy_its_about_freedom_not_beating_google.html. 
  4. ↑ "YaCy Technology Architecture". YaCy.net. http://yacy.net/Technology.html. 
  5. ↑ GitHub: YaCy Grid Crawler, YaCy Search Engine, 2021-02-28, pp. yacy / yacy_grid_crawler, https://github.com/yacy/yacy_grid_crawler, retrieved 2021-03-11 
  6. ↑ GitHub: YaCy Grid Parser, YaCy Search Engine, 2021-02-28, pp. The YaCy Grid is the second-generation implementation of YaCy, https://github.com/yacy/yacy_grid_parser, retrieved 2021-03-11 
  7. ↑ GitHub: YaCY Search, YaCy Search Engine, 2021-02-28, pp. yacy / yacy-search forked from cream/yacy-search, https://github.com/yacy/yacy-search, retrieved 2021-03-11 
  8. ↑ "forum.yacy.de". http://forum.yacy-websuche.de/. 
  9. ↑ "Download - YaCy". https://yacy.net/download_installation/#compile-yacy-yourself. 
  10. ↑ "En:DebianInstall". https://wiki.yacy.net/index.php/En:DebianInstall. 
  11. ↑ "Dev:TaskSharing". https://wiki.yacy.net/index.php/Dev:TaskSharing. 
  12. ↑ "#452422 - RFP: yacy -- distributed web crawler and search engine". https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=452422. 


Further reading

YaCy at LinuxReviews

External links



Retrieved from "https://handwiki.org/wiki/index.php?title=Software:YaCy&oldid=3464868"