DMOZ

From HandWiki

DMOZ (from directory.mozilla.org, an earlier domain name) was a multilingual open-content directory of World Wide Web links. The site and community who maintained it were also known as the Open Directory Project (ODP). It was owned by AOL (now a part of Verizon Media) but constructed and maintained by a community of volunteer editors.

DMOZ used a hierarchical ontology scheme for organizing site listings. Listings on a similar topic were grouped into categories which then included smaller categories.

DMOZ closed on March 17, 2017, because AOL no longer wished to support the project.[1][2] The website became a single landing page on that day, with links to a static archive of DMOZ, and to the DMOZ discussion forum, where plans to rebrand and relaunch the directory are being discussed.[2]

(As of September 2017), a non-editable mirror remained available at dmoztools.net,[3] and it was announced that while the DMOZ URL would not return, a successor version of the directory named Curlie would be provided.[4][5] (As of 2020) Curlie.org is still online, serving this purpose.

History

DMOZ was founded in the United States as Gnuhoo by Rich Skrenta and Bob Truel in 1998 while they were both working as engineers for Sun Microsystems. Chris Tolles, who worked at Sun Microsystems as the head of marketing for network security products, also signed on in 1998 as a co-founder of Gnuhoo along with co-founders Bryn Dole and Jeremy Wenokur. Skrenta had developed TASS, an ancestor of tin, the popular threaded Usenet newsreader for Unix systems. The original category structure of the Gnuhoo directory was based loosely on the structure of Usenet newsgroups then in existence.

The Gnuhoo directory went live on June 5, 1998. After Richard Stallman and the Free Software Foundation objected to the use of Gnu in the name, Gnuhoo was changed to NewHoo.[6] Yahoo! then objected to the use of Hoo in the name, prompting a proposed name change to ZURL.[7] Prior to switching to ZURL, NewHoo was acquired by Netscape Communications Corporation in October 1998 and became the Open Directory Project. Netscape released Open Directory data under the Open Directory License. Netscape was acquired by AOL shortly thereafter and DMOZ was one of the assets included in the acquisition.

DMOZ size by date, 1998 to 2015.

By the time Netscape assumed stewardship, the Open Directory Project had about 100,000 URLs indexed with contributions from about 4500 editors. On October 5, 1999, the number of URLs indexed by DMOZ reached one million. According to an unofficial estimate, the URLs in DMOZ numbered 1.6 million in April 2000, surpassing those in the Yahoo! Directory.[8] DMOZ achieved the milestones of indexing two million URLs on August 14, 2000, three million listings on November 18, 2001 and four million on December 3, 2003. As of April 2013 there were 5,169,995 sites listed in over 1,017,500 categories. On October 31, 2015, there were 3,996,412 sites listed in 1,026,706 categories.

In January 2006, DMOZ began publishing online reports to inform the public about the development of the project. The first report covered the year 2005. Monthly reports were issued subsequently until September 2006.[9] These reports gave greater insight into the functioning of the directory than the simplified statistics provided on the front page of the directory. The number of listings and categories cited on the front page included "Test" and "Bookmarks" categories but these were not included in the RDF dump offered to users. There were about 7330 active editors during August 2006.[9] 75,151 editors had contributed to the directory as of March 31, 2007.[10] As of April 2013, the number of contributing editors had increased to 97,584.[10]

System failure and editing outage, October to December 2006

On October 20, 2006, DMOZ's main server suffered a catastrophic failure[11] that prevented editors from working on the directory until December 18, 2006. During that period, an older build of the directory was visible to the public. On January 13, 2007, the Site Suggestion and Update Listings forms were again made available.[12] On January 26, 2007, weekly publication of RDF dumps resumed. To avoid future outages, the system resided on a redundant configuration of two Intel-based servers from then on.[13]

The site's interface was given an upgrade in 2016, branded "DMOZ 3.0", but AOL took it offline the following year.

Competing and spinoff projects

As DMOZ became more widely known, two other major web directories edited by volunteers and sponsored by Go.com and Zeal emerged, both now defunct. These directories did not license their content for open content distribution.[14][15]

The concept of using a large-scale community of editors to compile online content has been successfully applied to other types of projects. DMOZ's editing model directly inspired at least three other open content volunteer projects: music site MusicMoz, an open content restaurant directory known as ChefMoz[16] and an encyclopedia known as Open Site.[17] Finally, according to Larry Sanger, DMOZ was part of the inspiration for the Nupedia project, out of which Wikipedia grew.[18]

Logo history

Content

Gnuhoo borrowed the basic outline for its initial ontology from Usenet. In 1998, Rich Skrenta said, "I took a long list of groups and hand-edited them into a hierarchy."[19] For example, the topic covered by the comp.ai.alife newsgroup was represented by the category Computers/AI/Artificial_Life. The original divisions were for Adult, Arts, Business, Computers, Games, Health, Home, News, Recreation, Reference, Regional, Science, Shopping, Society, Sports and "World". While these sixteen top-level categories have remained intact, the ontology of second- and lower-level categories has undergone a gradual evolution; significant changes are initiated by discussion among editors and then implemented when consensus has been reached.

In July 1998, the directory became multilingual with the addition of the World top-level category. The remainder of the directory lists only English language sites. By May 2005, seventy-five languages were represented. The growth rate of the non-English components of the directory has been greater than the English component since 2002. While the English component of the directory held almost 75% of the sites in 2003, the World level grew to over 1.5 million sites as of May 2005, forming roughly one-third of the directory. The ontology in non-English categories generally mirrors that of the English directory, although exceptions which reflect language differences are quite common.

Several of the top-level categories have unique characteristics. The Adult category is not present on the directory homepage but it is fully available in the RDF dump that DMOZ provides. While the bulk of the directory is categorized primarily by topic, the Regional category is categorized primarily by region. This has led many to view DMOZ as two parallel directories: Regional and Topical.

On November 14, 2000, a special directory within DMOZ was created for people under 18 years of age.[20] Key factors distinguishing this "Kids and Teens" area from the main directory are:

  • stricter guidelines which limit the listing of sites to those which are targeted or "appropriate" for people under 18 years of age;[21]
  • category names as well as site descriptions use vocabulary which is "age appropriate";
  • age tags on each listing distinguish content appropriate for kids (age 12 and under), teens (13 to 15 years old) and mature teens (16 to 18 years old);
  • Kids and Teens content is available as a separate RDF dump;
  • editing permissions are such that the community is parallel to that of DMOZ.

By May 2005, this portion of DMOZ included over 32,000 site listings.

Since early 2004, the whole site has been in UTF-8 encoding. Prior to this, the encoding used to be ISO 8859-1 for English language categories and a language-dependent character set for other languages. The RDF dumps have been encoded in UTF-8 since early 2000.

Maintenance

Directory listings are maintained by editors. While some editors focus on the addition of new listings, others focus on maintaining the existing listings and some do both. This includes tasks such as the editing of individual listings to correct spelling and/or grammatical errors, as well as monitoring the status of linked sites. Still others go through site submissions to remove spam and duplicate submissions.

Robozilla is a Web crawler written to check the status of all sites listed in DMOZ. Periodically, Robozilla will flag sites which appear to have moved or disappeared and editors follow up to check the sites and take action. This process is critical for the directory in striving to achieve one of its founding goals: to reduce the link rot in web directories. Shortly after each run, the sites marked with errors are automatically moved to the unreviewed queue where editors may investigate them when time permits.

Due to the popularity of DMOZ and its resulting impact on search engine rankings (See PageRank), domains with lapsed registration that are listed on DMOZ have attracted domain hijacking, an issue that has been addressed by regularly removing expired domains from the directory.

While corporate funding and staff for DMOZ have diminished in recent years, volunteers have created editing tools such as linkcheckers to supplement Robozilla, category crawlers, spellcheckers, search tools that directly sift a recent RDF dump, bookmarklets to help automate some editing functions, mozilla based add-ons,[22] and tools to help work through unreviewed queues.

License and requirements

DMOZ data was previously made available under the terms of the Open Directory License, which required a specific DMOZ attribution table on every Web page that uses the data.

The Open Directory License also included a requirement that users of the data continually check DMOZ site for updates and discontinue use and distribution of the data or works derived from the data once an update occurs. This restriction prompted the Free Software Foundation to refer to the Open Directory License as a non-free documentation license, citing the right to redistribute a given version not being permanent and the requirement to check for changes to the license.

In 2011, DMOZ silently changed its license to a Creative Commons Attribution license , which is a free license (and GPL compatible).[23]

RDF dumps

DMOZ data is made available through an RDF-like dump that is published on a download server, older versions are also archived there.[24] New versions are usually generated weekly. A DMOZ editor has catalogued a number of bugs that are encountered in the DMOZ RDF dump, most importantly that the file format isn't RDF.[25] So while today the so-called RDF dump is valid XML, it is not valid RDF and as such, software to process the DMOZ RDF dump needs to be specifically written for DMOZ data.

Content users

DMOZ data powers the core directory services for many of the Web's largest search engines and portals, including Netscape Search, AOL Search, and Alexa. Google Directory used DMOZ information, until being shuttered in July 2011.[26]

Other uses are also made of DMOZ data. For example, in the spring of 2004 Overture announced a search service for third parties combining Yahoo! Directory search results with DMOZ titles, descriptions and category metadata. The search engine Gigablast announced on May 12, 2005 its searchable copy of DMOZ. The technology permits search of websites listed in specific categories, "in effect, instantly creating over 500,000 vertical search engines".[27]

(As of September 2006), DMOZ listed 313 English-language Web sites that use DMOZ data as well as 238 sites in other languages.[28] However, these figures do not reflect the full picture of use, as those sites that use DMOZ data without following the terms of the DMOZ license are not listed.

Policies and procedures

DMOZ was co-founded by Rich Skrenta (depicted in 2009, age 42).

Restrictions are imposed on who can become an DMOZ editor. The primary gatekeeping mechanism is an editor application process wherein editor candidates demonstrate their editing abilities, disclose affiliations that might pose a conflict of interest , and otherwise give a sense of how the applicant would likely mesh with the DMOZ culture and mission.[29] A majority of applications are rejected but reapplying is allowed and sometimes encouraged. The same standards apply to editors of all categories and subcategories.{{citation needed|date=December 2

DMOZ has its own internal forums, the contents of which are intended only for editors to communicate with each other primarily about editing topics. Access to the forums requires an editor account and editors are expected to keep the contents of these forums private.[30]


In August 2004, a new level of privileges called admin was introduced. Administrator status was granted to a number of long serving metas by staff. Administrators have the ability to grant editall+ privileges to other editors and to approve new directory-wide policies, powers which had previously only been available to root (staff) editors.[31]

All DMOZ editors are expected to abide by DMOZ's Editing Guidelines. These guidelines describe editing basics: which types of sites may be listed and which may not; how site listings should be titled and described in a loosely consistent manner; conventions for the naming and building of categories; conflict of interest limitations on the editing of sites which the editor may own or otherwise be affiliated with; and a code of conduct within the community.[32] Editors who are found to have violated these guidelines may be contacted by staff or senior editors, have their editing permissions cut back, or lose their editing privileges entirely. DMOZ Guidelines are periodically revised after discussion in editor forums.

Controversy and criticism

There have long been allegations that volunteer DMOZ editors give favorable treatment to their own websites while concomitantly thwarting the good faith efforts of their competition.[33] Such allegations are fielded by ODP's staff and meta editors, who have the authority to take disciplinary action against volunteer editors who are suspected of engaging in abusive editing practices.[34] In 2003, DMOZ introduced a new Public Abuse Report System that allows members of the general public to report and track allegations of abusive editor conduct using an online form.[35] Uninhibited discussion of DMOZ's purported shortcomings has become more common on mainstream webmaster discussion forums. Although site policies suggest that an individual site should be submitted to only one category,[36] as of October 2007, Topix.com, a news aggregation site operated by DMOZ founder Rich Skrenta, had more than 17,000 listings.[37]

Early in the history of DMOZ, its staff gave representatives of selected companies, such as Rolling Stone or CNN, editing access in order to list individual pages from their websites.[38] Links to individual CNN articles were added until 2004, but were entirely removed from the directory in January 2008[39] due to the content being outdated and not considered worth the effort to maintain. There have been no similar experiments with the editing policy since then.

Ownership and management

Screenshot taken in April 2017 showing DMOZ website to be closed.

Underlying some controversy surrounding DMOZ is its ownership and management. Some of the original GnuHoo volunteers felt that they had been deceived into joining a commercial enterprise.[6] To varying degrees, those complaints have continued up until the present.

At DMOZ's inception, there was little thought given to the idea of how DMOZ should be managed and there were no official forums, guidelines or FAQs. In essence, DMOZ began as a free for all.[40]

As time went on, the ODP Editor Forums became the de facto DMOZ parliament and when one of DMOZ's staff members would post an opinion in the forums, it would be considered an official ruling.[30] Even so, DMOZ staff began to give trusted senior editors additional editing privileges, including the ability to approve new editor applications, which eventually led to a stratified hierarchy of duties and privileges among DMOZ editors, with DMOZ's paid staff having the final say regarding DMOZ's policies and procedures.[31][41]

Robert Keating, a principal of Touchstone Consulting Group in Washington, D.C. since 2006, has worked as AOL's Program Manager for DMOZ since 2004. He started working for AOL in 1999 as Senior Editor for AOL Search, then as Managing Editor, AOL Search, DMOZ, and then as Media Ecosystem Manager, AOL Product Marketing.[42][43]

Editor removal procedures

DMOZ's editor removal procedures are overseen by DMOZ's staff and meta-editors. According to DMOZ's official editorial guidelines, editors are removed for abusive editing practices or uncivil behaviour. Discussions that may result in disciplinary action against volunteer editors take place in a private forum which can only be accessed by DMOZ's staff and meta editors. Volunteer editors who are being discussed are not given notice that such proceedings are taking place.[41] Some people find this arrangement distasteful, wanting instead a discussion modeled more like a trial held in the U.S. judicial system.[44]

In the article "Editor Removal Explained", DMOZ meta editor Arlarson states that "a great deal of confusion about the removal of editors from DMOZ results from false or misleading statements by former editors".[45]

The DMOZ's confidentiality guidelines prohibit any current DMOZ editors in a position to know anything from discussing the reasons for specific editor removals.[41] However, a generic list of reasons is for example given in the guidelines.[46] In the past, this has led to removed DMOZ editors wondering why they cannot log in at DMOZ to perform their editing work.[47][48]

Allegations that editors are removed for criticizing policies

David F. Prenatt, Jr., former DMOZ editor netesq, and another former editor known by the alias The Cunctator, both claim to have been removed for disagreeing with staff about changes to policies, particularly DMOZ's copyright policies. According to their claims, staff use the excuse of uncivil behaviour as a means to remove bothersome editors.[44][49][50]

Blacklisting allegations

Senior DMOZ editors have the ability to attach "warning" or "do not list" notes to individual domains but no editor has the unilateral ability to block certain sites from being listed. Sites with these notes might still be listed and at times notes are removed after some discussion.[51]

Hierarchical structure

Criticism of DMOZ's hierarchical structure emerged by around 2005. Many believe hierarchical directories are too complicated. With the emergence of Web 2.0, folksonomies began to appear, and some editors proposed that folksonomies, networks and directed graphs are more "natural" and easier to manage than hierarchies.[52][53][54]

Software

Search

The ODPSearch software is a derivative version of Isearch which is open-source, licensed under the Mozilla Public License.[55]

Editor forums

The ODP Editor Forums were originally run on software that was based on the proprietary Ultimate Bulletin Board system. In June 2003, they switched to the open source phpBB system. As of 2007, these forums are powered by a modified version of phpBB.

Bug tracking

The bug tracking software used by the ODP is Bugzilla and the web server Apache. Squid web proxy server was also used but it was removed in August 2007 when the storage servers were reorganized. All these applications are open source.

Interface

The DMOZ database/editing software is closed source (although Richard Skrenta has said in June 1998 that he was considering licensing it under the GNU General Public License). This has led to criticism from the aforementioned GNU project, many of whom also criticized the DMOZ content license. The content was later released under a Creative Commons license, which is compatible with the GNU license.[56]

As such, there have been some efforts to provide alternatives to DMOZ. These alternatives would allow communities of like-minded editors to set up and maintain their own open source/open content Web directories.

See also

References

  1. Sullivan, Danny (March 17, 2017). "DMOZ has officially closed after nearly 19 years of humans trying to organize the web". Search Engine Land. http://searchengineland.com/dmoz-has-officially-closed-271530. Retrieved July 17, 2017. 
  2. 2.0 2.1 "Why Dmoz Was Closed ?". April 16, 2017. https://www.resource-zone.com/forum/t/why-dmoz-was-closed.53529/. 
  3. "The Directory of the Web – This site includes information formerly made available via DMOZ". https://dmoztools.net/. Retrieved August 21, 2017. 
  4. "New dmoz". https://www.resource-zone.com/forum/t/new-dmoz.53614/. Retrieved August 21, 2017. 
  5. "Curlie: Present". 2017-03-29. Archived from the original on September 18, 2017. https://web.archive.org/web/20170918065934/http://curlie.org/top/present.html. 
  6. 6.0 6.1 "The GnuHoo BooBoo". Slashdot. http://slashdot.org/article.pl?sid=98/06/23/0849239. Retrieved April 27, 2007. 
  7. "Zurl Directory"
  8. ODP and Yahoo Size Charts by ODP editor geniac
  9. 9.0 9.1 ODP reports by ODP volunteer administrator chris2001
  10. 10.0 10.1 ODP Front Page . Retrieved August 15, 2006
  11. "Dmoz's Catastrophic Server/Hardware Failure". dmozgrunt.blogspot.com. October 27, 2006. http://dmozgrunt.blogspot.com/2006/10/dmozs-catastrophic-serverhardware.html. Retrieved July 17, 2017. 
  12. dmoz.org technical problems archive.li Retrieved July 17, 2017.
  13. The Hamsters' New Home, in: Open Directory newsletter issue Winter 2006. Retrieved December 26, 2006.
  14. "Terms of Use"
  15. "GO Network Terms of Service and Conditions of Use"
  16. ChefMoz Fine Dining Menu , in: Open Directory newsletter issue Autumn 2003
  17. help on open-site.org
  18. Moody, Glyn (July 13, 2006). "This time, it'll be a Wikipedia written by experts". The Guardian. https://www.theguardian.com/technology/2006/jul/13/media.newmedia. Retrieved July 17, 2017. 
  19. Sullivan, Danny (July 1, 1998). "NewHoo: Yahoo Built By The Masses". Search Engine Watch. https://searchenginewatch.com/sew/study/2065949/the-search-engine-report-july-1998-number#newhoo. Retrieved July 17, 2017. 
  20. Kids and Teens Launches! Open Directory Project Newsletter, November/December 2000
  21. "Kids&Teens Guidelines". Dmoz.org. http://www.dmoz.org/guidelines/kguidelines/. Retrieved June 18, 2014. 
  22. "ODPExtension" Mozilla based add-on, ODP Magic. formerly known as ODP Extension
  23. GNU Project: on the Creative Commons Attribution license
  24. "Open Directory RDF Dump". Rdf.dmoz.org. Archived from the original on June 26, 2014. https://web.archive.org/web/20140626235229/http://rdf.dmoz.org/. Retrieved June 18, 2014. 
  25. "ODP/dmoz Data Dump ToDo List". steevithak.com. http://www.steevithak.com/sw/odp/rdflist.html. Retrieved July 17, 2017. 
  26. "Google Streamlining: Say Goodbye to the Google Directory and Labs!". Pandia Search Engine News. July 21, 2011. http://www.pandia.com/sew/3963-google-streamlining-say-goodbye-to-the-google-directory-and-labs.html. Retrieved July 25, 2011. 
  27. "Gigablast Launches 500,000 Vertical Search Engines". Gigablast. May 12, 2005. http://www.gigablast.com/prdir.html. Retrieved July 17, 2017. 
  28. Category: Sites Using ODP Data on www.dmoz.org. Retrieved September 8, 2006.
  29. "Become an Editor at the Open Directory Project". Dmoz.org. http://www.dmoz.org/cgi-bin/apply.cgi. Retrieved June 18, 2014. 
  30. 30.0 30.1 "ODP Communication Guidelines". Dmoz.org. http://www.dmoz.org/guidelines/communication.html. Retrieved June 18, 2014. 
  31. 31.0 31.1 "Open Directory Project Administrator Guidelines". Dmoz.org. http://www.dmoz.org/guidelines/admin/. Retrieved June 18, 2014. 
  32. "ODP Directory Editorial Guidelines". Dmoz.org. http://www.dmoz.org/guidelines/. Retrieved June 18, 2014. 
  33. "How To: ODP Editor Is Competitor". webmasterworld.com. November 4, 2000. https://www.webmasterworld.com/forum17/94.htm. Retrieved July 17, 2017. 
  34. ODP Meta Guidelines: Editor Abuse and Removal . Retrieved October 9, 2008.
  35. "Open Directory Project: Public Abuse Report System". Report-abuse.dmoz.org. Archived from the original on August 7, 2010. https://web.archive.org/web/20100807152404/http://report-abuse.dmoz.org/. Retrieved June 18, 2014. 
  36. "How to suggest a site to the Open Directory". Dmoz.org. http://www.dmoz.org/add.html. Retrieved June 18, 2014. 
  37. Open Directory Project Search: "topix" (Retrieved October 18, 2007)
  38. "Multiple URL's in DMOZ". webmasterworld.com. January 30, 2003. https://www.webmasterworld.com/forum17/1330.htm. Retrieved July 17, 2017. 
  39. "Wayback Machine for http://www.dmoz.org/News/". https://web.archive.org/web/*/http://www.dmoz.org/News/. 
  40. "The Open Directory Project: The Spirit of the Web". laisha.com. http://www.laisha.com/zine/odphistory.html. Retrieved July 17, 2017. 
  41. 41.0 41.1 41.2 "Open Directory Project Meta Guidelines". December 31, 2013. Archived from the original on March 4, 2017. https://web.archive.org/web/20170304211520/http://www.dmoz.org/guidelines/meta/. 
  42. Meet AOL's DMOZ Staff Team , DMOZ Blog, January 8, 2009
  43. Robert Keating, LinkedIn, http://www.linkedin.com/pub/robert-keating/2/457/424 
  44. 44.0 44.1 Prenatt, David (May 29, 2000). "Life After ODP". Yahoo! Groups. https://groups.yahoo.com/neo/groups/xodp/conversations/topics/1. 
  45. Arlarson, Editor Removal Explained , Open Directory Project Newsletter (September 2000).
  46. "Guidelines: Account Removal". July 3, 2016. Archived from the original on March 17, 2017. http://dmoztools.net/docs/en/guidelines/accounts.html#removal. 
  47. "Editor account expired". resource-zone.com. July 27, 2004. https://www.resource-zone.com/forum/t/editor-account-expired.18061/. 
  48. Thread: Can't Login on Resource-Zone
  49. Prenatt, Jr, David F. (June 1, 2000). "Life After the Open Directory Project". traffick.com. Archived from the original on May 25, 2017. https://web.archive.org/web/20170525200320/http://www.traffick.com/story/06-2000-xodp.asp. Retrieved July 17, 2017. 
  50. CmdrTaco (October 24, 2000). "Dmoz (aka AOL) Changing Guidelines In Sketchy Way". Slashdot. http://slashdot.org/articles/00/10/24/1252232.shtml. Retrieved July 17, 2017. 
  51. Add Note to URL Feature , in ODP Documentation
  52. Hriţcu, C. (April 8, 2005). "Folksonomies vs. Ontologies". hritcu.wordpress.com. http://hritcu.wordpress.com/2005/04/08/folksonomies-vs-ontologies/. Retrieved July 17, 2017. 
  53. "Ontology is Overrated"
  54. Hammond, Tony; Hannay, Timo (April 2005). "Social Bookmarking Tools (I)". D-Lib Magazine. http://www.dlib.org/dlib/april05/hammond/04hammond.html. Retrieved July 17, 2017. 
  55. "Open Directory Search Guide". Dmoz.org. http://www.dmoz.org/docs/en/searchguide.html. Retrieved June 18, 2014. 
  56. "Licenses for Works of Practical Use besides Software and Documentation". GNU Project. https://www.gnu.org/licenses/license-list.html#OtherLicenses. Retrieved July 17, 2017. 

External links

  • No URL found. Please specify a URL here or add one to Wikidata. (closed as of 2017-03-14)
  • DMOZ at Curlie
  • dmoztools.net DMOZ static mirror
  • Curlie The successor to DMOZ.