Jump to content

Web scraping: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous editNext edit →
Content deleted Content added
VisualWikitext
m Reverted edits by Baklama to last revision by Shadowjams (HG)
Baklama (talk | contribs)
4 edits
Undid revision 315647905 by Shadowjams (talk)
Line 12: Line 12:
* Web-scraping software: There are many Web-scraping software available that can be used to customize Web-scraping solutions. These software may provide a Web recording interface that removes the necessity to manually write Web-scraping codes, or some scripting functions that can be used to extract and transform Web content, and database interfaces that can store the scraped data in local databases.
* Web-scraping software: There are many Web-scraping software available that can be used to customize Web-scraping solutions. These software may provide a Web recording interface that removes the necessity to manually write Web-scraping codes, or some scripting functions that can be used to extract and transform Web content, and database interfaces that can store the scraped data in local databases.
* Semantic annotation recognizing: The Web pages may embrace metadata or semantic markups/annotations which can be made use of to locate specific data snippets. If the annotations are embedded in the pages, as [[Microformat]] does, this technique can be viewed as a special case of DOM parsing. In another case, the annotations, organized into a semantic layer<ref>[http://www.gooseeker.com/en/node/knowledgebase/freeformat Semantic annotation based web scraping<!-- Bot generated title -->]</ref>, are stored and managed separated to the Web pages, so the Web scrapers can retrieve data schema and instructions from this layer before scraping the pages.
* Semantic annotation recognizing: The Web pages may embrace metadata or semantic markups/annotations which can be made use of to locate specific data snippets. If the annotations are embedded in the pages, as [[Microformat]] does, this technique can be viewed as a special case of DOM parsing. In another case, the annotations, organized into a semantic layer<ref>[http://www.gooseeker.com/en/node/knowledgebase/freeformat Semantic annotation based web scraping<!-- Bot generated title -->]</ref>, are stored and managed separated to the Web pages, so the Web scrapers can retrieve data schema and instructions from this layer before scraping the pages.

== Services ==

* [[Web-scraping Service Comparison]]
* [http://www.web-extraction.com/ WebExtraction]


==Legal issues==
==Legal issues==

Web scraping may be against the [[terms of use]] of some websites. The enforceability of these terms is unclear.<ref>{{cite web
Web scraping may be against the [[terms of use]] of some websites. The enforceability of these terms is unclear.<ref>{{cite web
|url=http://www.chillingeffects.org/linking/faq.cgi#QID596
|url=http://www.chillingeffects.org/linking/faq.cgi#QID596

Revision as of 04:49, 23 September 2009

Web scraping (or Web harvesting, Web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding certain full-fledged Web browsers, such as the Internet Explorer (IE) and the Mozilla Web browser. Web scraping is closely related to Web indexing, which indexes Web content using a bot and is a universal technique adopted by most search engines. In contrast, Web scraping focuses more on the transformation of unstructured Web content, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to Web automation, which simulates human Web browsing using computer software. Exemplary uses of Web scraping include online price comparison, weather data monitoring, website change detection, Web research, Web content mashup and Web data integration.

Techniques for Web scraping

Web scraping is still a field with active developments. The process of automatically collecting Web information shares a common goal with the semantic Web vision, which is a more ambitious initiative that still requires breakthroughs in text processing, semantic understanding, artificial intelligence and human-computer interactions. Web scraping, instead, favors practical solutions based on existing technologies even though some solutions are entirely ad hoc. Therefore, there are different levels of automations that existing Web-scraping technologies can provide:

  • Human copy-and-paste: Sometimes even the best Web-scraping technology can not replace human’s manual examination and copy-and-paste, and sometimes this may be the only workable solution when the websites for scraping explicitly setup barriers to prevent machine automation.
  • Text grepping and regular expression matching: A simple yet powerful approach to extract information from Web pages can be based on the unix grep command or regular expression matching facilities of programming languages (for instance Perl or Python).
  • HTTP programming: Static and dynamic Web pages can be retrieved by posting HTTP requests to the remote Web server using socket programming.
  • DOM parsing: By embedding a full-fledged Web browser, such as the Internet Explorer or the Mozilla Web browser control, programs can retrieve the dynamic contents generated by client side scripts. These Web browser controls also parse Web pages into a DOM tree, based on which programs can retrieve parts of the Web pages.
  • HTML parsers: Some semi-structured data query languages, such as the XML query language (XQL) and the hyper-text query language (HTQL), can be used to parse HTML pages and to retrieve and transform Web content.
  • Web-scraping software: There are many Web-scraping software available that can be used to customize Web-scraping solutions. These software may provide a Web recording interface that removes the necessity to manually write Web-scraping codes, or some scripting functions that can be used to extract and transform Web content, and database interfaces that can store the scraped data in local databases.
  • Semantic annotation recognizing: The Web pages may embrace metadata or semantic markups/annotations which can be made use of to locate specific data snippets. If the annotations are embedded in the pages, as Microformat does, this technique can be viewed as a special case of DOM parsing. In another case, the annotations, organized into a semantic layer[1], are stored and managed separated to the Web pages, so the Web scrapers can retrieve data schema and instructions from this layer before scraping the pages.

Services

Web scraping may be against the terms of use of some websites. The enforceability of these terms is unclear.[2] While outright duplication of original expression will in many cases be illegal, in the US the courts ruled in Feist Publications v. Rural Telephone Service that duplication of facts is allowable. Also, in a February, 2006 ruling, the Danish Maritime and Commercial Court (Copenhagen) found systematic crawling, indexing and deep linking by portal site ofir.dk of real estate site Home.dk not to conflict with Danish law or the database directive of the European Union.[3]

U.S. courts have acknowledged that users of "scrapers" or "robots" may be held liable for committing trespass to chattels,[4][5] which involves a computer system itself being considered personal property upon which the user of a scraper is trespassing. However, to succeed on a claim of trespass to chattels, the plaintiff must demonstrate that the defendant intentionally and without authorization interfered with the plaintiff's possessory interest in the computer system and that the defendant's unauthorized use caused damage to the plaintiff. Not all cases of web spidering brought before the courts have been considered trespass to chattels.[6]

In Australia, the Spam Act 2003 outlaws some forms of web harvesting.[7][8]

Technical measures to stop bots

A web master can use various measures to stop or slow a bot. Some techniques include:

  • If the application is well behaved, adding entries to robots.txt will be adhered to. You can stop Google and other well-behaved bots this way.
  • Blocking an IP address. This will also block all browsing from that address.
  • Sometimes bots declare who they are. Well behaved ones do (for example 'googlebot'). They can be blocked on that basis. Unfortunately, malicious bots may declare they are a normal browser.
  • Bots can be blocked by excess traffic monitoring.
  • Bots can be blocked with tools to verify that it is a real person accessing the site, such as the CAPTCHA project.
  • Sometimes bots can be blocked with carefully crafted Javascript.
  • Locating bots with a honeypot or other method to identify the IP addresses of automated crawlers.

Notes

  1. ^ Semantic annotation based web scraping
  2. ^ "FAQ about linking - Are website terms of use binding contracts?". www.chillingeffects.org. 2007-08-20. Retrieved 2007-08-20. {{cite web}}: Cite has empty unknown parameters: |accessdaymonth=, |month=, |accessyear=, |accessmonthday=, and |coauthors= (help)
  3. ^ "UDSKRIFT AF SØ- & HANDELSRETTENS DOMBOG" (PDF). bvhd.dk. 2006-02-24. Retrieved 2007-05-30. {{cite web}}: Cite has empty unknown parameters: |accessdaymonth=, |month=, |accessyear=, |accessmonthday=, and |coauthors= (help)
  4. ^ "Internet Law, Ch. 06: Trespass to Chattels". www.tomwbell.com. 2007-08-20. Retrieved 2007-08-20. {{cite web}}: Cite has empty unknown parameters: |accessdaymonth=, |month=, |accessyear=, |accessmonthday=, and |coauthors= (help)
  5. ^ "What are the "trespass to chattels" claims some companies or website owners have brought?". www.chillingeffects.org. 2007-08-20. Retrieved 2007-08-20. {{cite web}}: Cite has empty unknown parameters: |accessdaymonth=, |month=, |accessyear=, |accessmonthday=, and |coauthors= (help)
  6. ^ "Ticketmaster Corp. v. Tickets.com, Inc". 2007-08-20. Retrieved 2007-08-20. {{cite web}}: Cite has empty unknown parameters: |accessdaymonth=, |month=, |accessyear=, |accessmonthday=, and |coauthors= (help)
  7. ^ National Office for the Information Economy (2004). "Spam Act 2003: An overview for business" (PDF). Australian Communications Authority. p. 6. Retrieved 2009-03-09. {{cite web}}: Cite has empty unknown parameters: |accessdaymonth=, |accessyear=, |accessmonthday=, and |coauthors= (help); Unknown parameter |month= ignored (help)
  8. ^ National Office for the Information Economy (2004). "Spam Act 2003: A practical guide for business" (PDF). Australian Communications Authority. p. 20. Retrieved 2009-03-09. {{cite web}}: Cite has empty unknown parameters: |accessdaymonth=, |accessyear=, |accessmonthday=, and |coauthors= (help); Unknown parameter |month= ignored (help)

References

See also

Retrieved from "https://wiki.riteme.site/w/index.php?title=Web_scraping&oldid=315651213"