A study on the enhanced filtering method of the deduplication for bulk harvest of web records (대규모 웹 기록물의 원격수집을 위한 콘텐츠 중복 필터링 개선 연구)

이연수; 남성운; 윤대현

A study on the enhanced filtering method of the deduplication for bulk harvest of web records

The Korean Journal of Archival Studies
2013, (35), pp.133-160
Publisher : Korean Society Of Archival Studies
Research Area : Interdisciplinary Studies > Library and Information Science

이연수 ¹, 남성운 ¹, 윤대현 ²

¹국가기록원
²한국정보화진흥원

Accredited

ABSTRACT

As the network and electronic devices have been developed rapidly, the influences the web exerts on our daily lives have been increasing. Information created on the web has been playing more and more essential role as the important records which reflect each era. So there is a strong demand to archive information on the web by a standardized method. One of the methods is the snapshot strategy, which is crawling the web contents periodically using automatic software. But there are two problems in this strategy. First, it can harvest the same and duplicate contents and it is also possible that meaningless and useless contents can be crawled due to complex IT skills implemented on the web. In this paper, we will categorize the problems which can emerge when crawling web contents using snapshot strategy and present the possible solutions to settle the problems through the technical aspects by crawling the web contents in the public institutions.

KEYWORDS

web records, web contents, web crawling, deduplication

Citation status

* References for papers published after 2023 are currently being built.

[book] 이연수 / 2010 / 공공기관 웹 사이트 기록물 수집 사례분석 / 2010년도 한국멀티미디어학회 추계학술발표논문집

[book] Arvidson, A / 2001 / Kulturarw3 / Proc, Preserving the Present for the Future

[journal] Brewington, B.E. / 2000 / How dynamic is the web? / Computer Networks 33(1)

[other] Gomes, D. / 2006 / Design and selection criteria for a national web archive / Research and Advanced Technology for Digital Libraries

[journal] Hodges, D. / 1999 / The National Library of Canada’s digital library initiatives / Library hi tech 17(2)

[other] ISO / 2011 / Information and documentation-WARC fileformat(ISO 28500: 2009)

[web] / http://www.iso.org/iso/catalogue_detail.htm?csnumber=44717 / http://www.iso.org/iso/catalogue_detail.htm?csnumber=44717

[confproc] Koerbin, P / 2004 / The PANDORA Digital Archiving System (PANDAS) and managing Web archiving in Australia: A case study / 4th International Web Archiving Workshop

[confproc] K. Sigurosson / 2005 / Incremental crawling with Heritrix / Proceedings of the 5th International Web Archiving Workshop

[other] Leach, P. / 2005 / A Universally Unique IDentifier (UUID) URN Namespace / RFC 4122

[confproc] Mohr, G. / 2004 / Introduction to Heritrix / 4th International Web Archiving Workshop

[journal] Masanès, J / 2006 / Web archiving methods and approaches: A comparative study / Library trends 54(1)

[journal] Phillips, M. E / 2006 / What should we preserve? The question for heritage libraries in a digital world / Library trends 54(1)

[journal] Rauber, A. / 2002 / Uncovering information hidden in Web archives / D-Lib magazine 8(12)

[confproc] Sigurðsson, K / 2006 / Managing duplicates across sequential crawls / proceedings of the 6 International Web Archiving Workshop

KJCKorea
Journal Central

The Korean Journal of Archival Studies 2023 KCI Impact Factor : 0.98