Skip to content

joh-mue/ahar

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Introduction

AHAR (Appendable Hadoop Archive) is a tool for Hadoop Distributed File System (HDFS) that allows you to add new files to an already existing Hadoop Archive (HAR). AHAR is executable without any HDFS modification. For questions and feedback please contact Johannes or Thomas.

How to use

  1. Download or clone the repository and run mvn clean package -DskipTests
  2. Set the system variable $HADOOP_HOME to your hadoop path. AHAR will automatically detect your running HDFS instance. Alternative, you can set the NameNode address as an argument in step 3.
  3. Now you can run the jar in the target folder with the arguments cp <Files/Folder Path> . For example: “*java -jar AppendableHadoopArchive-1.0-allinone.jar cp /user/marc/file.log /user/marc/an.har”. You can also download the jar from here.

Implementation

AHAR appends the binary data of the new files to one of the part-n files of the HAR archive using a first fit algorithm. Afterwards, the index entries in the masterindex and index files are updated and rewritten to HDFS.

Known issues

AHAR uses HDFS append method to append new data to an existing HAR part files. Thus, you must have a cluster with three or more DataNodes or set part file replication to one for testing AHAR on a small cluster HDFS-4600 or HDFS-8960.

About

Appendable Hadoop Archive

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Java 100.0%