org.cripac.isee.vpe.data

Class HDFSReader

java.lang.Object

org.cripac.isee.vpe.data.HDFSReader

public class HDFSReader
extends Object

The HDFSReader class is the base class for classes that read and parse data from HDFS.

Author:

Ken Yu, CRIPAC, 2016

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
protected org.apache.hadoop.fs.FileSystem	hdfs

Constructor Summary

Constructors

Constructor and Description
HDFSReader() Create a HDFSReader with default configuration.

Method Summary

Modifier and Type	Method and Description
List<org.apache.hadoop.fs.Path>	listSubfiles(org.apache.hadoop.fs.Path path) If the path specifies a directory, list all the sub-files in it.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

conf

protected org.apache.hadoop.conf.Configuration conf

hdfs

protected org.apache.hadoop.fs.FileSystem hdfs

Constructor Detail

HDFSReader

public HDFSReader()
           throws IOException

Create a HDFSReader with default configuration.

Throws:

IOException - On error opening Hadoop Filesystem.

Method Detail

listSubfiles

public List<org.apache.hadoop.fs.Path> listSubfiles(@Nonnull
                                                    org.apache.hadoop.fs.Path path)
                                             throws IOException

If the path specifies a directory, list all the sub-files in it. If the path specifies a file, return the path itself.

Parameters:

path - The path of a directory or a file.

Returns:

Paths of sub-files if the path specifies a directory or the given path itself if the path specifies a file.

Throws:

IOException - On error reading files in Hadoop Filesystem.