public class OrcInputFormat<V extends WritableComparable> extends FileInputFormat<NullWritable,V>
FileInputFormat.Counter
INPUT_DIR_RECURSIVE, LOG, NUM_INPUT_FILES
Constructor and Description |
---|
OrcInputFormat() |
Modifier and Type | Method and Description |
---|---|
static Reader.Options |
buildOptions(Configuration conf,
Reader reader,
long start,
long length)
Build the Reader.Options object based on the JobConf and the range of
bytes.
|
RecordReader<NullWritable,V> |
getRecordReader(InputSplit inputSplit,
JobConf conf,
Reporter reporter) |
protected FileStatus[] |
listStatus(JobConf job)
Filter out the 0 byte files, so that we don't generate splits for the
empty ORC files.
|
static boolean[] |
parseInclude(TypeDescription schema,
String columnsStr)
Convert a string with a comma separated list of column ids into the
array of boolean that match the schemas.
|
static void |
setSearchArgument(Configuration conf,
SearchArgument sarg,
String[] columnNames)
Put the given SearchArgument into the configuration for an OrcInputFormat.
|
addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getInputPathFilter, getInputPaths, getSplitHosts, getSplits, isSplitable, makeSplit, setInputPathFilter, setInputPaths, setInputPaths, setMinSplitSize
public static boolean[] parseInclude(TypeDescription schema, String columnsStr)
schema
- the schema for the readercolumnsStr
- the comma separated list of column idspublic static void setSearchArgument(Configuration conf, SearchArgument sarg, String[] columnNames)
conf
- the configuration to modifysarg
- the SearchArgument to put in the configurationcolumnNames
- the list of column names for the SearchArgumentpublic static Reader.Options buildOptions(Configuration conf, Reader reader, long start, long length)
conf
- the job configuratoinreader
- the file footer readerstart
- the byte offset to start readerlength
- the number of bytes to readpublic RecordReader<NullWritable,V> getRecordReader(InputSplit inputSplit, JobConf conf, Reporter reporter) throws IOException
getRecordReader
in interface InputFormat<NullWritable,V extends WritableComparable>
getRecordReader
in class FileInputFormat<NullWritable,V extends WritableComparable>
IOException
protected FileStatus[] listStatus(JobConf job) throws IOException
listStatus
in class FileInputFormat<NullWritable,V extends WritableComparable>
job
- the job configurationIOException
Copyright © 2013–2021 The Apache Software Foundation. All rights reserved.