Columnar-Oriented File Format

A Columnar-Oriented File Format is a structured data file format for columnar data in columnar data files.

• Context:
  • It can effectively support Columnar Data Analysis.
  • …
• Example(s):
  • Parquet File Format.
  • ORC File Format.
  • …
• Counter-Example(s):
  • Row-Oriented File Format.
• See: Structured Data File.

---

---

References

2016

• https://blog.matthewrathbone.com/2016/09/01/a-beginners-guide-to-hadoop-storage-formats.html
  • QUOTE: The latest hotness in file formats for Hadoop is columnar file storage. Basically this means that instead of just storing rows of data adjacent to one another you also store column values adjacent to each other. So datasets are partitioned both horizontally and vertically. This is particularly useful if your data processing framework just needs access to a subset of data that is stored on disk as it can access all values of a single column very quickly without reading whole records.

    One huge benefit of columnar oriented file formats is that data in the same column tends to be compressed together which can yield some massive storage optimizations (as data in the same column tends to be similar).

    If you’re chopping and cutting up datasets regularly then these formats can be very beneficial to the speed of your application, but frankly if you have an application that usually needs entire rows of data then the columnar formats may actually be a detriment to performance due to the increased network activity required.

    Overall these formats can drastically optimize workloads, especially for Hive and Spark which tend to just read segments of records rather than the whole thing (which is more common in MapReduce).

    Of the two file formats I mention, Parquet seems to have the most community support and is the format I would use.

---