Skip to main content

Use snappy codec with Hive

By Anonymous
Listen:

[1] Snappy is a compression and decompression library, initially developed from Google and now integrated into Hadoop. Snappy acts about 10% faster than LZO, the biggest differences are the packaging and that snappy only provides a codec and does not have a container spec, whereas LZO has a file-format container and a compression codec. Snappy is shipped with CDH3u2 (for Clouderas Distribution) included in the hadoop-0.20 package or in [2] Apache hadoop Version 0.21.0 up.

The example I explain was initially created from Esteban, an Cloudera Customer Operations Engineer.

Create a sequenced file
$ seq 1 1000 | awk '{OFS="\001";print $1, $1 % 10}' > test_input.hive
$ cat test_input.hive |head -5
11
22
33
44

Upload into hdfs
$ hadoop dfs -mkdir /tmp/hivetest
$ hadoop dfs -put /home/hdfs/test_input.hive /tmp/hivetest

$ hadoop dfs -ls /tmp/hivetest
Found 1 items
-rw-r--r--   3 hdfs supergroup       5893 2012-01-19 09:58 /tmp/hivetest/test_input.hive

Process the plain file in hive with snappy
Now we create an external table in hive with the content of the uploaded file. An external table reference in hive has to be a directory.

hive> create external table hivetest1 (a int, b int) location '/tmp/hivetest';
OK
Time taken: 0.053 seconds
hive> select * from hivetest1 limit 1;
OK
1       1
Time taken: 0.155 seconds

To work with another compression codec as configured in hive-site.xml we have to enable and define the codec usually done with internal SET statements from the hive command line. For batch jobs the definitions should be set with "hive -e SET ...."

hive> SET hive.exec.compress.output=true;
hive> SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
hive> SET mapred.output.compression.type=BLOCK;
hive> create table hivetest2 (a int, b int);
OK
Time taken: 0.15 seconds

After we successfully created the table "hivetest2" we try to import the data from our generated sequence file into the second table and use the exported compression codec. Here, hive starts a mapreduce job and after finishing moves the data into the new table and overwrites existing data.
 
hive> insert overwrite table hivetest2 select * from hivetest1;
Total MapReduce jobs = 2
Launching Job 1 out of 2
[.. removed ..]
Ended Job = 1068060947, job is filtered out (removed at runtime).
Moving data to: hdfs://hadoop1.internal:9000/tmp/hive-hdfs/hive_2012-01-19_10-20-11_796_1729199454214158343/-ext-10000
Loading data to table default.hivetest2
Deleted hdfs://hadoop1.internal:9000/user/hive/warehouse/hivetest2
Table default.hivetest2 stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 4021]
1000 Rows loaded to hivetest2
OK
Time taken: 16.843 seconds

Now we want to see if we can read the data and if we used the compression codec we wanted. The mapreduce job created one file, as we expected.

hive> select * from hivetest2 limit 1;
OK
1       1
Time taken: 0.171 seconds
$ hadoop dfs -ls /user/hive/warehouse/hivetest2
Found 1 items
-rw-r--r--   3 hdfs supergroup       4021 2012-01-19 10:20 /user/hive/warehouse/hivetest2/000000_0.snappy

Finally read the file
$ hadoop fs -cat /user/hive/warehouse/hivetest2/000000_0.snappy | head -5
.¸11
22
33
44
55

We processed the file in hive with snappy, created an snappy output and can work with it. Remember, if you want to use snappy you have to set the codec in your mapreduce jobs.

We can also use LOAD DATA statements to process the data we created and compressed with snappy

hive> create table hivetest3 (a int, b int);
OK
hive> load data inpath '/user/hive/warehouse/hivetest2/000000_0.snappy' into table hivetest3;
Loading data to table default.hivetest3
OK
Time taken: 0.327 seconds
hive> select * from hivetest3 limit 5;
OK
1       1
2       2
3       3
4       4
5       5
Time taken: 0.178 seconds

[1] http://code.google.com/p/hadoop-snappy/
[2] https://issues.apache.org/jira/browse/HADOOP-7206

Labels:

Comments

Post a Comment

Popular posts from this blog

Why Is Customer Obsession Disappearing?

By
 It's wild that even with all the cool tech we've got these days, like AI solving complex equations and doing business across time zones in a flash, so many companies are still struggling with the basics: taking care of their customers.The drama around Coinbase's customer support is a prime example of even tech giants messing up. And it's not just Coinbase — it's a big-picture issue for the whole industry. At some point, the idea of "customer obsession" got replaced with "customer automation," and now we're seeing the problems that came with it. "Cases" What Not to Do Coinbase, as main example, has long been synonymous with making cryptocurrency accessible. Whether you’re a first-time buyer or a seasoned trader, their platform was once the gold standard for user experience. But lately, their customer support practices have been making headlines for all the wrong reasons: Coinbase - Stuck in the Loop:  Users have reported being caugh...
Post a Comment
Read more

MySQL Scaling in 2024

By
When your MySQL database reaches its performance limits, vertical scaling through hardware upgrades provides a temporary solution. Long-term growth, though, requires a more comprehensive approach. This involves optimizing the database strategically and integrating complementary technologies. Caching The implementation of a caching layer, such as Memcached or Redis , can result in a notable reduction in the load and an increase ni performance at MySQL. In-memory stores cache data that is accessed frequently, enabling near-instantaneous responses and freeing the database for other tasks. For applications with heavy read traffic on relatively static data (e.g. product catalogues, user profiles), caching represents a low-effort, high-impact solution. Consider a online shop product catalogue with thousands of items. With each visit to the website, the application queries the database in order to retrieve product details. By using caching, the retrieved details can be stored in Memcached (a...
Post a Comment
Read more

What the Heck is Superposition and Entanglement?

By
If you’ve ever heard the words superposition or entanglement thrown around in conversations about quantum physics, you may have nodded politely while your brain quietly filed them away in the "too confusing to deal with" folder.  These aren't just theoretical quirks; they're the foundation of mind-bending tech like Google's latest quantum chip, the Willow with its 105 qubits. Superposition challenges our understanding of reality, suggesting that particles don't have definite states until observed. This principle is crucial in quantum technologies, enabling phenomena like quantum computing and quantum cryptography. What's in for us? Short, nothing at the moment. 105 qubits sounds awesome, but it would neither crack encryption nor enhance AI in the next few years. There are some use cases for Willow, like drug (protein) discovery or solving certain mathematical problems when they aren't too complicated. Right now, Google managed to turn physical qubits ...
Post a Comment
Read more