Skip to main content

Use snappy codec with Hive

By Anonymous
Listen:

[1] Snappy is a compression and decompression library, initially developed from Google and now integrated into Hadoop. Snappy acts about 10% faster than LZO, the biggest differences are the packaging and that snappy only provides a codec and does not have a container spec, whereas LZO has a file-format container and a compression codec. Snappy is shipped with CDH3u2 (for Clouderas Distribution) included in the hadoop-0.20 package or in [2] Apache hadoop Version 0.21.0 up.

The example I explain was initially created from Esteban, an Cloudera Customer Operations Engineer.

Create a sequenced file
$ seq 1 1000 | awk '{OFS="\001";print $1, $1 % 10}' > test_input.hive
$ cat test_input.hive |head -5
11
22
33
44

Upload into hdfs
$ hadoop dfs -mkdir /tmp/hivetest
$ hadoop dfs -put /home/hdfs/test_input.hive /tmp/hivetest

$ hadoop dfs -ls /tmp/hivetest
Found 1 items
-rw-r--r--   3 hdfs supergroup       5893 2012-01-19 09:58 /tmp/hivetest/test_input.hive

Process the plain file in hive with snappy
Now we create an external table in hive with the content of the uploaded file. An external table reference in hive has to be a directory.

hive> create external table hivetest1 (a int, b int) location '/tmp/hivetest';
OK
Time taken: 0.053 seconds
hive> select * from hivetest1 limit 1;
OK
1       1
Time taken: 0.155 seconds

To work with another compression codec as configured in hive-site.xml we have to enable and define the codec usually done with internal SET statements from the hive command line. For batch jobs the definitions should be set with "hive -e SET ...."

hive> SET hive.exec.compress.output=true;
hive> SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
hive> SET mapred.output.compression.type=BLOCK;
hive> create table hivetest2 (a int, b int);
OK
Time taken: 0.15 seconds

After we successfully created the table "hivetest2" we try to import the data from our generated sequence file into the second table and use the exported compression codec. Here, hive starts a mapreduce job and after finishing moves the data into the new table and overwrites existing data.
 
hive> insert overwrite table hivetest2 select * from hivetest1;
Total MapReduce jobs = 2
Launching Job 1 out of 2
[.. removed ..]
Ended Job = 1068060947, job is filtered out (removed at runtime).
Moving data to: hdfs://hadoop1.internal:9000/tmp/hive-hdfs/hive_2012-01-19_10-20-11_796_1729199454214158343/-ext-10000
Loading data to table default.hivetest2
Deleted hdfs://hadoop1.internal:9000/user/hive/warehouse/hivetest2
Table default.hivetest2 stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 4021]
1000 Rows loaded to hivetest2
OK
Time taken: 16.843 seconds

Now we want to see if we can read the data and if we used the compression codec we wanted. The mapreduce job created one file, as we expected.

hive> select * from hivetest2 limit 1;
OK
1       1
Time taken: 0.171 seconds
$ hadoop dfs -ls /user/hive/warehouse/hivetest2
Found 1 items
-rw-r--r--   3 hdfs supergroup       4021 2012-01-19 10:20 /user/hive/warehouse/hivetest2/000000_0.snappy

Finally read the file
$ hadoop fs -cat /user/hive/warehouse/hivetest2/000000_0.snappy | head -5
.¸11
22
33
44
55

We processed the file in hive with snappy, created an snappy output and can work with it. Remember, if you want to use snappy you have to set the codec in your mapreduce jobs.

We can also use LOAD DATA statements to process the data we created and compressed with snappy

hive> create table hivetest3 (a int, b int);
OK
hive> load data inpath '/user/hive/warehouse/hivetest2/000000_0.snappy' into table hivetest3;
Loading data to table default.hivetest3
OK
Time taken: 0.327 seconds
hive> select * from hivetest3 limit 5;
OK
1       1
2       2
3       3
4       4
5       5
Time taken: 0.178 seconds

[1] http://code.google.com/p/hadoop-snappy/
[2] https://issues.apache.org/jira/browse/HADOOP-7206

Labels:

Comments

Post a Comment

Popular posts from this blog

Why Is Customer Obsession Disappearing?

By
 It's wild that even with all the cool tech we've got these days, like AI solving complex equations and doing business across time zones in a flash, so many companies are still struggling with the basics: taking care of their customers.The drama around Coinbase's customer support is a prime example of even tech giants messing up. And it's not just Coinbase — it's a big-picture issue for the whole industry. At some point, the idea of "customer obsession" got replaced with "customer automation," and now we're seeing the problems that came with it. "Cases" What Not to Do Coinbase, as main example, has long been synonymous with making cryptocurrency accessible. Whether you’re a first-time buyer or a seasoned trader, their platform was once the gold standard for user experience. But lately, their customer support practices have been making headlines for all the wrong reasons: Coinbase - Stuck in the Loop:  Users have reported being caugh...
Post a Comment
Read more

MySQL Scaling in 2024

By
When your MySQL database reaches its performance limits, vertical scaling through hardware upgrades provides a temporary solution. Long-term growth, though, requires a more comprehensive approach. This involves optimizing the database strategically and integrating complementary technologies. Caching The implementation of a caching layer, such as Memcached or Redis , can result in a notable reduction in the load and an increase ni performance at MySQL. In-memory stores cache data that is accessed frequently, enabling near-instantaneous responses and freeing the database for other tasks. For applications with heavy read traffic on relatively static data (e.g. product catalogues, user profiles), caching represents a low-effort, high-impact solution. Consider a online shop product catalogue with thousands of items. With each visit to the website, the application queries the database in order to retrieve product details. By using caching, the retrieved details can be stored in Memcached (a...
Post a Comment
Read more

Can AI Really Code?

By
My upcoming novel,  Catalyst , is set in a world where AI is a major player in shaping the human future. I did some research into how AI is currently being used in software development and found that it has some amazing capabilities, but also some limitations that are a bit concerning. I'd even go so far as to say that those models are a bit of a hoax. They're impressive, but they don't actually solve anything. Yes, AI coding assistants like Devin and Copilot are impressive in demos and demo videos. In reality, they're not as powerful as you'd think, but they're great for simple tasks like crafting email parsing functions or authentication flows. However, I ran into some issues when I tried to use it in more complex situations. When I asked the AI to " write a connector from a database to ingest data into Spark ," it didn't understand and made mistakes. And that is a pure, simple and so well documented task that every non-coder could do that by sim...
Post a Comment
Read more