{"id":5975,"date":"2021-08-13T08:36:13","date_gmt":"2021-08-13T08:36:13","guid":{"rendered":"https:\/\/northbaysolutions.com\/?p=5975"},"modified":"2025-03-20T11:29:52","modified_gmt":"2025-03-20T11:29:52","slug":"amazon-athena-beyond-the-basics-part-2","status":"publish","type":"post","link":"https:\/\/northbaysolutions.com\/big-data-data-lake-analytics\/amazon-athena-beyond-the-basics-part-2\/","title":{"rendered":"Amazon Athena: Beyond The Basics \u2013 Part 2"},"content":{"rendered":"
\"\"<\/span><\/div><\/div><\/div>

Optimizing file formats and compression<\/strong><\/p>\n<\/div>

In the earlier blog post Athena: Beyond the Basics \u2013 Part 1, we have examined working with twitter data and executing complex queries using Athena. In the current article, we will understand the pricing model, experiment with different file formats and compression techniques and perform analysis based on the results and decide the best price to performance solution for the current use case.<\/p>\n<\/div>

Athena Pricing<\/strong><\/p>\n<\/div>

The Athena service is priced by the amount of data scanned when running the queries. Selecting the appropriate format and compression and balancing those factors with response time for queries will yield considerable cost savings for the expected response times.<\/p>\n

https:\/\/aws.amazon.com\/athena\/pricing\/<\/p>\n

The following section will share the process of converting to multiple file formats and compressions for additional experimentation for calculating the costs and response time for Twitter use-case.<\/p>\n<\/div><\/div><\/div>

Converting File formats and Compress<\/strong><\/p>\n<\/div>

Athena currently doesn\u2019t support inserting\/updating data on S3. EMR cluster or similar has to be used to create new data set using preferred file format from existing data set. We are going to add EMR to the previous architecture to convert the data to different file formats and compression techniques<\/p>\n<\/div>

\"\"<\/span><\/div>

The following link covers great deal of converting into multiple file formats
\nhttp:\/\/docs.aws.amazon.com\/athena\/latest\/ug\/convert-to-columnar.html<\/p>\n

The approach for converting to different file formats and querying from Athena<\/p>\n

    \n
  • Create Hive table to the relevant target format and compression<\/li>\n<\/ul>\n<\/div><\/div><\/div>

    CREATE EXTERNAL TABLE ([column - data type])
    \nSTORED AS
    \nLOCATION
    \ntblproperties(\u201c)<\/p>\n<\/div><\/div><\/div>

      \n
    • Populate the data into new storage\/compression format<\/li>\n<\/ul>\n<\/div><\/div><\/div>

      INSERT INTO SELECT * from<\/p>\n<\/div><\/div><\/div>

        \n
      • Create Athena table based on the new dataset stored on S3. Currently, Athena catalog manager doesn\u2019t share Hive catalog<\/li>\n<\/ul>\n

        The following code snippets are used to create multiple versions of the same data set for experimenting with Athena<\/p>\n<\/div>

        JSON FORMAT:<\/strong><\/p>\n<\/div>

          \n
        • To convert from Json to snappy compression we execute this commands in HIVE<\/li>\n<\/ul>\n<\/div><\/div><\/div>

          SET hive.exec.compress.output=true;
          \nSET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
          \nSET mapred.output.compression.type=BLOCK;<\/p>\n<\/div><\/div><\/div>

          Then we create the snappy external table exactly the same as we\u2019ve created it as json but with different name (tweets_snappy for example) and in different location and insert from json table to snappy table<\/p>\n

            \n
          • To convert from Json to gzip compression we execute this commands in HIVE<\/li>\n<\/ul>\n<\/div><\/div><\/div>

            SET hive.exec.compress.output=true;
            \nSET mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
            \nSET mapred.output.compression.type=BLOCK;<\/p>\n<\/div><\/div><\/div>

            Then we create the gzip external table exactly the same as we\u2019ve created it as json but with different name (tweets_gzip for example) and in different location and insert from json table to gzip table<\/p>\n

            TIP:<\/strong> for Hive running faster is better to copy from a compressed table, so it has to read less data and do less maps.<\/p>\n

            PARQUET FORMAT:<\/p>\n

              \n
            • To convert from parquet to parquet_snappy compression we add the following to the end of CREATE parquet table statement:<\/li>\n<\/ul>\n<\/div><\/div><\/div>

              TBLPROPERTIES (\"PARQUET.COMPRESS\"=\"SNAPPY\");<\/p>\n<\/div><\/div><\/div>

              Then we insert from any other already created and with data (json, json_snappy, parquet\u2026) to parquet_snappy table<\/p>\n

                \n
              • To convert from parquet to parquet_gzip compression we we add the following to the end of CREATE parquet table statement:<\/li>\n<\/ul>\n<\/div><\/div><\/div>

                TBLPROPERTIES (\"PARQUET.COMPRESS\"=\"GZIP\");<\/p>\n<\/div><\/div><\/div>

                Then we insert from any other already created and with data (json, json_snappy, parquet\u2026) to parquet_gzip table<\/p>\n<\/div><\/div><\/div>

                ORC FORMAT:<\/strong><\/p>\n<\/div>

                  \n
                • To convert from orc to orc_snappy compression we add the following to the end of CREATE ORC table statement:<\/li>\n<\/ul>\n<\/div><\/div><\/div>

                  TBLPROPERTIES (\"ORC.COMPRESS\"=\"SNAPPY\");<\/p>\n<\/div><\/div><\/div>

                  Then we insert from any other already created and with data (json, json_snappy, parquet\u2026) to orc_snappy table<\/p>\n

                    \n
                  • To convert from orc to orc_gzip compression we we add the following to the end of CREATE orc table statement:<\/li>\n<\/ul>\n<\/div><\/div><\/div>

                    TBLPROPERTIES (\"ORC.COMPRESS\"=\"GZIP\");<\/p>\n<\/div><\/div><\/div>

                    Then we insert from any other already created and with data (json, json_snappy, parquet\u2026) to orc_gzip table.<\/p>\n

                    Based on the above code snippets JSON, ORC, PARQUET without and with compression (SNAPPY, GZIP) are created for a total of 9 tables<\/p>\n<\/div><\/div><\/div>

                    Queries for Analysis<\/strong><\/p>\n<\/div>

                    The following queries from the earlier blog post will be used to perform analysis for data scanned and response times.<\/p>\n<\/div>

                    Query 1: Total records in the data set<\/strong><\/p>\n<\/div><\/div><\/div>

                    Select count(*) from tweets<\/p>\n<\/div><\/div><\/div>

                    Query 2: Get all records<\/strong><\/p>\n<\/div><\/div><\/div>

                    select * from tweets<\/p>\n<\/div><\/div><\/div>

                    Query 3: Top Hashtags with at least 100 occurrences<\/strong><\/p>\n<\/div><\/div><\/div>

                    SELECT ht.text,
                    \ncount(*)
                    \nFROM tweets
                    \nCROSS JOIN UNNEST (entities.hashtags) AS t(ht)
                    \nGROUP BY ht.text
                    \nHAVING count(*)&gt;100
                    \nORDER by count(*) desc;<\/p>\n<\/div><\/div><\/div>

                    Query 4: Number of tweets from verified accounts with most followers<\/strong><\/p>\n<\/div><\/div><\/div>

                    SELECT user.screen_name,user.name,max(user.followers_count),count(*)
                    \nFROM tweets
                    \nWHERE user.verified='true'
                    \nGROUP BY user.screen_name,user.name
                    \nORDER BY cast(max(user.followers_count) as integer) DESC<\/p>\n<\/div><\/div><\/div>

                    Query 5: Top URL mentions in Tweets<\/strong><\/p>\n<\/div><\/div><\/div>

                    SELECT url_extract_host(u.expanded_url),
                    \ncount(*)
                    \nFROM tweets
                    \nCROSS JOIN UNNEST (entities.urls) AS t(u)
                    \nGROUP BY url_extract_host(u.expanded_url)
                    \nHAVING count(*)>100
                    \nORDER by count(*) desc;<\/p>\n<\/div><\/div><\/div>

                    Query 6: Hashtags tweeted along with \u201cAmazon\u201d<\/strong><\/p>\n<\/div><\/div><\/div>

                    WITH ht_list AS
                    \n(SELECT entities.hashtags
                    \nFROM tweets
                    \nCROSS JOIN UNNEST (entities.hashtags) AS t(ht)
                    \nWHERE ht.text LIKE 'amazon')
                    \nSELECT t AS \"hashtag\",count(*) AS \"occurences\" FROM ht_list
                    \nCROSS JOIN UNNEST (hashtags) AS t(t)
                    \nGROUP BY t
                    \nORDER BY count(*) desc;<\/p>\n<\/div><\/div><\/div>

                    Query 7: Find the number of tweets by language and sensitive media content<\/strong><\/p>\n<\/div><\/div><\/div>

                    SELECT lang, possibly_sensitive, count(*)
                    \nFROM tweets
                    \nGROUP BY lang, possibly_sensitive<\/p>\n<\/div><\/div><\/div>

                    Query execution metrics<\/strong><\/p>\n<\/div>

                    The queries are executed on Athena (from Athena Query manager). Response times and Data scanned metrics are captured with each execution. Though the data scanned is same across multiple executions of query and table combination, response time varied by few milliseconds and sometimes by couple of seconds. We have taken the average of response time over multiple executions for the following output<\/p>\n<\/div>

                    \"\"<\/span><\/div>
                    \"\"<\/span><\/div><\/div><\/div>

                    Testing Analysis<\/strong><\/p>\n<\/div>

                      \n
                    • For the current use case and the identified set of queries, ORC format is providing the best price to performance ($0.55 for executing all queries within the 30 seconds of aggregated response times). This can be attributed to columnar storage format and the query use cases used for the blog post<\/li>\n
                    • Converting into ORC format or PARQUET format can save the costs on Athena usage by over 90%<\/li>\n
                    • Compressing raw JSON to snappy or gzip can also significantly reduce the costs by over 80% but the response time did not improve<\/li>\n
                    • Snappy\/Gzip compression on PARQUET and ORC formats have limited benefits as the file formats are already compressed
                      \nAvoid using raw text format of dataset when querying with Athena<\/li>\n<\/ul>\n<\/div><\/div><\/div>

                      Conclusion<\/strong><\/p>\n<\/div>

                      The blog post is intended to share the approach of testing and experimenting with different file formats and compression techniques to optimize for costs and performance. The findings for the current use case may or may not be applicable for different queries and use cases. We recommend that different options are tested out before finalizing the formats and compression for huge data sets and data lake scenarios.<\/p>\n<\/div>

                      \u201cAthena: Beyond the Basics \u2013 Part 2\u201d<\/p>\n<\/div>

                      Additional references:<\/strong><\/p>\n<\/div>

                      List of SQL statements supported by Athena<\/p>\n

                      https:\/\/aws.amazon.com\/blogs\/big-data\/analyzing-data-in-s3-using-amazon-athena\/
                      \nhttps:\/\/orc.apache.org\/
                      \nhttps:\/\/parquet.apache.org\/<\/p>\n<\/div><\/div><\/div><\/div><\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":3,"featured_media":6023,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55,38],"tags":[57,88],"class_list":["post-5975","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-big-data-data-lake-analytics","tag-all-industries","tag-exclude"],"_links":{"self":[{"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/posts\/5975","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/comments?post=5975"}],"version-history":[{"count":3,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/posts\/5975\/revisions"}],"predecessor-version":[{"id":24345,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/posts\/5975\/revisions\/24345"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/media\/6023"}],"wp:attachment":[{"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/media?parent=5975"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/categories?post=5975"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/tags?post=5975"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}