{"id":4583,"date":"2021-08-04T08:31:44","date_gmt":"2021-08-04T08:31:44","guid":{"rendered":"https:\/\/northbaysolutions.com\/?p=4583"},"modified":"2025-03-20T12:44:42","modified_gmt":"2025-03-20T12:44:42","slug":"amazon-athena-beyond-the-basics-part-1","status":"publish","type":"post","link":"https:\/\/northbaysolutions.com\/blog\/amazon-athena-beyond-the-basics-part-1\/","title":{"rendered":"Amazon Athena: Beyond The Basics \u2013 Part 1"},"content":{"rendered":"
\"\"<\/span><\/div><\/div><\/div>

Working with Twitter (complex JSON) data set<\/strong><\/h3>\n<\/div>

Amazon Athena is a serverless interactive query service that allows analytics using standard SQL for data residing in S3. Before Athena, to query data sets on S3, Hive\/Presto\/Hue or similar tools had to be installed on top EMR service or integrated with other third party partner products.<\/p>\n

Athena also supports JDBC connectivity so the managed service can be easily integrated with wide variety of SQL and Visualization tools.<\/p>\n

https:\/\/aws.amazon.com\/athena\/<\/p>\n<\/div>

Lot of customers are interested in exploring Amazon Athena for their use case and looking for ways to optimize for performance and costs. As an APN Partner NorthBay has been working with Athena in testing and exploring various customer use cases. This is a multi-part blog series to share our findings as well as provide the audience with a jumpstart on working with Amazon Athena..<\/p>\n<\/div>

Twitter use case<\/strong><\/h3>\n<\/div>

Unstructured Data and semi-structured (typically JSON) is becoming typical for Big Data sets. We have chosen Twitter data as the data set to validate working on Athena with complex JSON\u2019s. The current blog post will share the details of querying Twitter data using Athena and executing complex queries based on the data set.<\/p>\n

The following is the architecture followed for the implementation:<\/p>\n<\/div>

\"\"<\/span><\/div>
    \n
  • Configure Twitter for API access<\/li>\n
  • Configure Kinesis Firehose to stream the output to S3<\/li>\n
  • Configure and run Tweepy to read Twitter feed and stream to Kinesis Firehose<\/li>\n
  • Define schema definition in Athena<\/li>\n
  • Query Twitter data from Athena Query Editor<\/li>\n
  • Query Twitter data using JDBC connection<\/li>\n
  • Query Twitter data from Quicksight<\/li>\n<\/ul>\n<\/div><\/div><\/div>

    Configure Twitter for API access<\/strong><\/h3>\n<\/div>

    To create this platform, you will need an AWS account and a Twitter application. Sign in with your Twitter account and create a new application at https:\/\/apps.twitter.com\/. Make sure your application is set for \u2018read-only\u2019 access. Next, choose Create My Access Token at the bottom of the Keys and Access Tokens tab. By this point, you should have four Twitter application keys: consumer key (API key), consumer secret (API secret), access token, and access token secret. Take note of these keys.<\/p>\n<\/div>

    https:\/\/aws.amazon.com\/blogs\/big-data\/building-a-near-real-time-discovery-platform-with-aws\/<\/p>\n<\/div><\/div><\/div>

    Configure Kinesis Firehose to stream the output to S3<\/strong><\/h3>\n<\/div>

    Create a Kinesis Firehose Delivery Stream as the destination for our data.<\/p>\n<\/div><\/div><\/div>

    \"\"<\/span><\/div>
    \"\"<\/span><\/div>

    Choose \u201cCreate bucket\u201d:<\/p>\n<\/div><\/div><\/div>

    Step 1: Configure Destination: Choose \u201cAmazon S3\u201d as Destination and select the existing S3 bucket or create a new Bucket for Firehose to persist the data.<\/p>\n<\/div><\/div><\/div>

    \"\"<\/span><\/div>
    \"\"<\/span><\/div><\/div><\/div>

    Once the Bucket is created, add a prefix to the data. In this case, json\/ prefix is added so all json data goes to the same bucket\/prefix<\/p>\n<\/div><\/div><\/div>

    Step2: Configuration: Kinesis Firehose allows for optimizations and configuration for Buffer sizes, interval, compression, encryption and security policies. These values can be chosen based on the streaming ingest frequency and optimal size of the output file in S3.<\/p>\n<\/div><\/div><\/div>

    \"\"<\/span><\/div><\/div><\/div>
    \"\"<\/span><\/div>

    We just have to click \u201cAllow\u201d in the new window without changing anything.<\/p>\n

    Step3 Review:<\/p>\n

    Review the configuration and click \u201cCreate Delivery Stream\u201d<\/p>\n<\/div>

    \"\"<\/span><\/div><\/div><\/div>

    In the Firehose Delivery Stream console, we can see our created Delivery Stream with status \u201cCREATING\u201d. Once the status changes to \u201cACTIVE\u201d we can start using the delivery stream.<\/p>\n<\/div><\/div><\/div>

    Ingest Twitter feeds from the feeder system (Tweepy\/Python)<\/strong><\/h3>\n<\/div>

    http:\/\/docs.tweepy.org\/en\/v3.5.0\/index.html<\/p>\n

    We need a stream producer\/feeder system to publish streaming data to Kinesis Firehose. Tweepy is an open-source python library that enables communication with Twitter. The following code can be run on an EC2 instance (with relevant IAM role to access Kinesis Firehose and Twitter API credentials from the earlier step in configuration file) to feed the stream that we created earlier.<\/p>\n<\/div><\/div><\/div>

    import tweepy
    \nfrom tweepy import Stream
    \nfrom tweepy import OAuthHandler
    \nfrom tweepy.streaming import StreamListener
    \nimport time
    \nimport argparse
    \nimport string
    \nimport config
    \nimport json
    \nimport boto3<\/p>\n<\/div><\/div><\/div>

    def get_parser():
    \n\"\"\"Get parser for command line arguments.\"\"\"
    \nparser = argparse.ArgumentParser(description=\"Twitter Downloader\")
    \nparser.add_argument(\"-q\",
    \n\"\u2013query\",
    \ndest=\"query\",
    \nhelp=\"Query\/Filter\",
    \ndefault='-')
    \nparser.add_argument(\"-d\",
    \n\"\u2013data-dir\",
    \ndest=\"data_dir\",
    \nhelp=\"Output\/Data Directory\")
    \nreturn parser<\/p>\n

    class MyListener(StreamListener):
    \n\"\"\"Custom StreamListener for streaming data.\"\"\"<\/p>\n

    def __init__(self, data_dir, query):
    \nquery_fname = format_filename(query)
    \nself.outfile = \"%s\/stream_%s.json\" % (data_dir, query_fname)<\/p>\n

    def on_data(self, data):
    \ntry:
    \nresult = send_record_to_firehose(data)
    \nprint(result)
    \nexcept BaseException as e:
    \nprint(\"Error on_data: %s\" % str(e))
    \ntime.sleep(5)
    \nreturn True<\/p>\n

    def on_error(self, status):
    \nprint(\"Error with status:\" + str(status))
    \nif status == 420:
    \nprint(\"You are being rate limited!!!.\")
    \nreturn True<\/p>\n

    def send_record_to_firehose(data):
    \n\"\"\" Sends Json response from tweeter to Kinesis Firehose Delivery Stream
    \nArguments:
    \ndata \u2014 json file from tweeter
    \nReturn:
    \nString \u2014 json response from Kinesis Firehos
    \n\"\"\"
    \nclient = boto3.client('firehose')
    \nresponse = client.put_record(
    \nDeliveryStreamName='twitter-to-s3\u2032,
    \nRecord=
    \n)
    \nreturn response<\/p>\n

    def format_filename(fname):
    \n\"\"\"Convert file name into a safe string.<\/p>\n

    Arguments:
    \nfname \u2014 the file name to convert
    \nReturn:
    \nString \u2014 converted file name
    \n\"\"\"
    \nreturn \".join(convert_valid(one_char) for one_char in fname)<\/p>\n

    def convert_valid(one_char):
    \n\"\"\"Convert a character into '_' if invalid.<\/p>\n

    Arguments:
    \none_char \u2014 the char to convert
    \nReturn:
    \nCharacter \u2014 converted char
    \n\"\"\"
    \nvalid_chars = \"-_.%s%s\" % (string.ascii_letters, string.digits)
    \nif one_char in valid_chars:
    \nreturn one_char
    \nelse:
    \nreturn '_'<\/p>\n

    @classmethod
    \ndef parse(cls, api, raw):
    \nstatus = cls.first_parse(api, raw)
    \nsetattr(status, 'json', json.dumps(raw))
    \nreturn status<\/p>\n

    if __name__ == '__main__':
    \nparser = get_parser()
    \nargs = parser.parse_args()
    \nauth = OAuthHandler(config.consumer_key, config.consumer_secret)
    \nauth.set_access_token(config.access_token, config.access_secret)
    \napi = tweepy.API(auth)
    \n# Added this logic to reconnect if it fails
    \nwhile True:
    \ntry:
    \ntwitter_stream = Stream(auth, MyListener(args.data_dir, args.query))
    \ntwitter_stream.filter(track=[args.query])
    \nexcept Exception:
    \n# Sure??, let's reconnect and keep tracking
    \ncontinue<\/p>\n<\/div><\/div><\/div>

    Define schema definition in Athena<\/strong><\/h3>\n<\/div>

    Catalog Manager UI is provided to define new tables in Athena. However, with complex JSON it is easier to run the schema definition DDL in the query editor. The following DDL is generated\/built based on Twitter data stream:<\/p>\n<\/div><\/div><\/div>

    CREATE EXTERNAL TABLE IF NOT EXISTS tweets (
    \ncreated_at string ,
    \nid string ,
    \nid_str string ,
    \ntext string ,
    \ndisplay_text_range ARRAY,
    \nsource string ,
    \ntruncated string ,
    \nuser struct< id:string , id_str:string , name:string , screen_name:string , location:string , description:string , protected:string , verified:string , followers_count:string , friends_count:string , listed_count:string , favourites_count:string , statuses_count:string , created_at:string , utc_offset:string , time_zone:string , geo_enabled:string , lang:string>,
    \nis_quote_status string,
    \nextended_tweet STRUCT< full_text:string, display_text_range:ARRAY,
    \nentities:STRUCT< media:ARRAY<STRUCT< id:string, id_str:string, indices:ARRAY,
    \nmedia_url:string,
    \nmedia_url_https:string,
    \nurl:string,
    \ndisplay_url:string,
    \nexpanded_url:string,
    \ntype:string,
    \nsizes:STRUCT< small:STRUCT<w:string, h:string, resize:string>,
    \nthumb:STRUCT<w:string, h:string, resize:string>>>>>>,
    \nretweet_count string,
    \nfavorite_count string,
    \nretweeted_status STRUCT< retweet_count:string, text:string>,
    \nentities STRUCT< urls:ARRAY<STRUCT<url:string, expanded_url:string, display_url:string, indices:ARRAY>>,
    \nuser_mentions:ARRAY<STRUCT>,
    \nhashtags:ARRAY>,
    \nfavorited string,
    \nretweeted string,
    \npossibly_sensitive string,
    \nfilter_level string,
    \nlang string)
    \nROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
    \nLOCATION 's3:\/\/twitter-to-s3\/json\/';<\/p>\n<\/div><\/div><\/div>

    Query Twitter data from Athena Query Editor<\/strong><\/h3>\n<\/div>

    Athena Query Editor provides a UI to submit Queries to Athena. The response also captures the run time and the data scanned which are incredibly useful for estimating costs and optimizing queries.<\/p>\n<\/div>

    \"\"<\/span><\/div>

    Query 1: Total records in the table<\/strong><\/p>\n<\/div><\/div><\/div>

    SELECT count(*) FROM tweets<\/p>\n<\/div><\/div><\/div>

    Query 2: Get sample of 10000 records<\/strong><\/p>\n<\/div><\/div><\/div>

    >
    \nSELECT * FROM tweets LIMIT 10000<\/p>\n<\/div><\/div><\/div>

    Query Twitter data from SQL client with JDBC connection<\/strong><\/p>\n<\/div>

    Detailed documentation is available from the following link to establish a connection to Athena from a client tool such as SQL Workbench:<\/p>\n

    http:\/\/docs.aws.amazon.com\/athena\/latest\/ug\/connect-with-jdbc.html<\/p>\n<\/div>

    Query 3: Top hashtags with at least 100 occurrences<\/strong><\/p>\n<\/div><\/div><\/div>

    SELECT ht.text,count(*)
    \nFROM tweets
    \nCROSS JOIN UNNEST (entities.hashtags) AS t(ht)
    \nGROUP BY ht.text
    \nHAVING count(*)>100
    \nORDER by count(*) desc<\/p>\n<\/div><\/div><\/div>

    Query 4: Number of Tweets from verified accounts with the most followers<\/strong><\/p>\n<\/div><\/div><\/div>

    SELECT user.screen_name,user.name,max(user.followers_count),count(*)
    \nFROM tweets
    \nWHERE user.verified='true'
    \nGROUP BY user.screen_name,user.name
    \nORDER BY cast(max(user.followers_count) as integer) DESC<\/p>\n<\/div><\/div><\/div>

    Query 5: Top URL mentions in Tweets<\/strong><\/p>\n<\/div><\/div><\/div>

    SELECT url_extract_host(u.expanded_url),
    \ncount(*)
    \nFROM tweets
    \nCROSS JOIN UNNEST (entities.urls) AS t(u)
    \nGROUP BY url_extract_host(u.expanded_url)
    \nHAVING count(*)&gt;100
    \nORDER by count(*) desc;<\/p>\n<\/div><\/div><\/div>

    Query 6: Hashtags tweeted along with \u201cAmazon\u201d<\/strong><\/p>\n<\/div><\/div><\/div>

    Query 6: Hashtags tweeted along with \u201cAmazon\u201dWITH ht_list AS
    \n(SELECT entities.hashtags
    \nFROM tweets
    \nCROSS JOIN UNNEST (entities.hashtags) AS t(ht)
    \nWHERE ht.text LIKE 'amazon')
    \nSELECT t AS \"hashtag\",count(*) AS \"occurences\" FROM ht_list
    \nCROSS JOIN UNNEST (hashtags) AS t(t)
    \nGROUP BY t
    \nORDER BY count(*) desc;<\/p>\n<\/div><\/div><\/div>

    Visualize Twitter data from Quicksight using Athena<\/strong><\/p>\n<\/div>

    The following blogpost provides information on querying from Athena using Quicksight<\/p>\n

    https:\/\/aws.amazon.com\/blogs\/big-data\/derive-insights-from-iot-in-minutes-using-aws-iot-amazon-kinesis-firehose-amazon-athena-and-amazon-quicksight\/<\/p>\n

    Quicksight currently does not support complex JSON\u2019s and expects the data types to be among the supported data types:<\/p>\n

    http:\/\/docs.aws.amazon.com\/quicksight\/latest\/user\/data-source-limits.html<\/p>\n

    The current dashboard displays sensitive Tweets by language from the data set:<\/p>\n<\/div>

    \"\"<\/span><\/div>

    The underlying query:<\/p>\n<\/div>

    Query 7: Find the number of Tweets by language and sensitive media content<\/strong><\/p>\n<\/div><\/div><\/div>

    SELECT lang,possibly_senstive,count(*)
    \nFROM tweets
    \nGROUP BY lang, possibly_sensitive<\/p>\n<\/div><\/div><\/div>

    Conclusion<\/strong><\/p>\n<\/div>

    Twitter analysis using Athena proves that the product can be leveraged for use cases involving complex data formats (unstructured\/semi-structured), can be automated using JDBC connections, and reveals basic insights using Quicksight (lack of support for arrays currently hinders analytics capabilities).<\/p>\n

    Athena can be excellent tool for \u201cS3 as a data lake\u201d use cases where the data is already staged in S3 and with serverless managed service, Athena takes precedence over previously used methods like Presto\/Hive\/Impala on EMR.<\/p>\n<\/div>

    \u201cAthena: Beyond the Basics \u2013 Part 2\u201d<\/p>\n<\/div>

    Additional references:<\/strong><\/h3>\n<\/div>

    List of SQL statements supported by Athena<\/p>\n

    http:\/\/docs.aws.amazon.com\/athena\/latest\/ug\/language-reference.html<\/p>\n<\/div><\/div><\/div><\/div><\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":3,"featured_media":4711,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55,38],"tags":[57,88],"class_list":["post-4583","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-big-data-data-lake-analytics","tag-all-industries","tag-exclude"],"_links":{"self":[{"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/posts\/4583","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/comments?post=4583"}],"version-history":[{"count":4,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/posts\/4583\/revisions"}],"predecessor-version":[{"id":24356,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/posts\/4583\/revisions\/24356"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/media\/4711"}],"wp:attachment":[{"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/media?parent=4583"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/categories?post=4583"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/northbaysolutions.com\/wp-json\/wp\/v2\/tags?post=4583"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}