Querying CloudTrail Logs from a centralized S3 Location using Athena

Querying CloudTrail Logs from a centralized S3 Location using Athena

Comments 0 Comment

Hello, it’s been some time since my last blog post. Today, I want to discuss the process of querying AWS CloudTrail logs stored in a centralized S3 bucket using AWS Athena. But before diving into the topic, let’s briefly introduce AWS CloudTrail and AWS Athena. AWS CloudTrail is a service that provides governance, compliance, and auditing capabilities for your AWS account. It records all user actions performed on AWS services and resources as events. By aggregating logs from multiple accounts,…

Read More Read More

Covid-19 Analysis with BigData Applications – Part 3

Covid-19 Analysis with BigData Applications – Part 3

Comments 0 Comment

Hi there! If you’ve been following these blog series, we are looking at a BigData project to analyze Covid-19 data. So far, we have looked at the overall architecture and ETL Spark jobs. In this post, let’s look the scheduler component (Lambda function) in this workflow. The main reason I’m using Lambda here is due to its serverless nature and native integration with other AWS services. For example, we could trigger it via a CloudWatch scheduled rule in regular basis…

Read More Read More

Covid-19 Analysis with BigData Applications – Part 2

Covid-19 Analysis with BigData Applications – Part 2

Comments 0 Comment

Hi again! On this post, I’ll explain on the second two ETL jobs: first one to process the Twitter data related to Covid-19 and second one will combine the data from previous two ETL jobs. As we have already covered the basic EMR concept earlier, I’ll directly get into the explanation of what is being done in these task. For ETL2, I’m creating a Hive table beforehand because this Twitter data is in semi-colon delimited format and isn’t easily parsed…

Read More Read More

Covid-19 Analysis with BigData Applications – Part 1

Covid-19 Analysis with BigData Applications – Part 1

Comments 0 Comment

Hi again! So, if you came here after reading the introduction post, we’ll be talking about the part that we run on EMR cluster i.e. ETL job. Across BigData community, the term ETL generally refers to Extract, Transform and Load. And for this project, I’m using Apache Spark, which happens to be one of the most popular open-source projects currently. Data Ingestion Setup Since we’ll be running multiple ETL jobs for this project, I’ll mostly focus on the first one…

Read More Read More

Covid-19 Analysis with BigData Applications – Introduction

Covid-19 Analysis with BigData Applications – Introduction

Comments 0 Comment

Hi there! As we know it, Covid-19 pandemic has been the prime highlight of this year. It has been directly affecting our way of living and has also made some serious dents to global social-economic dynamics. And there are lots of people, who have been continually working to understand, fight and control this epidemic. So, it is all of our common responsibility to help this endeavor with as much as we can. So, on this blog series, I’d like to…

Read More Read More

Why Should Every Person Start Writing Their Own Blog?

Why Should Every Person Start Writing Their Own Blog?

Comments 0 Comment

What would you do if you were required to write your own blog? It’s certain that the first thing popping up in our mind will be something like: “What am I even supposed to write? I’m not an author and I’ve got nothing to tell the world. It isn’t my cup of tea.” Yeah, that is more-or-less what I thought when around 3 years ago, my mentor had suggested me to start my own blog. I was a final year…

Read More Read More

What I Learned from My Trek to Rara Lake – Part 3

What I Learned from My Trek to Rara Lake – Part 3

Comments 0 Comment

Day 2 It was a cold morning. The sun was rising up over the horizon and the early morning fog was slowly clearing up. All of us were up by 6 AM and were getting ready for the day’s journey. It was our second morning of the trip and compared to previous morning, we felt a lot refreshed. The vehicle on which we were going to travel was scheduled to leave at 6:30, so we hurriedly reached the bus station….

Read More Read More

What I Learned from My Trek to Rara Lake – Part 2

What I Learned from My Trek to Rara Lake – Part 2

Comments 1 comment

The day I mentioned in part 1 was actually the third day of our trip. It all began on Sunday evening. Three of us gathered together and left from Kathmandu. It might sound unusual, but we had planned very little for this long and adventurous journey of ours. The trip was confirmed only before a day or two and none of us had any detailed idea on how to reach there. Our first destination was Surkhet, a district in Mid-Western region…

Read More Read More

What I learned from my Trek to Rara Lake – Part 1

What I learned from my Trek to Rara Lake – Part 1

Comments 2 comments

It’s a Wednesday evening in October and it’s already pitch dark. The crickets also just started to sing welcome song to the night. The weather is getting severely cold and the wind is sending chills to our spines. But the worst part is we cannot see any other human or human settlements as far as we can see to provide us any warmth. Tiredness, pain and exhaustion are taking control over our bodies. Every step is getting heavier. Nevertheless we…

Read More Read More

Why Should You Do a Bungy Jump At Least Once?

Why Should You Do a Bungy Jump At Least Once?

Comments 0 Comment

Well, you must have watched some videos on Facebook or Youtube, in which a person jumps from a high bridge like structure with their legs tied with some sort of long rope. I don’t know what you usually feel when you see that type of videos or photos. Maybe inspiration? Fear? Surprise? Guilt? Hatred? Or what you feel simply depends upon the person who did it or the state of your mind, right? Whatever you feel, it’s totally natural, so…

Read More Read More