Monitoring & Troubleshooting AWS Cloud

Aviatrix enhances AWS monitoring by integrating with AWS services and offering advanced networking capabilities, such as improved security features, global transit networking, and network segmentation. This integration allows users to leverage the full potential of AWS CloudWatch within the Aviatrix platform, offering a seamless experience for monitoring AWS resources in real time, tracking various metrics, and analyzing log data for detailed insights​​​.

What Does AWS CloudWatch Do?

Amazon CloudWatch serves as the backbone for AWS monitoring, offering real-time monitoring and performance reporting, custom dashboard interfaces for tracking metrics, system logs monitoring for operational insights, and the ability to set alarms and notifications based on predefined metrics thresholds. Its features like log streams, log groups, and CloudWatch Logs Insights enable deep analysis and troubleshooting capabilities for AWS resources, including the use of VPC Flow Logs to monitor network traffic and identify issues​​.

Aviatrix & CloudWatch

The synergy between Aviatrix and AWS CloudWatch exemplifies how cloud networking solutions can be optimized to offer enhanced visibility, security, and operational efficiency. By leveraging Aviatrix for AWS network management, businesses can access a layer of security and performance monitoring that complements and extends the native AWS monitoring capabilities provided by CloudWatch. This dual approach ensures that cloud environments are not only well-monitored but also adhere to best practices in cloud architecture, security, and compliance​.

Alarms can be triggered when certain values of the metrics are crossed so that corrective action can be taken. In addition to this, dashboards are available with metrics for almost every AWS service that is used.

Amazon CloudWatch Logs allows admins to monitor and store logs from various sources. CloudWatch Logs can monitor the logs from EC2 instances, and monitor for errors in the applications.

Some important concepts about CloudWatch Logs are:

• Log stream: a sequence of log events that have the same source
• Log group: a set of log streams that have the same policies about retention, monitoring, or access control

Amazon CloudWatch Logs Insights allows to interactively search and analyze the log data from CloudWatch Logs using a purpose-built query language. Amazon CloudWatch Logs Insights can automatically discover fields in the logs from various AWS services like Route 53 or VPC.

As mentioned, Amazon CloudWatch can have multiple sources of information and this diagram shows how other AWS services (just to name a few) are interacting with Amazon CloudWatch:

So, let’s see how CloudWatch can monitor the traffic going through your VPC using the VPC Flow Logs feature.

VPC Flow Logs is a feature that allows the operator to receive information about the traffic incoming and outgoing a VPC. The log data can be sent to Amazon CloudWatch or Amazon S3.

The feature can help the operator find what kind of traffic is reaching the VPC and assist with troubleshooting cases where specific traffic does not reach the VPC.

A flow log has specific records that will specify the source/destination IP, source/destination port, protocol, the number of bytes and packets, and the action associated with the traffic.

This is the diagram used for this exercise:

Currently, there is one VPC with two subnets and there is one EC2 instance in each of these two subnets. This is the VPC:

These are the two instances where one instance is in the public subnet(it has a public IPv4 address):

Before enabling the VPC flow logs, let’s check the network interfaces assigned to these two EC2 instances (we will need the network interface IDs later):

To enable flow logs, you just need to go through a few specific steps using this menu:

In the next menu, you will need to specify if you want to log the accepted connections, rejected connections, or both. Along with this, you need to specify if you want to send the logs to an S3 bucket or CloudWatch Logs service. The IAM role can be one existing already or you can create one:

In case the IAM role is not created, you can create one directly from the above menu by selecting “Set Up Permissions” which will lead you to the below menu:

After the VPC flow log is enabled, in the CloudWatch service, you should see the flow group:

If that is expanded, then two log streams should be available, one for each network interface that we have in the VPC. As mentioned above, we have two EC2 instances, each with only one network interface. As you can see, the log stream ID is formed from the network interface ID and the filter type (in this specific case, all, but it can be accepted or rejected as well):

The EC2 instance from the public subnet has a web server running, but the VPC has a network access control list that is blocking/denying HTTP access to port 80 from any source IP address:

However, ICMP and ssh to any IP addresses assigned to this VPC are allowed.

My IP from where I will try the ICMP/ssh/HTTP traffic is the following one:

After few pings, ssh, and HTTP connection attempts, here is the content of the log stream generated for the network interface attached to the EC2 instance from the public subnet:

As you can see, in the flow logs, it is displayed as the private IPv4 address assigned to this EC2 instance, not the public IPv4. This is how it is recorded in the VPC flow logs(the instance is tracked via the private IPv4 because an EC2 instance might or might not have a public IPv4, but it will for sure have a private one). This type of logging will capture the traffic between EC2 instances/subnets of the same VPC.

The above logs are just the logs for one minute(18:25 – 18:26).

Using CloudWatch Logs Insights, you can get the above information in a more readable format.

For instance, you can find how many rejections were from each IP. In the above log snapshot, it’s easy because there are only a few lines, but imagine that you need to have an idea from a long time interval where there could be thousands of possible hosts trying to access your VPC.

Few predefined queries can be used specifically for various types of logs:

Here is a better way to display how many rejections were there for each IP for the same 1-minute interval. For my IP, there were three:

One other useful thing is that I can see how much traffic was exchanged between my VPC and various IP addresses. This is for a 25 minute interval during which intermittently I was pinging the EC2 instance from my laptop and at the same time I was pinging the Google DNS server from the EC2 instance:

In case you would want to have this information graphically, you can create CloudWatch dashboards where interesting information can be displayed.

This is how a dashboard is created:

Then from the metrics section, choose Logs

And one of the predefined metrics, like IncomingBytes:

Then the above metric can be displayed on the dashboard. This is how the dashboard looks like after I added the metric related to the number of logs received for the VPC Flow log:

Some other types of metrics can be used to create dashboards. This is another dashboard where specific EC2 metrics are displayed (in this particular case, these metrics are from the EC2 instance from the public subnet of the VPC):

And this would be the end of this article regarding Amazon CloudWatch which can help you to monitor AWS resources and perform troubleshooting.

Amazon CloudWatch can receive logs from different sources and then present those logs in a useful and easily readable format.