AKS Monitoring At Scale – Part 1 (AMA/Container Insight)

May 1, 2023May 1, 2023 hereisakash AKS, alerting, azure, Container, Governance, logging, Monitoring, Policy, SRE

Azure Container Monitoring for Azure Kubernetes Services (AKS) is a vital aspect that varies from one deployment to another. Although enabling AKS Diagnostics Settings and sending logs to a Log Analytics Workspace might seem like enough, there are still a few moving parts to consider. In this blog post, we will discuss the importance of Container Insight and how to automatically and efficiently onboard it for any cluster.

First and Foremost, let me put ways of monitoring AKS Cluster and it’s component;

Container Insight (Workload Metrics/Logs)
AKS Control Plan Logs (Diagnostics Logs)
Prometheus Logs (New Offering)

In this post, I am going to cover `Container Insight` using Azure Monitor Agent (AMA). It is recommended that any AKS cluster should have it deployed as must, best practice. To deploy this, you need;

Log Analytics Workspace where you will send logs/metric to
User Managed Identity (Best Practice for Remediation Task) , Else System Assigned Managed Identity
Enable Add-On for Monitoring (Azure Monitoring Agent Container)

You can onboard any cluster with Container Insight as part of deployment by using a provided configuration. Alternatively, you could use Azure Policy to do it for you. In this blog post, we will show you how to automatically and efficiently onboard Container Insight for any cluster, using standardized configurations.

Have a review of demo which is covering scenario;

Key things and policy code,

Azure Policy to Onboard Container Insight – cloud_governance_monitoring/Azure/Monitoring/AKS_Monitoring_At_Scale_AMAConainerInsight at main · spotakash/cloud_governance_monitoring (github.com)
Some good documentation –
- Container Insights: https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-overview
- Container Insight via Azure Policy: https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-enable-aks-policy
- KQL Queries https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-log-query#resource-logs
Sample Code parameter for TF and CLI
- Terraform
- CLI

This is a personal experience sharing post that will provide you with practical insights on how to get started with Container Insight for AKS cluster monitoring. Don’t hesitate to take advantage of this best practice and ensure the performance and health of your Kubernetes cluster and container workloads. Also, please feel free to add your comment. Please note, technology keep changing at rapid pace. Don’t be surprised we shall have better alternatives.

Coming blog posts, I may cover following topics;

AKS Monitoring using Managed Prometheus and Managed Grafana at Scale
AKS Diagnostic Monitoring at Scale
AKS all Monitoring using ALZ Terraform Module Custom Policy Extension
How to have AKS Enterprise Wide Dashboards for Central Monitoring

Azure LogicApps with ISE (integration service environment)

November 23, 2020 hereisakash azure, integration service environment, ise, logicapp, private endpoint, private link, private storage account

How cool is Azure Logic Apps, we all know. Building automated workflows without writing a single line of code (or almost negligible) is simply amazing. Out-of-box connectors make it enterprise grade solution such that we can invoke unique flow based upon business logic based upon feeds coming from different system. These different systems can be B2B, SaaS Products, basically provide Enterprise Integration Pack with Logic Apps following Industry standard around Electronic Data Interchange (EDI)/ Enterprise Application Integration (EAI).

Logic App is Platform as a Service. We don’t need to worry about base environment like OS/Patch/availability etc.

Recently, I have concluded major project where we used intensive capability to automate various business flows. These flow can be as simple as writing all incoming sales enquiry to azure storage blob or creating service request for any specific type of incident scenario reported by Azure Log Analytics etc. I’ll try to cover scenario below, how we overcome, what were key observation.

Requirement: Due to security compliance all Azure Storage Account were Private in nature. Thus, they can be accessed only via Private Link (through Private Endpoint). On-Premise system can send their feeds to Read Blob Container via Hybrid Connectivity Established. But, how will we process feeds getting written in ‘Read Blob’. Logic App by default does not sit on VNET. In any circumstance, we can not take either storage account or logic app outside of same geography.

Solution: In May 2019 Integration Service environment (ISE) was being launched. Using ISE, Logic App can be easily be brought inside VNET vicinity. Thus, can allow to consume or interact with service or environment which are behind firewalls rules or private Endpoints. ISE, basically provision LogicApps inside a VNET Subnet.

Observations: Few of key observations;

ISE takes sometime to provision. Don’t get impatient, behind the scenario Azure is provision a lot of service and connectors so that specific LogicApps can operate with ease.
ISE will require as VNET and minimum 4 dedicated Subnet. (Review Azure Document before going ahead). Plan is well ahead. You can not mix up this subnet with anything else because these are managed subnet thereafter.
ISE will required integration service account to be provision for the service you want to connect example: Blob will require Azure Storage Integration Service Account etc.
ISE has it’s own cost. Run your simulation on Azure Calculator Prior. Study well and use ISE only as last resort. Sometime, solution such Managed Identity etc. or moving LogicApps to different region can help solve.
ISE can give you extension to your on-premise system via ISE Connector example: a LogicApps to be invoke based upon any Sales Invoice Process inside SAP running on-premise.

Scenario I tested;

Existing On-premise system were drop log file inside private blob. Soon after file gets drop, LogicApps needs to trigger and fetch file from Read Container–> Sent Email Notification–>Create New Blob in Write Container –> Notify Subscriber on file data parameters.

Twist was Private Storage Account. So as I described above, LogicApps with ISE was being provisioned. Normal LogicApps could not reach out to Private Blob (Using Managed Identity there is still an option but it may not full fill enterprise requirement of providing connectors).

With ISE, LogicApps can read Private Blob Container via Private Link Service provisioned.

Private Link DNS configuration, which gets provisioned.

ISE can be seen added to LogicApps

Integration Account in ISE for Logic App calling

LogiCApplSE Integration accountS
Integration account

Azure Blob API connection

Managed Connector started Showing

Empty Subnet for ISE: 4 dedicated subnet required to be available for ISE.

Email Notifications for both read and write operation via LogicApp from the Private Blob Container

Akash Kumar
o
vpn -01 a55e2b54cb67c87. txt
_J0b
Akash Kumar
vpn-01a55e2b54cb67c87.txt Added in READ
New File Added
11:12 PM
11:12 PM
This n
Thu
TO: Yc

At last, ISE is interesting feature. It should be evaluated well and executed. Unlike Function (which runs on top of App Service) LogicApps can only be integrated to VNet using ISE. If there is stronger need, use it. It is cool.

Small effort to learn and share.

2019: personal learning index

January 1, 2020 hereisakash aws, azure, gcp, learning, togaf

I am firm believer that learning should never stop. No one is perfectionist, No one can learn everything, but one must keep on trying hard to learn new things/trend/technology.

On my personal learning index, 2018 was focus on building base for AWS similarly 2019 went for building base for GCP. I have Azure practitioner since 2012, therefore for last few years on Azure i have moved focused from traditional IaaS to PaaS/CICD as well as Data/AI.

Learning is of no use if we don’t apply in day-to-day life. Thanks to my job, where i get almost every requirement as new requirement. Thus, every solution i build not only allow me to use my learning but also push me further to explore and learn more.

I have tried to collate my learning KPI which i have achieved in last year 2019. Using this as based, i shall move forward in 2020. Like in sales target will keep on increasing YoY or QoQ, similarly self-target be it learning or knowledge should keep in increase.

	Azure	AWS	GCP	TOGAF
Platform utilized for learning and off-course a ton of native documentation read (nothing beats that)	Linux Academy, Microsoft Learn, EDX	Linux Academy, EDX, AWS Quickstart	Linux Academy, Qwiklabs, Coursera,	Togaf Online Guide, Udemy course on Togaf
Unique Course Focuses	Designing and Build IOT Solution, Security, DevOps	Developing Solution on AWS (PaaS side), DevOps (CICD using AWS native)	GCP PCA, Hybrid Networking, Network Specialization,
Time Spent in Online Courses (hours)	20+	6+	60+	75+
Labs Attempted	15+	25+	50+	More of day to day Practice learning, specially ADM Guidelines & Techniques
Boot-camp Attended	SAP HANA on Azure (onsite) Designing & Building AI Solution using Azure Cognitive Services		First focus was building base
Certification Achieved	AZ 300 AZ900 AZ103		GCP-PCA,	Togaf 9
Next Steps for 2020	AZ 301	AWS 2 Specialty (prefer Security and Networking)	GCP-Network Professional (this one already failed once & clueless why despite 100% sure on 84% answers) and Security	Enhance Practicality of Togaf in daily operation

My personal favorite has been and will always be ‘labs‘. Unless you get your hands dirty in implementing the solution, you can not learn from Online course or documentations. Thus, i chose platform to learn which can give me lot of labs to performs.

Such matrix help me to keep me laser focused on goal. In fact, i have built a mind-map chart which also helps me to build my learning path without focusing too much too many things.

At last, I am not at all firm believer of certifications. But, unfortunately on this basis, this is what Industry recognize you as expert, only after you are certified on xyz with abc specialty. Therefore, it is wise to have these badges but do not compromise on true learning/knowledge.

I tried to collate things and share it with you all as an experience so that you may try to replicate something similar for your learning journey. Most of you may be already expert on domain, thus find it very basic but it might be helpful for someone which may have not started such journey yet.

Thank You and Happy New Year 2020 !!!

Cloud Gyan

Sharing Thoughts Around Cloud Computing Solutions.

azure

AKS Monitoring At Scale – Part 1 (AMA/Container Insight)

Azure LogicApps with ISE (integration service environment)

2019: personal learning index