Home / IT & Computer Science / AWS / Microsoft Future Ready: DevOps Development, Implementation and Azure Automation / Cost Tracking and Optimisation
Cost Tracking and Optimisation
.
Hello, and welcome back to this video on cost tracking and optimising the cloud. Well, after all the work that we’ve done with infrastructure as code, all the stuff that we’ve automated, it’s just so easy to lose track of the spend, right? I mean you start provisioning stuff, and you start creating machines as you need it. You start trashing them when you don’t need them. But where is the sense of being able to have a spot check to see whether the resources that you’re creating are being consumed or whether there is scope for optimization?
Now traditionally, there hasn’t been any tools available within Azure that would just do an advisory check on the resource consumption behaviour in your Azure subscription and give you feedback. Well, there is now, and it’s called Azure Advisor. Azure Advisor does exactly what it says on the tin. It looks at the usage and the assets and the resources that you’ve created within your subscription and gives you recommendation and advice not only on optimising the spend but from a security point of view, from the high availability point of view, from a network point of view. So we’ll look at we look at one aspect of what it offers. And then we’ll also look at Azure automation once again.
There is a pretty neat feature in there which is introduced to Operations Management Suite– it’s also called OMS– that allows you to set up a job that can run across your Azure subscription and help you track virtual machine shut down and start. So we’ll look at that as well. And then that’s really it.
So let’s talk about Azure Advisor. Azure advisor has four elements to it. The first is availability, security, performance, and cost. So let me flip into the Azure Portal very quickly. As you can see here, I’m in the Azure portal. If I go into More Services and I search for Advisor, I get this blade here. And within the advisor, I can select the subscription. I can seek advise on a specific resource group.
And the benefit here is if you’re using a subscription for too much, then the advice can be confusing, right, because if you’re using that subscription for dev and test as well as production, then the recommendations you’re getting for performance, you might not be interested to apply those recommendations in dev and test. But maybe they’re relevant for production. Selecting the resource group kind of narrows down the advisory on the resources that are quite limited to that resource group. But we’ll just now run the engine for the entire subscription because we don’t have too much going on this subscription. So it does take a few minutes to complete, but it starts to give you results right away.
You know, it’s showing up here a roll up of all the recommendations. For example, you can see that there are two recommendations on high availability. There’s one element on security as well. So I click on Security, it’s saying improve the security of Azure resources, and there are four recommendations here. So let’s look at these. This one’s quite severe– enable advanced security for subscription. I’m presuming that I don’t have two factor setup, and I don’t have encryption set up on my storage account, which, if I was storing production category data or maybe highly secure classification data, then that would not be a good thing.
But yeah, I mean, some of the recommendations that you get, again, you can assess whether they are relevant to the workloads that you’re running. And the advisor becomes smarter as you kind of take the recommendations and react to them. Look at high availability. It’s telling you that the nodes are not set up for fault tolerance. Again, these are dev tests workloads, so I’m not really interested in load balancing or making them fault tolerant. But if this was a production setup, then that would be a recommendation I can use right off the bat. You know, surprisingly, there are no recommendations on performance and cost.
Specifically on cost, there aren’t because this is a paid subscription I’m using, and I’m very diligently switching off all the virtual machines at the end of the training and at the end of the day. So no recommendations on that front. But if this was an enterprise subscription where workloads were not probably being killed or switched off, then I would expect to see some recommendations here as well. All right, so that was a quick walkthrough of the advisor recommendations. I’m going to quickly drop back into the slides and show you the Azure automation start-stop VM functionality here.
What it does is it leverages the skill platform that Azure automation is and some of the insights that the operation management suite already gathers by looking at the virtual machines that are running in Azure subscription. And it’s able to just hook up to the machines and offer you a capability where it can switch them off and switch them on. But if you have too many virtual machines, you’re not going to go into each virtual machine and check in the log whether it was switched off or switched on or whether the switch off failed because of a certain reason. So what this functionality does is it kind of rolls that data up into a single tile, right?
From within one tile, you can see the activity across the estate and then drill in into the specifics. And if they’re failures, you can get to the logs. But at least you start at a very high level. So let’s look at that. To set this up, I’m going to go back into the Azure portal, and I’m going to go into the accounts that I created, the automation account. You see I’ve got a few options here. Now in order to enable OMS, we’ll add the accounts. In this case, we’re adding a new Azure automation account. Let’s specify a name. Since we’re creating this for OMS, let’s just have that reflected in the name. So call it IAC demo, OMS demo 03.
We put it in the same subscription. Let’s create a new resource group, and let’s select the location as is. So click Create. All right, so the account’s created. If I refresh, I am able to try refreshing it again. Now I’m able to see the IAC OMS demo 03 account. Its provisioned. It’s there. The next thing we’re going to do is go and enable this automation solution that will basically switch off machines across the estate. In order to do that, I need to come up here and click the green icon. Scroll down to the intelligence and analytics section. Click on see all and search for virtual machine.
Scroll down. Look for– there we go– start-stop VMs during off hours. so click on that and click on Create. At this point, it gives you an option to create an OMS workspace. Now OMS workspace is effectively a working area for Operations Management Suite. Not only does it use that space to do the connections, but it also gathers the logs from the nodes that it’s applying the automation on. Now the benefit here is if you connect all your nodes up to a single OMS workspace and the logs are gathered, then you can perform log analytics on top of it. So you can just launch the browser in the OMS workspace and start searching for keywords.
Behind the scenes, it’s gathering the data. It’s finding correlations in the data. While the integration between OMS and Azure automation in this very example is very limited, but it does open up the new possibilities for using OMS for the other features that it offers. And if you have used System Centre in the past, then you’re going to find a lot of familiarity with the OMS solution. So I’m going to click on Create a New Workspace, and let’s call this workspace IAC demo OMS 03 or maybe IAC OMS demo 03. That’s better, yeah. And you just need to be careful to put it in the same resource group as to where we provisioned the Azure automation account.
So we provisioned the Azure automation account in IAC OMS demo 03 resource group. So let’s put it in here. Select the location as West Europe. I’m going to change the– I’m going to keep the pricing tier as per node. Click OK. Click OK here.
All right, so the validation is successful. We now get the option to select the automation account. Before selecting the automation account, I’m going to make one change to that automation account. When I provision that automation account, I provisioned it as a free account. And since I’ve selected OMS to be a paid account, they’re not going to work together. So I’m just going to open up the Azure Automation account and change the pricing tier from free to basic. And the reason I’m doing that is because in my subscription, I’ve exhausted the free minutes as we have been doing a lot of demos with PowerShell runbooks.
But hey, if you just wanted to try it out, it’s free to try out until you burn out the free 500 minutes that you get per month. So let’s brows back to the actual portal and then select automation accounts. Look for IAC OMS demo 03.
Scroll down to the section where you have pricing and usage. Click on free. Change it from free to basic. Click OK, and now we’ve upgraded it. So with that done, let’s come back to the Add Solution screen. Select IAC OMS demo 03. And we just need to configure a few parameters. So this section here, the target group name, gives you the ability to apply this automation to either one resource group or multiple resource groups. Now since I’ve selected the option star, this applies across the estate to all the resource groups. Now if you were doing this in an enterprise, you wouldn’t select star.
You would probably call out some of the resource groups because in my subscription, I’m only running dev test workloads. I want this to run across the estate. The next option you have is the VM exclude list. Now if there’s certain VMs within the resource groups that you don’t want this to apply to, the switch off and switch on, then you can list them here. Comma separate them, and in that case, they’ll be excluded from this exercise. Now you have to set up a schedule. Now while the automation solution says switch off, switch on in off peak, you define what the off peak is for your organisation.
So in this, as you can see on the screen, the schedule is currently set to afternoon.
So I’m going to change that to zero 8:00 o’clock AM.
In the evening, switch off at 0 7:00 PM. And that’s why it makes more sense to set it up at the resource group level because depending on the workloads that you run in resource groups, your preference on schedules would be different. So click OK. It’s just going to do a round of validation. And if I click Create, it’s going to start provisioning the OMS workspace and integrated with the Azure Automation account behind the scenes. The first step really is to validate that the parameters that we’ve specified here are valid or not. So it’s just going to work its way through behind the scenes. And by the looks of it, it’s completed. That was quick.
So let’s now click on the automation account and come into the IAC OMS demo 03. As you can see here, not much has changed, but the integration has been successful. It usually takes a few hours to kind of sync everything up. And because I’ve selected all the resource groups to be applied, it will take a little bit longer to accomplish the background integration. But if I scroll down, I should now see that the solution section is enabled. Previously, it was greyed out. And so the new OMS workspace we created, that’s now linked up here. So if I click the unlink option, I basically will detach the OMS, the operations management workspace that I’ve created from Azure automation account.
So we’re going to click solutions, and it’s still onboarding the solution. So it doesn’t reflect here. But if we give it 5, 10 minutes and come back to it and just click on the Solution page again, we will see that the solution of switching off and switching on Azure VMs would start to show up here. So we’ll pause and come back and see when it’s available in a few minutes. So if I click Refresh here you would see that the asset count has gone up from 20-odd that it was to 41 now. That just means it’s created more variables to record the name of the assets that it needs to work on in the background.
And it might fail for some of them because some of the resource groups I have within my subscription are kind of stale. But it will generally succeed for most of them. So coming back to the Azure automation account now, if I scroll down to the section where I have the option for Related Resources and click on Solutions now, I do see the start-stop VM solution imported here. And if you click on it, it takes you right into the almost workspace that’s associated to it.
And as you can see here, you’ve got some options here to pick and choose the virtual machines where you want to connect the OMS solution with the virtual machine so that you can get the log, get the log analytics piece going, capture some of the other insights, you know, like performance usage from within the virtual machines. It’s capable of also connecting with storage accounts. And I think that’s quite useful because if you’re using block store or a table storage there, then it’s useful to get log files and analytics from there as well. As you can see where, it says the solution resources.
There are four solution resources– start-stop VMs, start by resource group, and there are a couple other in there. But in summary here, it’s performing an assessment of the estate at the moment. And the first time around, it takes a couple of hours to complete. But once the assessment is completed
and the clock hits 7:00 PM, it will run the job and record the output of the switching off and switching on and then roll that up at the summary tile here. But we’re not going to wait for four hours to figure out whether it worked or not. But you get the gist of where and how you need to enable it and where you need to see the results.
So with that, we’ve kind of covered a couple of key things in this video, we’ve talked about Azure advisor, which I think is just the beginning of something great, where it can learn from the resources that you’re creating within the subscription, give you recommendations which are just not compute-related but network, security, high availability-related. So you don’t need consultants like me to come in and tell you that your setup isn’t high availability. Azure Advisor can well do it for you. The other thing we looked at is the automation solutions that are available and the extensions to it.
So if you think about it, the start-stop VM automation task which we picked up from the marketplace is really an extension to the automation account and also bridges the gap between automation account and OMS where traditionally, they both worked in isolation but now it provides a bridge between the two.
During this course, we’ve explored and worked with Infrastructure as Code and Automation. In reality, it’s so easy to lose track of your spend within your environments.
Keeping track of the resources being consumed and optimising your environments can become messy when you create and provision machines as you need them, and trash them when you don’t need them.
In this step, we’ll discover how Azure Advisor can help you optimise your environment by checking your resource consumption and giving you feedback. We will also look at Auto-Shutdowns and AutoStart for Virtual Machines to minimise waste and control costs.
Optimisation with Azure Advisor
Your resource configurations and usage telemetry can be analysed to optimise Azure deployments with Azure Advisor. Advisor acts as a consultant to recommend best practices to improve cost-effectiveness, high availability, and security of your resources.
You can access Advisor through the Azure portal by selecting Browse, and then scrolling to Azure Advisor. The Advisor dashboard will show personalised recommendations for a selected subscription, which are divided into the following four categories:
- High availability, to ensure and improve continuity of business-critical applications.
- Security, to detect threats and vulnerabilities that could lead to potential security breaches.
- Performance, to enhance the speed of your applications.
- Cost, to optimise and reduce overall expenditures.
Auto-Shutdowns for VMs
To minimise waste and control costs, virtual machine (VM) auto-shutdown is available to all Azure Resource Manager-based Azure VMs and to VMs in Azure DevTest Labs.
After you have created them in DevTest Labs, the reusable templates can be used everywhere. To set this feature:
- Go to your VM blade in Azure portal.
- Select Auto-shutdown in the resource menu on the left side.
- You will see an expanded auto-shutdown settings page where you can specify the auto-shutdown time and time zone. You can also configure to send a notification to your webhook URL 15 minutes before auto-shutdown.
You can now configure auto-shutdown for a single lab VM in addition to the auto-shutdown policy that applies to all the lab VMs.
VM AutoStart on DevTest Labs
In addition to scheduling automatic shutdown for the VMs in a lab, you can use Azure DevTest Labs to automatically start selected VMs at the scheduled time. Along with setting the time, you can set the days of the week for the auto-start.
The auto-start policy requires you to explicitly select a VM and apply the policy from its context menu after you enable the policy. That way, you won’t easily run into the situation where unwanted VMs are accidentally auto-started and cause unexpected spending.
Start/Stop VMs During Off Hours
In Azure Automation, a new feature has been released to Preview that starts and stops Azure Resource Manager virtual machines on a user-defined schedule and also provides insight into the success of the Automation jobs that start and stop VMs with OMS Log Analytics. To configure the solution in your Automation account, see the Solution Components and Configuration.
Once added, a StartStopVM View tile will be added to your OMS dashboard to show how many runbook jobs have started and completed successfully:
In your Automation account you can access and manage the solution by selecting the Solutions tile and then the Start-Stop-VM[Workspace] solution from the list:
Here you can look at important details along with the StartStopVM tile and you can also open your OMS workspace and perform further analysis of the job records by selecting All settings > Quick Start > OMS Portal.
Join the discussion
Do you currently track costs and optimise your environments? How do you track costs? How do you optimise your environments?Use the Discussion section below and let us know your thoughts. Try to respond to at least one other post and once you’re happy with your contribution, click the Mark as complete button to move on to the next step.
This article is from the online course:
Microsoft Future Ready: DevOps Development, Implementation and Azure Automation
Created by
This article is from the free online
Microsoft Future Ready: DevOps Development, Implementation and Azure Automation
Reach your personal and professional goals
Unlock access to hundreds of expert online courses and degrees from top universities and educators to gain accredited qualifications and professional CV-building certificates.
Join over 18 million learners to launch, switch or build upon your career, all at your own pace, across a wide range of topic areas.
