I have been using Microsoft Azure Cloud Services since PDC 2008 when it was first announced. Ever since, I’ve been a
huge fan of “cloud services”, the cattle VMs in the cloud that are stateless. In all those years, I have never seen this error, until yesterday:
There is not enough space on the disk.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.WriteCore(Byte buffer, Int32 offset, Int32 count)
at System.IO.BinaryWriter.Write(Byte buffer, Int32 index, Int32 count)
Help! Where did that come from! I decided to set up a remote desktop connection to one of my VMs and see if any of the disks were full or near being full. Nope!
The stack trace of the exception told me the exception originated from creating a temporary file, so I decided to check all the obvious Windows temp paths which were squeaky clean. The next thing I looked at were quotas: were any disk quotas enabled? No.
But there are folder quotas enabled on an Azure Cloud Services Virtual Machine!
The one that has a hard quota of 100 MB caught my eye. The path was C:\Resources\temp\…. Putting one and one together, I deducted that Azure was redirecting my application’s temporary folder to this one. And indeed, a few searches and this was confirmed:
cloud services do redirect the temporary folder and limit it with a hard quota. But I needed more temporary space…
Increasing temporary disk space on Azure Cloud Services
Turns out the System.IO namespace has several calls to get the temporary path (for example in
Path.GetTempPath()) which all use the Win32 API’s under the hood. For which
the docs read:
The GetTempPath function checks for the existence of environment variables in the following order and uses the first path found:
- The path specified by the TMP environment variable.
- The path specified by the TEMP environment variable.
- The path specified by the USERPROFILE environment variable.
- The Windows directory.
Fantastic, so all we have to do is create a folder on the VM that has no quota (or larger quota) and set the TMP and/or TEMP environment variables to point to it.
Let’s start with the first step: creating a folder that will serve as the temporary folder on our VM. We can do this from our Visual Studio cloud service project. For each role, we can create a Local Resource that has a given quota (make sure to not exceed
local resource limit for the VM size you are using!)
The next step would be setting the TMP / TEMP environment variables. We can do this by adding the following code into the role’s
RoleEntryPoint (pasting full class here for reference):
Code highlighting produced by Actipro CodeHighlighter (freeware)
-->public class WorkerRole : RoleEntryPoint
private const string _customTempPathResource = "CustomTempPath";
private CancellationTokenSource _cancellationTokenSource;
public override bool OnStart()
// Set TEMP path on current role
string customTempPath = RoleEnvironment.GetLocalResource(_customTempPathResource).RootPath;
<!-- Code inserted with Steve Dunn's Windows Live Writer Code Formatter Plugin. http://dunnhq.com -->
That’s it! A fresh deploy and our temporary files are now stored in a bigger folder.
would you validate a new API is compatible with an old API? While upgrading frameworks in a web application we’re building, that was exactly the question we were asking ourselves. Sure, we could write synthetic tests on each endpoint, but is that representative?
Users typically find
insane better ways to test an API, so why not replay actual requests against the new API?
In this post, we’ll see how we can do exactly this using IIS and
Apache JMeter. I’ve been using JMeter quite often in the past years doing web development, as it’s one of the most customizable load test and functional test tools for web applications. The interface is quite spartan, but don’t let that discourage you from
using JMeter. After all, this is Sparta!
Collecting IIS logs
Of course, one of the first things to do before being able to replay actual request logs is collecting those logs. Depending on the server configuration, these can be stored in many locations, such as
C:\inetpub\logs\LogFiles or C:\Windows\system32\LogFiles\W3SVC1. I will leave it up to you where to find them.
In our case, we’re using Azure Cloud Services to host our web applications. IIS logs are stored in a location similar to
C:\Resources\Directory\<deploymentid>.<solename>.DiagnosticStore\LogFiles\Web\W3SVC<numbers>. If you’d be on Azure Web Sites, logs
must be enabled first before they can be downloaded.
Converting IIS logs
By default, IIS logs are in the
W3C-IIS format. Which is great, as many tools are able to parse that format. Except for JMeter,which works with
NCSA Common Log Format. Not to worry though! Fetch the
RConvLog tool and invoke it on the IIS log file that should be replayed later on.
We’re running RConvLog on our log, which will be converted to the NCSA log format. We’re also providing RConvLog with an idea about the time zone the logs were generated in. Since Azure runs in UTC, we can just tell it it’s all UTC by passing in
Setting up a JMeter test plan
Time for action! After launching JMeter, we can start setting up our test plan. Using the context
Add | Threads | Thread Group menu, we can add a thread group. A thread group in JMeter is what simulates users. We can decide how many users are active at the same time (number of threads), how dispersed requests are (ramp-up period)
and how many requests will be made in total (loop count * number of threads). The following configuration will simulate 10.000 requests with at most 100 concurrent users. Note that when replaying logs, having 10.000 requests means only the first 10.000 requests
from the log will be replayed (if in the next step, the OrderPreservingLogParser is selected). If the log file holds 40.000 and we want to replay them all, we’ll have to do the math and ensure we actually will do that number of requests.
Yes, spartan. I already mentioned that. Next up, we can use the Add | Sampler | Access Log Sampler context menu. We can now specify where our log file that is to be replayed lives. Since NCSA log files don’t hold the hostname or
server port, we can configure them here. For example if we want to replay logs against
localhost, that’s what we’d enter under Server. We’ll also have to enter the log file location so that JMeter knows which log file to read from.
Important thing to change here! The log parser class. JMeter comes with several of them and can be extended with custom classes as well. The three that come in the box are the
TCLogParser which processes the access log independently for each thread. The
SharedTCLogParser and OrderPreservingLogParser share access to the file, where each thread gets the next entry in the log. Let’s pick the
OrderPreservingLogParser so that the access log is read and replayed line by line.
All that’s left is using the Add | Listener | Aggregate Report context menu so that we can have a look at the results. That’s pretty much it. We should now save our test plan so we can run JMeter.
Replaying the logs with JMeter
Clicking the green Run button launches our virtual users and processes logs. The
Aggregate Report will list every request made, show its timings and the error rate.
That’s about it. But there are some considerations to make…
What have we tested so far? Not a lot, to be honest. We’ve replayed IIS request logs but have not validated they return the expected results. So… how to do that? Using the
Add | Assertions context menu, we can add assertions on status code and response contents. That’s great for functional tests, but replaying 100.000 entries is a bit harder to validate… For the test case we’ve opened this blog post
with, we’ve created an Excel file that has the HTTP status codes and response size (they are in the logs) and compare them with the results we see in JMeter.
Maybe we’re not interested in the actual result, but in what is going on in our application? Turns out replaying IIS request logs can help us there, to. For the application we’re converting, we’re using
Azure AppInsights to collect real-time telemetry from our application. We could also use a profiler like
JetBrains’ dotMemory and dotTrace, we can subject our running application to close inspection while JMeter simulates a real load against it.
And of course, we're testing only anonymous GET requests here. While JMeter supports sending cookies and request types other than GET, there are other tools to test those scenarios as well, like