A luser's guide to Iometer

Hi, my name is Dan, and I'm a software development luser who sometimes dreams of being a sysadmin. Actually, what I really want is the authority to pick all the cool hardware, but with none of the "the server's down!" accountability. If you hear of that job, lemmeno. In the meantime...

If you haven't lived under a rock during the last 5 years, you're aware that virtualization is taking over. This is a good thing. The trouble is, supporting virtualized environments requires a completely new skill set. The ROI is there, but it takes a real knowledge investment! And if you install virtualized servers in a company with overworked and stressed out BOFH's that aren't able or willing to make that learning investment, you can end up worse off than before.

In this particular case, I installed an instance of JIRA and Confluence for a client as part of our big development project. I've installed it before at a couple other places, and performance has always been really good. But at this client, it sucks. As in 10-second-response-time sucks. The instances are running on the VMWare ESX platform, which is the key variable that's different. The trouble for me as a developer in this problem is that:
a) I'm not a VMWare tuning expert.
b) I'm not a BOFH, so naturally, the infrastructure group hates me. Just kidding, they really like me...I know that knife I found in my "Let's deploy EJBs and crush the souls of our poor operations staff" book was just for fun.

But seriously, what we had here was a real failure to communicate. I knew that this thing just wasn't running right. But I didn't know how to narrow the problem down and quantitatively convince them of the issue. I was able to see that it wasn't CPU bound just by watching the CPU meter. I knew it wasn't the database, because I tried a couple different databases, each with the same performance. During my local testing on my laptop, the full production instance ran great. So I was kinda sure that it was I/O bound, but I needed proof.

Enter Iometer. Iometer is an open source tool, originally written by Intel, and open-sourced in 2001. It's become one of the most popular storage performance and benchmarking utilities available. Even VMWare recommends it as a tool you can use to analyze storage throughput.

I'm still an Iometer n00b, so take what I say with a grain of salt, and teach me if you know more than I do please! e.g. One of the questions I have is this one over at serverfault.

Short story long, the punchline is this: I ran an Iometer test comparing performance between my development laptop (7200 RPM drive) and the virtual server instance running on a SAN. Here's the results:







Pretty amazing, huh? Amazingly bad for the server - it's nearly 3 times as slow as my feeble little laptop! Like I mentioned, I'm not an Iometer expert by any means, and there are different ways to configure the tests. But I'd be surprised if you could configure the tests to find this big of a performance difference between systems that were anywhere near each other's performance.

On how to run the tests - here's one tutorial on configuring it, and here's another. The only tricky things I found:
1. On the Disk Targets tab, there's a setting for the Maximum Disk Size, and it's specified in sectors. It's been a couple centuries since I worked with sectors, but fortunately google is your friend.
2. The user guide is a little confusing about what a yellow disk target icon with a red slash through it means. If it's yellow with a red slash, it just means that IOmeter will need to create its iobw.tst file on the drive when it starts the test - you don't have to do anything. Do select the drive you want to test against though - this will be marked with a little X next to the drive when you select it.

Turns out that diagnosing I/O performance is easier than I thought it would be. Now that just leaves the hard part of trying to get our BOFH friends to fix problems like these when they happen...good luck!

How to share google calendar with family and exchange

Update 9/22: Google must've seen my post and reacted :)
http://gmailblog.blogspot.com/2009/09/push-gmail-for-iphone-and-windows.html
-------------

A while back I gave my better half my old iphone. I was all jazzed up to be able to sync our calendars, but it turned out to be not as easy as I thought it should be. I know there's mac.me and all, but coughing up all that money to AT&T for their service makes one want to act like their dad with their furnace thermostats. (Thank goodness for electronic thermostats, where you can wage the battle once and be done with it...but I digress.)

Here was the situation:
I have an iphone, have my work mail (and primary calendar) fed through an exchange server, and use google apps (gmail) for my personal email.
My wife has an iphone, and a gmail account.

The mission: Share our family calendar - allowing me to see family events that show up on her calendar, and vice-versa.

You'd think this would be pretty easy, but anyone that's been screwing around with these stupid calendar syncing apps for the last few years knows that it's actually really hard. You can get pretty close nowadays with google's default sharing, and even with neuvasync, but at least when I went through this experiment, neither of those solutions had implemented the ability to notify the invitees if the person adding the event did it through their phone. See this thread and this one as examples of the issue. Notifications *do* work if the person adds an event via the google calendar website, but that doesn't help if you want to add an event through your phone.

But what I did find to work was the above combo + google sync. If I have that running, *and* have auto-accept turned on in my google calendar settings, any invitations that are added to my google calendar account get automatically sync'd to my exchange calendar, which are then pushed out to my phone...voila - near real-time synchonization. This requires an extra step, and relies on the google sync app running, but it solves that problem. Next up - wondering why she still prefers the dead-tree version...

The Youth Soccer Project Pattern

Have you ever seen a soccer/football game played by 5 year olds?

The Youth Soccer Project Pattern is an observation I've seen, regarding how the members of a project will run around as one big pack, chasing the ball all over the field in committees. In this environment, there also tend to be lots of soccer moms yelling praises at the sidelines (AKA project managers). There might one or two young stars that can make their way through the mess. And there's also bound to be one or two kids that are just hanging out in some corner of the field, picking daisies. In software projects those are often the testers that haven't been properly included in the team, or are those testers who are punching the clock and would rather pick daisies than really get into the action.

Contrast this with experienced teams that have players qualified at every position, and know where they should be at all times. Teamwork consists of short, well executed collaborations. Not to mention the fact that the players have spent considerable time together already, so the osmotic communication is a given. The players know exactly what their goal is. Of course in software, this goal can change drastically based on the situation, but once you're coding, you better know what the end goal is. Otherwise you're just wasting time, and maybe even heading in the wrong direction.

So, what kind of team are you playing with or coaching?

When evaluating design decisions, let the numbers talk

[The title of this article is as much a reminder to myself as it is advice to others.]

One of my clients has a decision to make about when they should use an ESB to expose services in their upcoming architecture.

What's interesting about this decision is that there is a heated debate around it (I say use them only when absolutely necessary, while the company's long-term services company says use them for every public service interface), and this is going to be an expensive decision. It makes for some interesting dynamics in the decision making process.

The conversation became useful when we started talking about concrete ways to compare the approaches for building a particular service. We came up with this rough evaluation criteria:

Category	Tests	Metrics
Performance	1 user, 1 request	response time
Scalability	X users, Y requests per second	avg response time
Initial Effort		Number of hours to implement
Maintainability	Add an attribute	Number of hours to implement
	Change the interface	Number of hours to implement
	Version the service	Number of hours to implement

It's that last category that really intrigued me. It's obviously a difficult thing to measure with all the variables involved, but I think you can get pretty close with some forethought, and maintainability is the key question here. For example, you can eliminate massive developer differences by having the same person implement all variations of the approaches, or at least pick developers that are about the same in skill and speed levels. You'd also want to eliminate (yet still measure) learning curve factors in any approach, since initial learning curves might inappropriately throw off the long-term comparison measurements. You'd still want to measure those, since that's going to be a factor when you need to ramp up another person to support this application.

Ideally there'd be something like the above in a standard format, much like how the Pet Store application is used to compare platform stacks. I did a little bit of searching on this subject to see if there were any other standard operations to consider as part of an evaluation, but didn't find anything worth looking into. What other metrics or tools have others found to be useful in situations like this?

Simple and easy site monitoring with PowerShell

In the *nix world, I really like monit for watching and managing small to medium apps. A while back we wrote an intranet app in Grails, running on Tomcat on a Windows 2003 server. The client was hosting the app themselves, but didn't have any IT staff whatsoever, so monitoring and managing this app was a little tricky.

I searched around for something like monit for windows, but didn't find any good free or cheap tools to do what it does, which is primarily to monitor and restart services upon failure. There's a million monitor and alert tools out there, but none that I found had a clean way to restart my Tomcat service if something went wrong. The closest thing I found was Hyperic's open source monitoring solution, but after 30 minutes of futzing around, it seemed too klunky to me for this small problem. I'm sure it's fine, and for bigger solutions it would probably fit well.

Then I ran across a couple PowerShell scripts that intrigued me - the first was for scraping a web page. After that find, it was just a matter of googling to find the remaining piece - restarting a windows service. Bingo! I hadn't done any PowerShell scripting yet, but it turns out to be a really sweet solution for this type of problem. All I had to do after that was to tweak the code to be a little more reusable, and I also created a simple grails view that would exercise the integration features I wanted to include in my ping test. From there it was just a matter of calling the script on a regular basis. In my case, I used the windows task scheduler. Here's what the script looks like:

$webClient = new-object System.Net.WebClient

###################################################
# BEGIN USER-EDITABLE VARIABLES

# the URL to ping
$HeartbeatUrl = "http://someplace.com/somepage/"

# the response string to look for that indicates things are working ok
$SuccessResponseString = "Some Text"

# the name of the windows service to restart (the service name, not the display name)
$ServiceName = "Tomcat6"

# the log file used for monitoring output
$LogFile = "c:\temp\heartbeat.log"

# used to indicate that the service has failed since the last time we checked.
$FailureLogFile = "c:\temp\failure.log"

# END USER-EDITABLE VARIABLES
###################################################

# create the log file if it doesn't already exist.
if (!(Test-Path $LogFile)) {
   New-Item $LogFile -type file
}

$startTime = get-date
$output = $webClient.DownloadString($HeartbeatUrl)
$endTime = get-date

if ($output -like "*" + $SuccessResponseString + "*") {
   # uncomment the below line if you want positive confirmation
   #"Success`t`t" + $startTime.DateTime + "`t`t" + ($endTime - $startTime).TotalSeconds + " seconds" >> $LogFile

   # remove the FailureLog if it exists to indicate we're in good shape.
   if (Test-Path $FailureLogFile) {
       Remove-Item $FailureLogFile
   }

}
else {
   "Fail`t`t" + $startTime.DateTime + "`t`t" + ($endTime - $startTime).TotalSeconds + " seconds" >> $LogFile

   # restart the service if this is the first time it's failed since the last successful check.
   if (!(Test-Path $FailureLogFile)) {
       New-Item $FailureLogFile -type file
       "Initial failure:" + $startTime.DateTime >> $FailureLogFile
       Restart-Service $ServiceName
   }
}

I posted my question (and my own answer eventually) on stackoverflow here, so keep an eye on that if you're looking for improvements to it. There's a lot more you can do with this script, like email upon failure, or monitor multiple sites. Hopefully this helps some other folks out there with problems like this. It was a really nice experience for me; simple and powerful. So hats off to the authors of PowerShell - I'll be reaching for that tool more and more in the future.

Domain names for daughters

Though the biggest domain name purchases are probably long-gone since the initial dot-com boom, they still remain a valued property. They'll never be as valuable as real land of course, since we can always create more of them, as opposed to things like lake-front property.

The recent mad rush for facebook profile names reminded me of this new global scale of gold rush activity. And yes, I'm a little annoyed that someone that has been farming this interweb as long as I have was offered a name like dan.tanner2 (in my day, we used Mosiac. And we liked it!). It also reminded me of something I'm hoping will come in handy for my two daughters some day - their very own short and memorable domain names.

Finding a decent available domain name is a tenaciously difficult PITA. It's an exercise in creativity and luck, and girls names are especially tough. Not only do you have to wrestle with the general lack of availability, but also the likelihood that they'll change their name when they get married. In my case, that's sometime after age 28, when they've agreed to start dating.

In my case, my girls are named Maria Mae and Daniela Teresa. All good variants of maria and daniela were taken, which would've been ideal. And no last names were considered because of the potential marriage name change. After tinkering around for a while, here's what I ended up saving for them:
mariamae.com and danielat.com
They're both fairly mnemonic, and they're both flexible enough to last. And they're both shorter than my domain name!

It'll be a few years, but hopefully they'll be useful. Not to mention give me a little boost in the cool dad category. Not that I'll need any help in that category.

Now...where did my favorite plaid pants go?

In praise of refreshingly efficient meetings

Everybody dreads bad meetings. Meetings are often too long, too often, or don't accomplish anything. At some companies, this is what they are all like.

You can implement processes to help have better meetings (which is another meeting of course). Heck, I just attended a meeting last week whose title was how to structure our meetings to have less of them.

Like everything else in the thought-based industries, the effectiveness of a meeting is 10% process and 90% people.

The other day, I attended a meeting that was remarkably crisp and effective. It was a brainstorming session, so I wouldn't have expected it. The meeting started with a general leaning towards one solution to the problem, and 15 minutes later (15 minutes early), the meeting ended with a clear path towards the exact opposite style of solution.

I walked out of the meeting thinking, "Wow! That was fun!"
To which I quickly looked in the mirror wondering why the hell I would have just enjoyed being in a meeting. Developers have been bred to hate meetings, so I obviously needed to do some serious soul-searching.

What seemed unique to this meeting vs. so many others were the following attributes:

• An inherent sense of urgency by role: The key person in the room was a company officer with a ton of experience making decisions on a limited schedule. Sure, everybody's busy, and some people are really busy, and have that ability to keep people in the room at attention. Maybe it goes with the title, maybe it's decades of experience, maybe a little of both.
  
• Flexibility in thought (an open mind): There weren't any fixed or hidden agendas, which allowed the discussion to flow quickly through the options without getting hung up.
• Real-time calculations: Part of the discussion involved ROI. We could've marked that data as follow-up items, which would've left the discussion muddy and forced another meeting, but it wasn't necessary. With the business having a good handle on their critical numbers, and the technology team having a good rough estimate for the project size, we were able to do the math in our heads and flip through the choices in minutes.

The simplest written analogy to this that I've seen is the chapter titled "Rattle Yer Dags" in the book Adrenaline Junkies and Template Zombies, by the legendary Tom DeMarco and Atlantic Systems Guild. (Highly recommended.)

And you can follow the guidelines given when you google "how to run an effective meeting". That's all good stuff...the unfortunate reality for most is that meetings like this don't happen often enough.

So, cheers to refreshingly efficient meetings.

Gotta go...I'm late for another meeting.