Month: February 2011

Success!

At this time last week I was at work, in the “WSUP War Room” as we pushed the new hardware and software live. I worked from 7:00am to 11:30pm that day.

We basically replaced EVERYTHING, so we were expecting this week to be really really rough. We had been set to work 24/7 for as long as we needed to get our website up and stable again… I was going to be on the morning shift. We were all prepared for the worst – we’d prepped upper level management about what could happen, gotten all ready to basically live at work for a while…

…and then nothing happened. It went totally smooth! This week as very boring, which is awesome, considering how much had changed! There’s been only 20 bugs filed this week (compared to 200 the first week after the ERP system upgrade!), and only four emergency bug fixes pushed to production. (Even a normal release will have 4-6.)

All that work, and it paid off. 🙂

It reminded me of a chat I had with Brody at MFM last year – we get ourselves ready for the worse… calculate all the things that could go wrong… and then “or that could happen” – none of those things happen and it’s all easy in the end.

Such an amazing project. Can’t wait to see what comes next!

…and we’re live!

We’re live!

This past weekend was the GoLive for the new systems at work. We updated the following:

  • New Physical Hardware
  • New RHEL Version
  • All OS Instances are now VMs in VMWare
  • Databases updated to the latest version of Oracle
  • A few DBs went to RAC
  • Latest version of Apache
  • New “routing” system where all web requests hit an “edge node” before being proxied back to the application cluster where they belong
  • Migrated J2EE containers (from an old one to a new modern one)
  • New application deployment process
  • Updated FTP servers
  • New Load Balancers
  • Updated systems monitoring software
  • Our developers had to port over 200 applications from the old app server to the new one — a HUGE undertaking!

Sooo many changes. I was at work from 7:00AM till 11:00PM on Saturday, but that was it! We were scared we were going to have to work for days closing out all of the Sev1 issues, but we were done in less than 24 hours! (I was suppose to be the day shift — thankfully it never happened!)

Around 9:00PM we sorta realized that if we kicked it into gear we could finish up everything in one shot, so that’s what we did. The last few hours were a madhouse, but at the end of the night it was all done.

Today was the first day of having the full production load, and the day was very very boring. No issues on the systems side! (One big software issue came up today, which has already been patched.)

There are some lingering Sev2 and Sev3 issues that will go out in a week and a half, but for as many changes as we had… to have no Sev1 issues less than 48 hours later is amazing! (In the IT world a Sev1 is an issue that has so much impact to the business that it needs to be worked 24/7 — which we were prepared to do.)

This was by far the biggest project I’ve ever worked on. We had over 100 people on this team – I was one of the leads on the systems side.

Yaay! Alll that hard work paid off!

Moving Day!

Tomorrow is moving day!

When I moved to Texas from Ohio I still owned a house up north – I was having to pay rent and a mortgage at once, in addition to paying to heat an empty house in a cold part of the country for months. It was a very difficult time.

That all changed in July, 2010 when I accepted a low-ball bid to sell my house in Ohio. It was the first offer I’d gotten in seven months, and it might have been another seven months until I saw another one. I lost several thousand dollars that day (I had to take a loan – which I am still paying off to this day – to cover the mortgage at closing).

However! Tomorrow I finally get to move into a much bigger apartment. I’ve been living in a tiny 580 square foot apartment for the last 13 months. This apartment is so small that Moose’s pen takes up most of the living room. The new place is over twice as big! I get my home office back.

I have been looking forward to this day since I moved to Austin. My life is finally getting back on track and things are coming together. Woo! 🙂