I thought I’d take a few minutes and explain why it seems like I’ve fallen into a black hole as of late. Just to warn you, this is a pretty technical post and you won’t hurt my feelings if you skip it. 🙂

For the folks that don’t, in real life I am a Systems Engineer. More specifically I work on web technologies as a Systems Engineer. (The title on my business card is “Web Systems Engineer”.) I work for a medium-to-large size company (5,000 employees worldwide) that’s based out of here in Austin, TX. Unless you’re in the test and measurement industry, or robotics I suppose, you’ve never heard of us. I won’t mention the name here since it’s unimportant, but if you’d really like to know contact me directly and I’ll fill you in.

The group I work for is responsible for the care and feeding of all public-facing web assets under the company’s main public-facing URLs. ie, if you can get it to it as someone that doesn’t work there, odds are, we own it… or at least we’re the ones that keep it running. When I say I’m “fixing something that broke on the Internet”, that’s exactly what I mean.

We’re a mostly Linux shop. Before coming on board I worked with NetWare, Windows and Linux all at the same time, but this job has allowed me to just focus on Linux. My bash scripting skills are way stronger than they’ve ever been.

One of my hobbies is to write web-based applications. Things like RabbitStats, and the registration system for RCFM. It’s just something I do for fun, but I do almost all of my fun programming work in Java and use the J2EE stack. I don’t write Java code at work, but our website is written using J2EE technologies… so I’m a very good fit where I am. I’m a Systems Engineer that understands Java and the J2EE stack really well since I use it a lot in my personal time. (I do write code at work, a lot of it, but mostly bash shell scripts.)

Right now I am on a project to replace almost all of the infrastructure that powers our website. We’re currently using Oracle Application Server (OAS), which is old. We’ve simply outgrown it, and it’s a dead product anyhow. We’re hitting the limits of what a 32-bit JVM can do, etc… it’s just time to move on to the 64-bit world, which we’re doing. My current project is being on the core team that’s moving us from OAS to WebLogic.

This has been a pretty big undertaking. Our web code has had many years of organic growth we’ve had to unwind – one of the key parts was just figuring out what we had to port in the first place! To make it even more interesting my company mostly hires kids directly out of school with little real-world programming knowledge, so the code we have running is often quite interesting. (To put it mildly.)

My role has been two-fold. First, I handle application deployments. I wrote a series of scripts that take code out of our version control system and deploy it into WebLogic. Programmers will check applications (and all of their parts, like properties files) into the version control system and then give us a path where we can find it to deploy it. This, of course, is all done in the name of production control, which is critical as always.

The second role I’ve been doing is coming up with the main Apache web server configuration. This hasn’t been easy as we have hundreds of domains that were all grown organically over time. I’ve had to peel the onion and come up with a way to make the configuration clean and easy to manage. Just like the Java code the Apache configuration is now checked into the version control system, and then I wrote another series of scripts that syncs out the configuration out to the edge servers all automatically.

We’re also changing the Apache configurations on the fly as applications are deployed (mapping J2EE context roots, for example). All of this is done via scripts and there’s little manual work involved… now it’s all running pretty smoothly, but it was a lot of work to get here.

Our company, like most others I believe, goes though a series of critical tests before taking any new systems and code changes live. We call these “CRPs”, or “Conference Room Pilot”. I’m guessing that’s a fairly standard name, but this has been my first exposure to it. CRP1, which is what we just finished up, is a test of where we are and what bugs need to be fixed. CRP2, which is coming up after the first of the year, is suppose to be pretty much production-ready code that the business is testing and signing off on as having the correct functionality.

After CRP2 comes a “Mock GoLive”, where we actually deploy all of the code to see how long it takes and how our documentation and procedures look, followed by a “GoLive” where we actually take the new stuff into production. After all of the rehearsing and testing we do generally GoLives go pretty smoothly. If they don’t something went fairly majorly wrong.

So anyhow, the last two weeks have been CRP1 for this project, and it’s been brutal. There’s still a lot of functionally missing as we wait for the developers get all of the code migrated over to the new platform. (My team, the infrastructure group, is actually doing really well. Our stuff worked pretty well, it’s the developers that we’re waiting on.) Last week the developers were required to work till 8:00pm three nights, and we stayed late too to help them out as needed. (Code deploys, debugging, whatever.) I didn’t HAVE to stay late like that, but it’s the right thing to do since we’re all a team.

The few weeks leading up to CRP1 was all about getting the systems ready for the developers to use, including setting up the dev servers, getting software installed, writing scripts, etc. All in all I’ve just been working my tail off.

I’m really looking forward to the next few weeks. Most of the company is on vacation, we’re between CRPs and it’s just hopefully some time to relax as we get ready go to into CRP2 and Mock. I can’t wait to catch up on my sleep and just do things other than thinking about work as I’ve been working 60 – 70 hours a week the last few weeks during the runup to CRP1 and then CRP1 itself.

Sorry I’ve most likely bored you to death. I know this was a really technical post that’s rather out of character for me, I just wanted to share why I’ve been gone so much. 🙂