Production Environments

So, what I wrote last night got me thinking.

When did things start to go wrong at my last job? When it was implied that I shouldn’t be touching the production environment. I wasn’t told directly not to do it, but it was hinted at very strongly. And I know that the folks that WERE suppose to be doing it got in trouble for asking for my help. $:\$

NI had a weird priority inversion where operations was something you got promoted OUT of, rather than INTO. I started out in ops, as everyone does there, but then got promoted out of it. I was “rewarded” with getting to work on project work rather than work in ops. I kept working operations kind of stuff, because that’s what I liked, anyhow… and that got other people upset. So I backed off. Way off. And then I got unhappy with my job.

That never really clicked till just now.

I think part of the problem is that my team was several times larger than it needed to be. When I left there were 11 people on it, and our environment was about 1/100th the size of the one I work on now. (And we have less people.) We needed work for the younger admins to do. By me jumping in there and doing it, I was taking learning experiences away from the green admins. Or something. I’m not really sure. I like teaching and would have taught if asked…?

Anyhow!

The things I learn by just explaining! 🙂

Things are Pretty Okay!

Looking back at my most recent series of posts I was little embarrassed at how I only seem to write about bad stuffs! I guess that’s correct, most of the time, because lately I’ve been using writing to work out emotions.

But it kinds gives a bad impression of me! I am not an unhappy person at all! In fact, things are actually going pretty well. 🙂

I am very quickly settling into my new life here on the West Coast. My budget is finally starting to recover. I’m not there yet, but I can see the light at the end of the tunnel, at least. It’s still going to be tight for the next 12-18 months or so, but after that, things should start getting better. I’m very happy about this.

Transition stuffs are going very well. I’m still not up to the full dose of HRT yet (my doctor starts you off slow and works you into it, to make it as easy on you as possible), but I think we’re getting close. Every four weeks I go into the office and she takes some blood work to see how I’m doing. My meds are adjusted based on the results of that blood work. It’s a lot more scientific than I thought it’d be, and that’s pretty cool. (And quite comforting! Yaaaaay for data driven decision making!) Even more of a surprise is that she does NOT monitor hormone levels directly – she uses my Prolactin levels instead. As of three weeks ago I had finally left the male range, but I’m still a ways away from the normal female range for a non-nursing female of my age. This stuff takes years, after all. 🙂

Today I wore a dress outside of my apartment for the first time in my life. I figured I’d just test the waters and try it when I went to make my breakfast run (there’s a bagel shop I go to so much they greet me by name when I walk in). I figured it was a really safe place to experiment with. Well, that went so well that I didn’t change clothes the rest of the day. It’s an amazing feeling to me! Both in how people react to me (I’m no different than any other lady in the place), and that I’m so comfortable with myself at this point. At lunch I accidentally spilled my drink and the manager at Taco Bell was all like “Oh, it’s okay Miss, we’ll clean it up! Here, let me get you a new drink!” It was a nice feeling!

Last Friday I wrote a skirt to work, again for the first time, and it also was a total non-issue. I even went on the Muni train to see my therapist and took the ferry to and from downtown San Francisco… looks like the thing that was holding me back was me… which is kinda normal for me, I think.

The emotional side of this is very different than I figured it would be. Both in the timeline and how strong it is. I’m already becoming a very different person. Here’s an example… last week I went to see a foot doctor about a constant pain I’ve had in my right foot for many years. (It was never a problem, really, until I moved here and started walking a lot.) I showed him the back of my shoes and how it’s all torn up inside and he was kinda shocked. After that he looked at my feet and quickly determined that I have Achilles Tendon Bursitis on my right foot. He mumbled something about “need to get a drop of steroid in there” and told me to flip over on my belly, which I did.

He then injected my foot with a cortisone shot. OMG did that hurt. Like, really, really, really hurt. I was hurting so bad that I just started crying right there in the office. I was overwhelmed with a feeling of “I need to cuddle something,” but I didn’t have a plushie with me, so I just cuddled with my purse. (It was the only thing I had with me. I didn’t know this was coming, or else I would have brought a friend with me.) The doctor could see I was hurting (duh), and was like “It’s okay sweetie. Take as much time as you need to recover. When you’re ready, get your shoes back on, and let me see you walk around the office a bit so I know you’re okay.” So I sat there in the doctor’s office, cuddling a purse and crying for a few minutes. It wouldn’t have gone down that way before… I would have just walked it off. Nor do I think it would have hurt so much, testosterone is good at covering up things like that.

I am _totally_ not complaining here! I like the new me! Very very much so. It’s just very different than before, and I know I’m roughly three months into a 24 to 36 month thing… so there’s still a lot more to come.

My new job is going great, too. We just wrapped up a really big project that’s pretty much been my whole world at this company. (It kicked off right as I was getting there.) I’m excited to see what comes next, as there’s a lot of things coming in the future, and all of it is going to need support from Ops in some way.

I talked with my team lead for a few minutes this morning. I said that when I knew how big of a project this was going to be (a few months into my employment), I decided to step back and focus on the things I’m good at, and then pick up other holes in my knowledge later. My goal was to take the day-to-day heat off the rest of the team so they could work on the project work. He then told me something that, to me at least, is a really big compliment. He said that one of the things they saw in me during the interview process is that I’m a really good Production Engineer. They knew they’d be able to drop me into the Production environment and thrive right away, which I have done. The holes in my knowledge are over project work – things that happen before something gets to production – but the actual part of running services in production is my thing.

I’d never really thought about it like that, but yeah, that’s exactly right. I do thrive in a production environment. I understand very well that my customers are the company’s customers, and that we want them to give us money… and they do so with the machines I’m running. I kinda find some project work to be really boring, while other people thrive in it. (And other people hate the pressure of working on the front lines!)

Needless to say, I know without a shadow of a doubt I’m in the right job and that I love what I do. It’s a small company, so who knows how long things are gonna last, but for right now, while the band is together, I’m really enjoying it. I’m doing my best to savor it while it lasts, because it may not last for forever!

I could go on and on here. 🙂

Lots of things are going very well. I’m sorry that the last few months of posts were completely negative… it’s not actually like that! I guess things are so good that when bad things come along they seem reeeealllllly bad and I turn to writing to work out my emotions.

Lop lop lop! 🙂 ❤

…and we’re live!

We’re live!

This past weekend was the GoLive for the new systems at work. We updated the following:

New Physical Hardware
New RHEL Version
All OS Instances are now VMs in VMWare
Databases updated to the latest version of Oracle
A few DBs went to RAC
Latest version of Apache
New “routing” system where all web requests hit an “edge node” before being proxied back to the application cluster where they belong
Migrated J2EE containers (from an old one to a new modern one)
New application deployment process
Updated FTP servers
New Load Balancers
Updated systems monitoring software
Our developers had to port over 200 applications from the old app server to the new one — a HUGE undertaking!

Sooo many changes. I was at work from 7:00AM till 11:00PM on Saturday, but that was it! We were scared we were going to have to work for days closing out all of the Sev1 issues, but we were done in less than 24 hours! (I was suppose to be the day shift — thankfully it never happened!)

Around 9:00PM we sorta realized that if we kicked it into gear we could finish up everything in one shot, so that’s what we did. The last few hours were a madhouse, but at the end of the night it was all done.

Today was the first day of having the full production load, and the day was very very boring. No issues on the systems side! (One big software issue came up today, which has already been patched.)

There are some lingering Sev2 and Sev3 issues that will go out in a week and a half, but for as many changes as we had… to have no Sev1 issues less than 48 hours later is amazing! (In the IT world a Sev1 is an issue that has so much impact to the business that it needs to be worked 24/7 — which we were prepared to do.)

This was by far the biggest project I’ve ever worked on. We had over 100 people on this team – I was one of the leads on the systems side.

Yaay! Alll that hard work paid off!

Where I’ve Been

I thought I’d take a few minutes and explain why it seems like I’ve fallen into a black hole as of late. Just to warn you, this is a pretty technical post and you won’t hurt my feelings if you skip it. 🙂

For the folks that don’t, in real life I am a Systems Engineer. More specifically I work on web technologies as a Systems Engineer. (The title on my business card is “Web Systems Engineer”.) I work for a medium-to-large size company (5,000 employees worldwide) that’s based out of here in Austin, TX. Unless you’re in the test and measurement industry, or robotics I suppose, you’ve never heard of us. I won’t mention the name here since it’s unimportant, but if you’d really like to know contact me directly and I’ll fill you in.

The group I work for is responsible for the care and feeding of all public-facing web assets under the company’s main public-facing URLs. ie, if you can get it to it as someone that doesn’t work there, odds are, we own it… or at least we’re the ones that keep it running. When I say I’m “fixing something that broke on the Internet”, that’s exactly what I mean.

We’re a mostly Linux shop. Before coming on board I worked with NetWare, Windows and Linux all at the same time, but this job has allowed me to just focus on Linux. My bash scripting skills are way stronger than they’ve ever been.

One of my hobbies is to write web-based applications. Things like RabbitStats, and the registration system for RCFM. It’s just something I do for fun, but I do almost all of my fun programming work in Java and use the J2EE stack. I don’t write Java code at work, but our website is written using J2EE technologies… so I’m a very good fit where I am. I’m a Systems Engineer that understands Java and the J2EE stack really well since I use it a lot in my personal time. (I do write code at work, a lot of it, but mostly bash shell scripts.)

Right now I am on a project to replace almost all of the infrastructure that powers our website. We’re currently using Oracle Application Server (OAS), which is old. We’ve simply outgrown it, and it’s a dead product anyhow. We’re hitting the limits of what a 32-bit JVM can do, etc… it’s just time to move on to the 64-bit world, which we’re doing. My current project is being on the core team that’s moving us from OAS to WebLogic.

This has been a pretty big undertaking. Our web code has had many years of organic growth we’ve had to unwind – one of the key parts was just figuring out what we had to port in the first place! To make it even more interesting my company mostly hires kids directly out of school with little real-world programming knowledge, so the code we have running is often quite interesting. (To put it mildly.)

My role has been two-fold. First, I handle application deployments. I wrote a series of scripts that take code out of our version control system and deploy it into WebLogic. Programmers will check applications (and all of their parts, like properties files) into the version control system and then give us a path where we can find it to deploy it. This, of course, is all done in the name of production control, which is critical as always.

The second role I’ve been doing is coming up with the main Apache web server configuration. This hasn’t been easy as we have hundreds of domains that were all grown organically over time. I’ve had to peel the onion and come up with a way to make the configuration clean and easy to manage. Just like the Java code the Apache configuration is now checked into the version control system, and then I wrote another series of scripts that syncs out the configuration out to the edge servers all automatically.

We’re also changing the Apache configurations on the fly as applications are deployed (mapping J2EE context roots, for example). All of this is done via scripts and there’s little manual work involved… now it’s all running pretty smoothly, but it was a lot of work to get here.

Our company, like most others I believe, goes though a series of critical tests before taking any new systems and code changes live. We call these “CRPs”, or “Conference Room Pilot”. I’m guessing that’s a fairly standard name, but this has been my first exposure to it. CRP1, which is what we just finished up, is a test of where we are and what bugs need to be fixed. CRP2, which is coming up after the first of the year, is suppose to be pretty much production-ready code that the business is testing and signing off on as having the correct functionality.

After CRP2 comes a “Mock GoLive”, where we actually deploy all of the code to see how long it takes and how our documentation and procedures look, followed by a “GoLive” where we actually take the new stuff into production. After all of the rehearsing and testing we do generally GoLives go pretty smoothly. If they don’t something went fairly majorly wrong.

So anyhow, the last two weeks have been CRP1 for this project, and it’s been brutal. There’s still a lot of functionally missing as we wait for the developers get all of the code migrated over to the new platform. (My team, the infrastructure group, is actually doing really well. Our stuff worked pretty well, it’s the developers that we’re waiting on.) Last week the developers were required to work till 8:00pm three nights, and we stayed late too to help them out as needed. (Code deploys, debugging, whatever.) I didn’t HAVE to stay late like that, but it’s the right thing to do since we’re all a team.

The few weeks leading up to CRP1 was all about getting the systems ready for the developers to use, including setting up the dev servers, getting software installed, writing scripts, etc. All in all I’ve just been working my tail off.

I’m really looking forward to the next few weeks. Most of the company is on vacation, we’re between CRPs and it’s just hopefully some time to relax as we get ready go to into CRP2 and Mock. I can’t wait to catch up on my sleep and just do things other than thinking about work as I’ve been working 60 – 70 hours a week the last few weeks during the runup to CRP1 and then CRP1 itself.

Sorry I’ve most likely bored you to death. I know this was a really technical post that’s rather out of character for me, I just wanted to share why I’ve been gone so much. 🙂