Work

The Friday Before a Holiday

by

Twas the friday before a holiday and it seemed like no one wanted to work. A holiday weekend is upon us. The office was kind of dead. I got a bunch of work done. We have at least 3 people starting on Tuesday, and another 2 later in the week. James is keeping busy getting the computers ready. He was out the door right after I said he could go home early. Danny on the other hand seemed to not get the concept of getting out early for a holiday. I thought he felt he was going to miss something if he left.

All in all a good week at work. Busy, but good. I need to tackle some hardware design stuff next week. I didn’t want to think about it after the meeting I had on it Thursday. Just too much to digest. Next week will be interesting…

NASA Operations as a Role Model

by

I have always been fascinated with space, space travel, the stars, etc. I always wanted to be a pilot, but alas my vision does not permit that. I still have a love for all things fast! Because of my love of space, I have always been interested in NASA, the space shuttle, etc. My dad is also into that stuff, so he partly drove my interest. I would never have thought of making similarities regarding the technical work I do and NASA.

A few years back Gus got me thinking about technical operations. How a data center is run. How a corporate network is run. He always made the comparison of run the network like NASA runs a space mission. Granted I will be the first person to say what I do is not NASA, but it is the process that we are trying to mimic and not space travel!

Now at first I was like how do you compare the two? The answer is it is all in the procedure. It doesn’t matter what you are doing (putting a person into space, or launching a website) it is the operational theory that ensures that things go right. Sounds far fetched? Maybe. Maybe you have to manage some sort of technical operations to understand?

The basis for this theory as I see it is the following:

1. Never assume anything

2. Have a minute by minute plan for all tasks

3. Have contingencies for everything (when possible). Also note points of no return, roll back’s etc.

4. If a mistake happens, isolate it and ensure it never happens again.

There is a ton more things to go over, but sitting here those are the biggest things that come to mind. It boils down to never leaving anything to chance.

This type of thinking got me (kicking and screaming sometimes) to put together firm policies and procedures to keep our network operating. Every time we have an issue come up it is, how do we prevent this from happening again? I am learning that Technical Operations is really a way of thinking, not just a job!

I am thinking about it more today because I am in the middle of reading “Failure is not an option” by Gene Kranz. Gus gave me the book over a year ago. I started to read it but got side tracked. For some reason I sat down to read it today and I finished half of the book. It is a fascinating read. I don’t know why I stopped reading it last summer. It makes me want to continue my efforts at “operationalizing” things at work. And everyone wonders why I am such a pain in the ass about documenting everything. I do have a method for my madness! It is all about the process. No one person can hold any bit of information. That is why the NASA guys would pretend people had accidents during simulations to ensure that if the real thing happened, everything would work normally! Like I said, a fascinating read. Back to the book for me!

All Consuming Project

by

The move downstairs is taking up allot more of my time than I would have liked. I am on my way home now. Kai and Co. will make sure the cubes are finished and the cable work is done right.

Now that the contracts are back, our data center wants to schedule the move of our gear to our new cage. I need to call them tomorrow. Hopefully we can do everything and have ourselves up in the new cage by mid June. Then all we have to worry about is IPCC.

On other notes I am working on getting a DC into VMware, still. I got side tracked since I had to goto Kingston. I will demote one of our DC’s tomorrow and bring it online as a virtual machine.

Still waiting to run the active directory upgrade at our data center. this will enable us to have 2003 domain controllers. This will be helpful for Biztalk 2004.

Cube Work

by

The furniture guys are doing there thing today and tomorrow. I feel like the 3rd wheel, but it does look good that we are keeping tabs on what is going on. Don’t you love obligations? I did get some work done today, but since it is cramped in the IS office up here I was unable to get on a windows box and do some of the work I wanted to do. I have my powerbook only on this trip. I didn’t want to mess with my thinkpad until I get a new one, so I am on my mac only this trip. I have done it before on day trips. I can do most of what I need on it, but not everything.

Tomorrow I hope they finish moving the cubes downstairs. they haven’t started yet. They (furniture guys) setup the offices first today. The cubes are tomorrow.

More later…

Travel Day

by

I don’t have many of them any more, but today was a travel day (sort of). I used to consider a “travel day” a day where most of what I did was travel from one place to another. Now that I don’t travel as much (and am much happier for it) I don’t have as many as I used to. Dam, the last time I was on a plane was September. Most of my travel is train travel now a days.

I am at the hotel now. I got hear relatively early. Normally when I stay here (my companies other office) I am usually at work late. Today was different. I needed to be here for 2 days while work went on during normal business hours. That means I got to keep normal hours, well sort of.

I learned that getting to the hotel at 6PM means dinner at 7, 7:30. After checking mail, making calls, etc, I am bored and it is only 8:30. So I sit here on the bed with my ipod listening to toons, while I write on my powerbook. I will try to goto bed early, but I don’t know if that will happen.

I had the surf and turf at the hotel restaurant again tonight. I had it the last time I was up with Jayson. it was good, and only $3 more than the normal steak I get. I had a drink and felt nice and relaxed. I didn’t have more, since it was really early and I didn’t want to be tired and out of it the rest of the night. that was just in case I decide to be productive later.

Storage

by

Jayson is pushing an online disk storage system. He is thinking NAS. This will solve a problem we have with one nights tape backup taking so long is that it gets in the way with SQL server db backups for the next day. We just don’t have enough time in the day to run the amount of tape we need. if we backup to a disk array and then to tape weekly we can increase restore time, and backup times. from an engineering standpoint it is a great idea.

We toyed with it last year, but never got to budgeting for it. Now we think we really need it. I have to see if we can get the money for it. Jay is going to do some more research for me and then we will come up with options.

Construction (Still Going On)

by

I am off to Kingston for 2 days of construction fun tomorrow. We are moving existing cubes downstairs, and putting in the remaining furniture. When I mean we, I mean the furniture guys are coming and I am going to “supervise”. I am going for 2 days since the furniture people think it will take that long. The cable guys are coming out the day after the furniture guys leave to finish up the cable. Once that is done we can move the remaining reps downstairs and be done with this long delayed project.

Fear not, the next 2 major projects is already underway. Turns out we are big enough and have the right people in place that I can run two major upgrades at the same time. My stress level is up there, but we can do it. I have NYC people (myself included) working on upgrades (now officially moving) of our cage to a new location in our data center. Kingston on the other hand is working on IPCC. I am still involved in that, but Kai will have to handle some of the day to day details since he will manage the product after it is deployed. Along the way the dev team has noted that they will be building an entirely new website environment to accommodate the next generation site they are building. That means in addition to what I just mentioned we need to find space for, design, quote, build and then manage new Dev, QA, & production environments. that means servers, databases, web servers, etc. All this while we maintain our existing equipment. It will be an interesting project.

Stuff

by

I had to deal with fixing contracts for our cage upgrade. Bob finally signed off on them at like 5pm and I faxed them back to our provider. Now we wait for a completion date.

I also worked on moving AD (active directory) roles around on our servers. This will enable me decommission a server and move it into a virtual machine. We have an old Dell 42xx series server that is just taking up space. It barely runs as it is, so it is a good idea to make it a virtual machine and free up precious rack space. Yup, still loving VMware!

Thursday night I stayed late moving print servers, and FMSO roles around. The testing of the log-on scripts with the printer changes took longer than I hoped so I didn’t get home till like 8:30PM.

Danny was worried about disaster recovery issues with moving more gear to virtual machines. I downloaded the white papers on the subject from vmware and will read them on the train to Kingston Monday. I am off to upstate for 2 days for furniture construction.

Travel Day Complete

by

I am done with my day of travel. Amtrak sucked. The trains were older and in worse shape than normal. When I asked the conductor he said that the normal trains are in use on Acela since those trains were down for maintenance. Something about the “elite” route gets the good trains! The train was late 20 minutes going up, and 45 minutes late coming home. I should be used to it by now, but I am not. On a plus note Amtrak’s automated IVR was awesome. I understand why they won all sorts of awards for the voice responce system. Hey I deal with telecom, network and VOIP gear most of my day, so hearing a good IVR actually interests me now. Some people look at nice cars, I admire IVR’s. No one ever said I was normal.

We got 10 users downstairs. Kai had issues with the other group of 10 they wanted to move. Turns out the electrical guys didn’t plug in a conduit so a bank of stations didn’t have power. that set us back enough that we didn’t have time to finish moving users today. They should still meet the deadline of friday to have half the users downstairs.

I also dealt with issues regarding our upgrading our cage at our data center, and canceling circuits in NYC. Also design woes about upgrading our circuits in our call center. I have been busy with our telecom provider to iron out all the issues. More meetings with them tomorrow!

A New Call Center, Well Almost

by

Tomorrow (if all goes well) my company will open its new floor of its call center. We are moving our reps down to our new floor (with tons of space for expansion) and corporate people (those that are in that office), escalations groups, etc. will remain upstairs. Tuesday marks the first day we bring reps down. We are taking it slow with only 1 team of 10. By the end of the week we will ramp up half the call center or more down there. Then we move some existing furniture down that is currently in use. Cable, and bring the rest of the people down. It has been a logistics challenge. Not quite a nightmare, but challenging.

I don’t “have to” go tomorrow, but I wanted to see the end result of all that work that I (and others) took part in. I will most likely be back for a few days next week as the construction people do the cubes and cable for the second half of the center. Dan will be on vacation and even though I am a “technical guy” I am the next person who is aware of the entire project.