Work

Problem Resolution

by

We had a demo of some really cool problem resolution software today. It can watch what goes on throughout all the servers and clients and correlates a problem for you. It does way more than that, but it is a difficult concept to write about. I am sold, but we need to evaluate it a bit more before we can make a decision.

Work was normal otherwise. Well normal except for the circuit failure we had in our call center computer room Thursday and again this morning. I don’t know who is stupid enough to close the door on a computer room that feels like it is 110 degrees inside while a fan is at the door blowing hot air out of it? Long story. We had some HVAC issues that caused other power related issues. The power issues caused some problems on thursday with our IPCC system. It was resolved by a reboot, but it was fun for a bit while we troubleshooted the issue. Otherwise I have been able to catch up on the backlog of issues this week since my boss was out sick.

Creative Problem Solving With VMware Player

by

Recently we had to deal with a rare request (or rare for where I work). We had a user who has a legitimate request for wanting linux on his desktop computer. The problem is for now we are a Windows XP desktop shop. Several systems staff have Linux or Mac’s but that is kind of unique since they are systems. The user in question wanted to try an odd flavor of linux as either a dual boot or a second desktop computer. This poses two issues for us. One, we have no security standard for end users to have linux. Secondly the flavor of linux he wanted to use was not what we currently use at all.

The solution for me seemed very obvious. Give the user VMware Player, and get them a pre-built Virtual Machine of Red Hat Enterprise Workstation (the linux desktop OS that we use internally). This way we don’t need to have a rouge desktop out there different than everything else we deploy to users. Also if there is a problem we can remove the VM and start from scratch quickly.

The user was not so keen on this idea since they wanted linux as their primary OS. We where solving an individual problem they had in the confines of our standards using VMware Player. So far things are working out just fine. The major issue I have is that the user is resourceful, and he may just go out to the VMNT site and download other pre-built VM’s that we have not authorized. I wonder if there is a way to limit what VM’s can run on a machines VMware Player?

Our long term intentions are to deploy all new desktop and laptop computers with the VMware Player already installed so if we need to do something like this again, we just send the user the VM and off they go. It is amazing how after a few years of using this software, we keep coming up with new ways of leveraging it in our organization. And since player is free, we didn’t even need to buy extra copies of VMware Workstation.

Technorati Tags: , , , , ,

Performance Issues Resolved

by

After like 4 passes at the configuration of our network (switches, firewall’s, load balancers, etc) Danny found an abonormality that we wanted to correct and see what happened. On our core switch side we had the port where our Pix went set to full duplex 100meg, but on our Pix (configured years earlier than our core) it was set to auto. Turns out the Pix does not auto sense the 100meg full, but does not error out in the situation. You don’t even get lost packets, but you do get some collisions. Well some is an understatement.

Later in the day we set the port on the pix to be full duplex 100meg and within a few hours our metrics back to normal. This little change took us weeks to find. This is not the first time I have been burned by a port mismatch. Knowing that we even took steps to prevent this, or we thought we did.

It is frustrating to find such a little issue that does not show with errors causes so much problems..

Rough Days At Work

by

I have had a few rough weeks at work. Hopefully a fix that went into place today will make things better. I know I am getting worn down, and I think others around me are also. We just don’t talk about it. Not much I can really talk about on this blog. I have written more extensively on my Work blog (password protected for my protection).

Because of the extra work, not much else to report on the social front.

Site Issues & Network Disasters

by

I took yesterday off to catch up on some sleep. I ended up by the office in the late afternoon to go with Jayson backpack shopping. Our website performance (or alleged) issues continued on. All of our research boils down to we don’t think there is anything wrong, or shall I say anything new wrong. We know our application needs to be improved. That is why the development team has spent over a year designing and building a next generation application. I think certain business people are missing there numbers and blaming the issue on a site problem. They got one bad metric and they stomp all over it. It has been a long week.

On another note I was just sitting down to figure out what I wanted to eat for dinner last night when I got a call from Jayson that an internal website was offline and someone called about it. Turns out several several servers where down. They where all plugged into the same switch module. I ended up having to go to our data center and meet Jayson to fix the issue. It was as simple as re-seating the module and it started working again. To be safe we moved all critical systems off that module onto another switch module. I didn’t get home until after 1AM today. So for a day off, I worked almost a full day’s work. Nice!!!

Long, Long Day

by

I had a really long stressful day at work. Absolutely nothing I can discuss here. I did write a long entry on my work blog, but since it is password protected and private it can’t be really discussed. Lets just say when I finally left the office it felt like I had ran a marathon. I was just physically and emotionally wiped out. I did get a chance to relax a bit in little Italy with Jayson, Scott, & Gretchen for dinner. They where cool enough to wait for me to finish work before ordering dinner. Thanks guys!

Due to one of the big issues I was working on today I am canceling my trip to our other office tomorrow. I need to make sure things are on track here.

I am still having trouble sleeping. I was up again last night. Not as long as some nights last week but it still sucked. It also didn’t help that I got woken up at a quarter to 1 with a problem at work. I wasn’t on the phone for long but the damage was done. I can’t get pissed about stuff like that because it is part of the job, but man I was actually sleeping. It is so rare these days. I was hoping for an early night tonight but it didn’t happen. I am off to bed now, but I wanted to write a bit before I goto bed. it does make me relax a bit when I write a little.

VMware Saves The Day

by

I have been talking the praises of VMware ever since I was introduced to it a few years ago. This week it moved up a peg in my book. We have been having issues with some production hardware for some time. It is a long story but lets just say that by fixing the machine we could very well destroy it since we have had bad results with similar equipment. In theory the state that the machine was in could remain stable indefinitely, but who was going to risk another drive failing (SATA RAID issue that is unnecessary to go into details about). Rebuilding the box was out of the question since it would take way too long to do so. Since it was an application that is slated to be retired in a few months, it isn’t worth putting in tons of man hours setting up a new box, but we need the app in service. Rolling the dice and hoping the machine stayed up was not a risk anyone was willing to make.

Then the solution came to us. Lets virtualize the machine. We had been testing it for a while and the results looked great. We had even taken some non production box’s and virtualized them for use on one of our new ESX servers, so it looked like a viable option. The issue was convincing our dev staff and my boss that this was what we needed. The dev staff needed some convincing. Now that the work is done they are still worried about some quirk creeping up in the system. My boss was more enthusiastic about the prospect.

The only down side that came of the whole situation was that the Physical to Virtual conversion took triple the amount of time we had hoped or planned for. Was it worth it? Yes, but we could have lived without the hours of waiting.

I am continually amazed at what this software can do.

Technorati Tags: , , , ,

Call Manager Approved

by

I finally got our Cisco Call Manager PBX approved for our office move later this year. After the new landlord didn’t want to piggyback us on their system did my CEO finally agree to getting a CCM for our NYC office. Actually we wont be getting a physical CCM, but a voice gateway router and phones. We will be piggy backing off of our existing call manager cluster in our Call Center.

Now we need to select a vendor and place the order. I like both vendors, but one of them is the front runner now. We meet with them next week to go over final details of the deal. In addition to the phone system we are also getting a new Cisco Catalyst 4507 for the office to act as a core switch. Our existing one is 6 years old and probably wont scale to our planned growth. That and the 4507 will do POE (power over ethernet) and gigabit.

Now we need to actually sign the lease and get a move date. That is work for next week.

Proposals

by

Today we meet with the first possible integrator for our new phone system project in our NYC office. They presented us with their first proposal. The price they came back with was very high! We had them change the design and take out a bunch of stuff to try and slice the quote in half. We shall see what they come back with again. I also await a quote from the other integrator we are looking at.

I also meet with people regarding a pilot program for open office 2.0. we are going to roll it out to 30 people at the call center to see if it will not only replace open office 1.1 for the call center but also possibly replace Office for people who have it. We estimate it will save us allot of money. Can’t say how much but it is not a small number for us. Our CFO heard great things about it from my boss who is trying it out, and then heard the cost savings and wanted to know when we could get a test done! We are taking this one slow, but we believe the return on the investment in this open source product will be huge.

Bandwith

by

I meet with our Voice/Data provider today with our CFO to discuss renewing our contract with them. One account rep and his boss flew out from down south, and our local rep showed up also. They offered us a nice discount off of what we are currently paying since our volume is up, but our CFO thinks we can get a better deal. Let the games begin. This is the part of my job I don’t like. It is necessary but I don’t have to like it. Other topics we discussed where open issues we having as well as some new network options we have been thinking about. it was productive but long discussion.

Other issues that came up today was dealing with the ever growing space problem. We had to plan on moving 2 people to accommodate a new hire starting next week.