Day: 21 April 2003

MRTG never lies

by

As you may know or guessed by these posts I actually work in the technology field. More specifically I deal with network & server operations for my company. Because we do that we monitor lots and lots of things on our systems. Anywhere from network traffic to temperature of equipment, to performance counters. This gives us an idea of what is going on when we are not watching the systems. Some of this is manual (stuff gets logged but to understand what is going on we need to look at it) and some things are automated (something is not working we get paged about it). That being said late this morning I took a look at our network monitoring pages for the weekend. I was surprised to see huge utilization between two of our sites. More specifically a database server in one site is sending data to two servers in another site. We use MRTG to watch bandwidth. It is a free linux tool. It is great. I for one am just getting into linux, but stuff like MRTG gets me wanting to learn more. So I bring the problem to everyone’s attention. Now here is the problem, half the people dealing with our database servers are saying nothing is going on at the time we see a problem. The other half agrees something is wrong but not sure what. Turns out, oh yeah we run some stuff late at night and an anomaly happened to cause it to send out 600 meg files and not 20 meg files, or something like that. So some of these same fun guys are saying that nothing is going on, but our T-1 pipe is flooded for 4 hours with traffic because some servers are trying to transfer well over a gig of data.

The moral of this story is that tools like MRTG don’t lie. We knew (well, Gus, Keith and I knew) from the first second we saw this what was causing the problem. We just didn’t know why it was happening. If it wasn’t for some sherlockian tactics from the man “Gus” we would be shooting in the dark. Never doubt the tools you use to figure out problems. When you doubt their ability it is time to get new tools, or test them. MRTG has not failed us yet, but we still have skeptics claiming “everything is ok”. Wake up and smell the coffee. Or wake up and drink some coffee to stay awake.

This whole problem boils down to a sherlock holmes type mystery. That is how I think of it. You have a problem, and you have to logically deduce what it is, and why it happens. Don’t ignore the obvious. Coming up with a complicated explanation as to why something is going on is stupid when a simple one fits the same criteria. There is a term for that, that I cannot spell so I won’t mention it here. I sometimes think that is all I really do when it comes to fixing problems. I try to use deductive reasoning. Probably why I like sherlock holmes so much.

This blog may seem like a rant, but it really is not. it is to say pay attention to the obvious, and think things through. Some people forget to do that, or just never think.

In other work news, we are still working to get a deal for voice service in our new facility. Dealing with telecom people is a nightmare. I thought dealing with routers was hard. And I remember when I thought programing a switch was a daunting task. Boy was I naive.

I spent the entire day dealing with telecom quotes or the above mentioned bandwidth issues. What is crazy is I know we have tons more in monitoring to deploy and watch. Right now we are finding tons of stuff. I wonder what we are letting slip through the cracks? Maybe nothing, but you never know.

I am left work at like 7pm tonight so I won’t get home till like 8pm. I need to make dinner, and relax a bit. today was very stressful day.

More work, less hacking, unless it is for work

by

It is monday again, and I am off to work. Today we meet with another telecom provider to see if they have a good deal. We need to make a decision this week.

OK, time for me to gripe about something. I am on the train, and cannot stand it when people receive cell phone calls (or make them) and have an entire conversation. In addition to that these people usually talk like a plane is flying by. Now I am all for cell phone use. I have 2 of them, and I don’t even have a home phone (another story). When I am out and about I will receive calls but only talk for a min if it is not important. If it is a work emergency (which I get more often than I would like) I talk. What is with this casual talking on a train with 100 other people. We don’t care what you have to say to your friends or family. We don’t need cell phone restriction laws, we need some etiquette from people. Ok, I am done ranting.

Speaking of cell phones I am kind of bummed that my main phone (an AT&T GSM Provided Sony Ericsson T68i) won’t work in our office in Kingston. Actually it wont work in kingston at all. I am not sure what will work well up there. I know verizon works, and t-mobile, but not sure if either of them provides good signal at our site. I have a verizon phone but I wanted to cancel it. No I am paying the monthly fee to wait and see. I also have a T-Mobile SIM card, but no phone. I am holding onto that for a bit longer also.

I gotta tell you, I go through phones (just like PDA’s) fast. I just upgrade to the latest and greatest allot. I was on ebay last night looking for a phone for the CEO of my company. He is going to europe and wanted a phone. He has a SIM already, but no phone. Since I didn’t have any extra he said to look for one. Well after looking I don’t see any GSM phones that do everything I want, except for the one I already have. For my CEO we are just going to rent one, but I started looking for myself also. Like I said I don’t have a phone for my T-Mobile SIM. It is weird. For the first time in a while I am ready to spend money.

OK, back to work talk. This post was supposed to be about work. I ended up sending back the low end Compaq 1U server we got. FYI, 1U is the size of it. Computer racks are broken down by U’s. Each standard rack has 42U. A 1U server takes up 1U of that 42. They are really small. It turns out the compaq server we bought does not have a RAID card for IDE like were were told 3 times. The were going to send me one for free, but I decided to send it back instead. Part of my reasoning is that we were going to get 2 of these. And I want to get them and see if I like them. if I do like them I want to standardize on them later on. If we got a free raid card, I would have to pay for them in the future. That would put the cost well over what I wanted to budget for these low end machines. I ended up ordering a super micro box and saved like $600 off my original quote. It is a no name brand, but we already use no name VA Linux machines, and this thing has a 3 year warrantee. I will see if they are any good.

Still Tivo hacking, not so thirsty anymore.

by

I was able to extract 1 show onto a DVD. I can get them into MPEG2 format which is what a DVD is made up of, but I am still having issues with the part where I actually get it onto a dvd that is viewable in my DVD player. I did this one, and it seems to work. Even better the video and audio at the end seems to be in sync. That is a problem with alot of these conversions. The down side is I left the burner on all night and I don’t know when it finished. it took a while. Also unless the numbers are wrong a 1 hour tv show took up 3 gig’s of space. The MPEG2 file on my hard drive only took up 1 gig, so I am not sure if that is correct. This whole thing is a work in progress.

I am going to show Gus my work today. I was on IM with him last night. he was excited about the possibilities for him. I think he is out of room on his Tivo but does not want to delete anything.

Blog roll online

by

I can do html or any other coding when it is just cutting and pasting! That is how I was able to get blogroll to work on my site. Well actually I can do a little bit more than just cut and paste, but sometimes it feels that way. I have not become daring enough to try to customize my moveabletype setup too much. I am slowly working up to a redesign on the main template layout and color scheme.