Work

MRTG never lies

by

As you may know or guessed by these posts I actually work in the technology field. More specifically I deal with network & server operations for my company. Because we do that we monitor lots and lots of things on our systems. Anywhere from network traffic to temperature of equipment, to performance counters. This gives us an idea of what is going on when we are not watching the systems. Some of this is manual (stuff gets logged but to understand what is going on we need to look at it) and some things are automated (something is not working we get paged about it). That being said late this morning I took a look at our network monitoring pages for the weekend. I was surprised to see huge utilization between two of our sites. More specifically a database server in one site is sending data to two servers in another site. We use MRTG to watch bandwidth. It is a free linux tool. It is great. I for one am just getting into linux, but stuff like MRTG gets me wanting to learn more. So I bring the problem to everyone’s attention. Now here is the problem, half the people dealing with our database servers are saying nothing is going on at the time we see a problem. The other half agrees something is wrong but not sure what. Turns out, oh yeah we run some stuff late at night and an anomaly happened to cause it to send out 600 meg files and not 20 meg files, or something like that. So some of these same fun guys are saying that nothing is going on, but our T-1 pipe is flooded for 4 hours with traffic because some servers are trying to transfer well over a gig of data.

The moral of this story is that tools like MRTG don’t lie. We knew (well, Gus, Keith and I knew) from the first second we saw this what was causing the problem. We just didn’t know why it was happening. If it wasn’t for some sherlockian tactics from the man “Gus” we would be shooting in the dark. Never doubt the tools you use to figure out problems. When you doubt their ability it is time to get new tools, or test them. MRTG has not failed us yet, but we still have skeptics claiming “everything is ok”. Wake up and smell the coffee. Or wake up and drink some coffee to stay awake.

This whole problem boils down to a sherlock holmes type mystery. That is how I think of it. You have a problem, and you have to logically deduce what it is, and why it happens. Don’t ignore the obvious. Coming up with a complicated explanation as to why something is going on is stupid when a simple one fits the same criteria. There is a term for that, that I cannot spell so I won’t mention it here. I sometimes think that is all I really do when it comes to fixing problems. I try to use deductive reasoning. Probably why I like sherlock holmes so much.

This blog may seem like a rant, but it really is not. it is to say pay attention to the obvious, and think things through. Some people forget to do that, or just never think.

In other work news, we are still working to get a deal for voice service in our new facility. Dealing with telecom people is a nightmare. I thought dealing with routers was hard. And I remember when I thought programing a switch was a daunting task. Boy was I naive.

I spent the entire day dealing with telecom quotes or the above mentioned bandwidth issues. What is crazy is I know we have tons more in monitoring to deploy and watch. Right now we are finding tons of stuff. I wonder what we are letting slip through the cracks? Maybe nothing, but you never know.

I am left work at like 7pm tonight so I won’t get home till like 8pm. I need to make dinner, and relax a bit. today was very stressful day.

More work, less hacking, unless it is for work

by

It is monday again, and I am off to work. Today we meet with another telecom provider to see if they have a good deal. We need to make a decision this week.

OK, time for me to gripe about something. I am on the train, and cannot stand it when people receive cell phone calls (or make them) and have an entire conversation. In addition to that these people usually talk like a plane is flying by. Now I am all for cell phone use. I have 2 of them, and I don’t even have a home phone (another story). When I am out and about I will receive calls but only talk for a min if it is not important. If it is a work emergency (which I get more often than I would like) I talk. What is with this casual talking on a train with 100 other people. We don’t care what you have to say to your friends or family. We don’t need cell phone restriction laws, we need some etiquette from people. Ok, I am done ranting.

Speaking of cell phones I am kind of bummed that my main phone (an AT&T GSM Provided Sony Ericsson T68i) won’t work in our office in Kingston. Actually it wont work in kingston at all. I am not sure what will work well up there. I know verizon works, and t-mobile, but not sure if either of them provides good signal at our site. I have a verizon phone but I wanted to cancel it. No I am paying the monthly fee to wait and see. I also have a T-Mobile SIM card, but no phone. I am holding onto that for a bit longer also.

I gotta tell you, I go through phones (just like PDA’s) fast. I just upgrade to the latest and greatest allot. I was on ebay last night looking for a phone for the CEO of my company. He is going to europe and wanted a phone. He has a SIM already, but no phone. Since I didn’t have any extra he said to look for one. Well after looking I don’t see any GSM phones that do everything I want, except for the one I already have. For my CEO we are just going to rent one, but I started looking for myself also. Like I said I don’t have a phone for my T-Mobile SIM. It is weird. For the first time in a while I am ready to spend money.

OK, back to work talk. This post was supposed to be about work. I ended up sending back the low end Compaq 1U server we got. FYI, 1U is the size of it. Computer racks are broken down by U’s. Each standard rack has 42U. A 1U server takes up 1U of that 42. They are really small. It turns out the compaq server we bought does not have a RAID card for IDE like were were told 3 times. The were going to send me one for free, but I decided to send it back instead. Part of my reasoning is that we were going to get 2 of these. And I want to get them and see if I like them. if I do like them I want to standardize on them later on. If we got a free raid card, I would have to pay for them in the future. That would put the cost well over what I wanted to budget for these low end machines. I ended up ordering a super micro box and saved like $600 off my original quote. It is a no name brand, but we already use no name VA Linux machines, and this thing has a 3 year warrantee. I will see if they are any good.

Bidding wars, no its not a movie

by

So we have the bidding going on for services at my new office in kingston. We got people asking to provide Internet service that I never even heard of. We also have two big voice providers coming in to bid this week and next. This week is the contender. (no names just yet), next week is the guys who have our biz, but need to work to keep it.

I also got news that we have a new hire for monday. I ordered his computer last night and should have it today. I hope justin can build it in time. We are out of phone ports so that will be interesting. we also have no place to put the guy so a phone is no big deal. it is now starting to get fun with space.

I am in the subway tunnel now, so I will hopefully write more later.

Kingston project is a go!

by

Even though officially our project was a go for a while, we finally sat down and ironed out time table’s, and things. Our CEO wants everything ready at the facility by June 6. June 6, by the way was the date we moved into our current office last year. That poses a problem for me since I have vacation plans for that week. I am in barbados with my family. My boss Gus says to chill and wait and see if things actually shape up next week. Realistically I don’t think we can make that date, but I am told to aim for that date, I have to do everything possible to do it. Even if doing all that won’t change the fact that we wont make that date. it is an interesting position.

I also got the cd-rom / hard drive for the new server we bought. I believe I talked on monday about the discount server we got from comaq that didn’t ship with a cd-rom or floppy drive. Well that problem is resolved. I need to build the dam thing now.

It just started raining in NYC. I wore light khakis thinking it would be nice out. They are going to get messed up by the dirt and water kicked up when I walk around now. Oh, well.

I am on my own today, well I have justin but he can’t do everything I need him to yet. Keith is off due to the holiday. Not really sure why since he is not very observant, but I think he said something about having to travel to jersey or something. Should be an interesting day. We shall see…

It might be summer, & an anniversary of sorts!

by

Today is the first day this year that I left my house without a coat. It feels like summer. It is like 65 degree’s now going up to 80. I am happy the way it is right now. It is perfect. The AC in our office is not working yet. I think there is a problem. If I had this blog last summer I would have written about all the great problems we had with AC in our office. But I didn’t so use your imagination. It was hot. This year is not shaping up very good either right now. On a plus note the AC in my office works, but it dies under the strain of keeping air cool everywhere. even with the door closed it will get hot on the first 90+ day. we shall see what happens.

OK, due to me taking a late train yesterday night I actually had 3 blog entries yesterday. that will not happen that often. I really should have had 2, but I broke up one into personal stuff, & work stuff.

I am working on figuring out how feasible it is to move my entire photo web site over to my apache server. I want to get it to work, but I don’t want to have to do allot of work. there are tons of pages, and about a gig of images. it will be a big job if I can’t automate it or get frontpage extensions to work on apache. Hopefully I will have time this weekend to look at it some more.

I am taking Dan to bestbuy today to purchase wireless gear for his new apartment. Dan is the director of operations at the company I work for. He is really cool, and we get along great. That said, I don’t normally do computer work for people outside of work. I can’t stand it. Dan understands that so we are going to buy the stuff only. I like buying things! Actually some people I don’t mind helping, but others who have no business asking for help or other people who just expect it, I don’t like doing stuff for. He is in neither category so it will be fun to setup some wireless stuff. Wireless is good.

It dawned on me this morning that yesterday was an anniversary of sorts for me. I started at MFS network technologies (a division of worldcom at the time) 6 years ago on tax day. That was 1997 if people can’t do math. That really marks my beginning of my technology career. My first official help-desk job didn’t come till AuGust 1997, but I did plenty of technical and help-desk work at worldcom so I consider that my start. I figure when you are upgrading file servers, building new computers for people and auditing software you are in the technology field. That was just a bit of history about my tech roots.

Today’s work agenda, tech meeting, then tech management meeting, then trip to bestbuy, then kingston move meeting. Just full of fun meetings today! I also need to call and harass some vendor’s (see yesterdays posts).

Tonight’s the hockey game. I can’t wait. I broke out my Islanders jersey and it is in my bag for later. I am at a point where I cannot wear jersey’s of any kind to work. it sucks but I will deal. At least I can still wear tee shirts. I think it is funny to sit in a meeting with vendors in a tee shirt and khakis while they are in suits. it is very amusing.

Well I feel us descending into the train tunnel so I will stop here. Details on the game and other stuff tomorrow.

Tuesday With vendors, and web problems.

by

Well I am starting to write this on the east bound train home tuesday. Not sure when this post will be finished but I am stuck leaving work at 7:30PM tonight. It is daylight out so I don’t even realize that it is late anymore. There is still work to do and it is light out, you forget it is 7pm.

We had some problems some web sites today. It was not fun. We think it is related to a bunch of different problems ranging from running out of disk space on one web server to huge amount of searching on a database server. It probably will take days to sift through the data.

I ended up meeting with our telecom vendor today. It was a meeting that was planned when I was out on friday so I was sort of surprised. I think it went well. We told them point blank that we were not happy with them, and we need problems fixed now or we wont use them on in our new office. they seemed eager to do what we want (of course who wouldn’t when you dangle money to someone). We are on sales guy number 4. This guy seems to have a clue so we will see how things go from there. it is tough. I like what these guys say, but I have heard speeches before and then get no results. if our new sales guy can do what he claims I will be happy. the question is will he do what he claims? I am betting allot on that answer.

In other news, Keith is officially a dumb-ass. His problem with the RAID array I spoke about before was that the card came loose and he didn’t check it while he tried to fix it for 2 1/2 hours yesterday. At least he figured it out on his own. FYI, I use the term dumb-ass lovingly:) If I really don’t like them I would just call them an asshole.

Sales guys suck, & other Tuesday news

by

Have I mentioned that sales guys (people, gotta be politically correct) suck? They do. I am trying to sift through all the quotes and crap I am getting while preparing to build this new office, and people don’t have a clue. I have one phone provider call me last week going they can’t provide local phone server in the area we are moving to. I was like, ok sorry. I guess we can’t use you then since I want one company to do everything for me. Then monday I get a call from that guys boss going, well we kinda can do local service in that area. I was like make it simple and make it a good price and I will look at you. Until then these guys are talking BS. I cannot wait to get the bidding out of the way and just build the thing.

Our weekly management meeting was canceled yesterday. I think there was a board meeting or something. It will happen today instead.

So I got my new low end compaq rack mounted server. I mean low end when I say it. This thing is an IDE drive machine with RAID in it. it is so cheap that it didn’t even come with a CD-ROM & Floppy drive. Normally I assume a computer will ship with at least 1 input device such as a floppy drive. got a call out to those people to get something so I can load the OS onto it. Like I said sales people are moron’s.

The only other things of note on monday were we had some patches of problems with one of our database servers. Our engineers are looking at it. That means I need to get my crazy 8 ball out and help them with the diagnosis. Also Keith is having problems with an IDE RAID system. I will look at it with him today.

Work again, Monday Meetings & hardware purchases

by

It is monday again. The most hated day of the week. I used to not like monday’s just because it was the first day back from the weekend. My new issue with monday is it is the day of meetings. Actually it used to be the day of meetings, now it just feels like it. Monday is the day that all the managers in my company get together to let everyone know what is going on. That means writing up agenda’s, and other fun meeting stuff.

Monday is also the day in most companies where new hire’s start. We are no different. Thankfully we don’t have any today. I actually have someone new starting tuesday, but we probibly will have like 5 new people on next monday. Not sure yet. No one tells me till the last minute normally.

This week I need to finish setting up the temporary firewall I will use to test the new network for our kingston office. For setup purposes I am creating a new isolated network for all the gear we are getting for that office. I am using an old firewall we stopped using in february. Once that is done I will need to begin getting quotes for servers. The next step in the design would be file servers and a domain to put all the computers in. We currently use a mix of compaq and dell servers. Most of our machines are dell, with some database machines being compaq’s. Alot of our older equipment is VA Linux machines. I personally don’t like the VA Linux equipment. The company stopped making hardware, and the machines are not hot swapable anything in them. Granted their track record is ok. I have only had problems with 3 of them that I can remember, and we have over a dozen of them. For this new office I have not yet figured out if we will go compaq, or dell. It will depend on price most likely. I also have an IBM guy wanting to sell us stuff. They say that they can meet the other guy’s prices, but I haven’t seen that happen yet. Sales guys, are sales guys.

Another cool project we have going on is upgrading our data center to a gigabit backbone. We have the design down, and now we need to buy and deploy the hardware. I probibly will purchase that stuff this week. that will be cool to deploy.

Obviously since I am talking about work stuff I need to be vauge enough so I can write it here, but also descriptive enough since I want this blog so I can write about what I am doing and thinking. That will be a fine line I walk when doing this.

The scrub haircut guy & other misc. items

by

Yesterday was a good day, I think. Got a lot done at work. We still are no where with picking a hardware vendor, but at least we getting other stuff done. Got construction requirements done for the space. I was bad and had burger king for lunch. I don?t do it so often anymore so I think it is ok. Keith was going down there, what was I to do? I also took care of my metro card problem. I never got my mail and ride ticket for April so I had to fill out an affidavit and get it notarized. It was a pain, but it is all sent out and I hopefully will get my money back and my may ticket on time.

Got my hair cut also. The guy I always goto was on vacation so his dad did my hair. He is good, but never cuts it short enough. I will probably have to go back in like 2 weeks. I like those guys. I goto an old fashioned barber. It is father and son who run it. These two Italian guys just cutting hair. they are cheap also. My friend jayson showed me this place years ago. I was paying double what I pay now for a haircut at the ?stylist? and it was not as good.

After I got my hair cut I picked up food at the omega diner and got my mom. We went over to my sisters to eat dinner. My dad was already there babysitting. Everyone is all worried because Morgan cry?s right after eating. I guess that is a bad thing for babies. Got to play with her for a while. Then I got to hold her while she slept. I love doing that. It is so peaceful. I had to go home though because it looked like we were having problems at work. Anyone at work reading this, when I get 20 pages while holding my niece I am not a happy camper!!! Besides playing with Morgan I also sent all of her baby pictures to myself via ftp. My sister is running out of hard drive space, and she does not have a cd burner. My next project is to find her one.

I got home around 9pm. I installed my new hard drive for my desktop computer. 160 gigs of stuff. I started to move stuff off my old drives. Got one of them done and found out the one that broke was the data drive with all my personal files on it. I am so happy I made a backup of most of it to DVD before I did this upgrade. It is bad, but not too bad. I gave up on moving my mp3?s since I have a more recent copy on the powerbook I am using. I will back it up later. The only thing I need to do now with my ?modified? not upgraded computer is to reinstall windows xp. I have been using the same install for over a year. That is like a record for me. I tend to reinstall operating systems every few months. Now I am busier I do it less often. Hopefully the mac will require less reinstall?s of the OS also.

Spoke to Steve. We are still on for Thursday. Him and tom are driving down around 4pm. That means they will probably be late for the 8pm concert. We shall see.

I am running late today. I just got up and could not get going. So I missed the 2 regular trains I take. I should be like 20-30 minutes late for work. Still well ahead of the rest of the department. Actually looking at my pager just now I realized I may be a little later than that. I don?t think we are doing a staff meeting today since 4 people are out of the office this week. I just paged Gus asking him what we will be doing.

Well we are now entering the famous train tunnel to NYC. It is rainy today, again. My hands also hurt and it is only 9am. I hope they get better because I type and mouse click all day! More later, maybe?

tuesday again (an original title)

by

Tuesday again. I realized yesterday after I wrote this blog one reason I didn?t write in a while was I didn?t have the laptop during the train ride. Took Thursday off and saved weight on Friday and Monday by not bringing the powerbook to work. Oh well.

So I am really pissed, I was hanging out a little after work yesterday before I came home and a guy I work with dave hit me in the head with a mini basketball. It was an accedent, but it hit me right in the middle of my glasses. Since these glasses are so light weight (AKA easy to break), they are all out of bent. I can still wear them ok, but they feel like crap. I need to goto the lenscrafters near work sometime to fix it. The problem is I don?t go eat lunch anymore. We order in all the time. Also it snowed in New York yesterday so it is cold nasty and wet out today. It is april and it is still snowing. I will be honest and say that I would rather it snow than be 85 degree?s right now. My apartments air conditioners don?t come on till may 15, so it better be dam cold till then.

People actually read this blog. I am amazed. Well maybe not people, but a person read it. Of course that person happens to be the boss. Crazy, yet lovable Gus. Knowing that he may read this, I can safely say he keeps me on my toes and challenged. But he also adds to the blood pressure! Actually he was very funny Monday. He hurt his back over the weekend, and took some pain med?s Monday. He was loopy most of the day. Keith said he went over to his office to do something and Gus just staired at the keyboard for a few minutes before he remembered what he was supposed to do. Of course I won?t make fun of anyone hurting themselves working out, since I don?t and I should!!!

Trying to remember what is on my agenda for today. I am supposed to meet with some executives from a company called expanets. They support our phone switch in our call center. They are really bad so we are screaming at some executives. It may make us feel better, but I doubt that they will do anything new. I am a pessimist that way. Today is also day two of the no developers week. Or less developers than normal week. So far so good.

I want to add features to this blog to allow responces to my posts. I half want that to see if anyone is reading these. The other half of me doesn?t want it because I probably will get freaks posting stupid crap.

Looking out of the train just now it looks like it is going to snow again, or it is just dreary outside. The whole sky is all white, or white/grey. Not a beautiful site.

Well I think I have said my peace for the morning. More later, maybe!!!