If I thought I Was Busy Before

If I thought I was busy at work before, I am in way over my head now. One of the guys on my team left yesterday. We are now short handed. The big problem is the small issues now slowing down getting the bigger projects done.

I have 1 Windows 2003 Server with Active directory done for our nyc office. it is already in production. I need to finish moving stuff off one other server and then move the AD roles to the 2k3 server. Then I can rebuild the other box, and move on in the upgrade cycle.

A major web site for a customer of ours is launching tonight. Thankfully it didn’t require too much time on my teams part, but it ate up resources.

As I mentioned before I am also trying to merge our mail systems for 1 department. I have 2 call centers who are supposed to interact with each other, but they are on 2 different mail systems. We are going to merge our NYC call center people with our Kingston mail system, since the kingston system is newer and it is cheaper to maintain. Then when that is done we will have all our call center people on one mail system and the remainder of the company on another. The good news is the new mail system (Imail 8.0) is awesome. I have been a fan of Imail for years, since like version 5.0 I think. I never used it at work, but have at home or when I hosted my site with someone. So far I am very happy with it.

Upgrades Galore

Over the past few days I have completed building my first Windows 2003 Domain Controller for our NYC office. Last night I moved a few DHCP scopes onto it. I will finish that this weekend. I also moved our callcenter mail server to our data center. It completes my migration to IMail 8.0. It is a nice little mail program. Next up is moving NYC call center people to this mail server. That requires changing their email address’. No one will be happy about that. I am working on a migration plan now.

I also need to learn more about rsync. We are going to use it for some new things starting soon.

Busy At Work, What Else Is New

Work this week has been challenging. I got nothing I planned on doing yesterday completed. I was busy fixing other issue that crept up. Kai from our Kingston office was down for a meeting and also to just work for a day out of NY. He never did that before so we wanted to bring him down.

We are in the midst of several major projects. Remote recording of voice calls, creating a beta test for remote agents, move a mail server, upgrade NYC’s domain controllers, evaluate SMS, build a new service domain, and oh yeah we had 4 people get hired in the past week. My team is just a bit busy. Don’t get me wrong, busy is good. Overwhelmed is not. We are currently busy but quickly moving to overwhelmed. Hopefully we can get everything done without a problem.

Automatic Updates

Tonight we try our second wave of automatic windows updates for desktop computers. We (keith & I) finally got the Microsoft windows update server working. This server replaces Microsoft’s own windows update site for your company. you can pick and choose what updates your desktops get and when. It saves us from downloading patches we have no idea about. It is a great idea. I looked into this service back in August, but had trouble with programing the client machines to auto update. I then didn’t have time to finish the project. A few weeks ago keith took up the research into the auto update server. He did everything I did and then learned out how to program Active Directory to tell computers in the domain to look for the auto update server. We took the existing server I had up and running and made some changes in active directory and now we have an automatic update system. no more running around to each computer for patching security wholes.

Last night we ran a small test. Tonight we do 1 department. If all goes well again, we will do another department tomorrow, and the rest of the NY office next week. So far all looks good. We will know for sure tomorrow morning.

New Support Schedule

Today we are trying out a new support schedule for the Kingston Call Center. We are going to have a support guy onsite from when they open at 8AM to 8PM. Hopefully it will help in keeping things running normally. The change has been a long time coming but we finally have the people in place to do it. The next step is full coverage, but that may be a way’s off since you need more than 2 shifts to cover 16 hours a day, 7 days a week.

Other Projects

Other things I am trying to work on are… I just downloaded Connectix (now Microsoft’s) Virtual PC for Windows. I want to see if it is better than VMWare. It was up on MSDN so I decided to give it a try. I also grabbed the beta for SMS 2003. Gets me wondering when they will release it since 2003 is almost up.

My exchange project is not dead yet, but I have no time to do anything on it, so it sits idle. I am also looking at the new Microsoft Chat Server. We currently use a Jabber server from Tipic. I actually like it allot and it was very cheap. I want to see how the Microsoft one measures up.

Besides projects to evaluate new software I am also working on tons of other stuff. I am scheduling an update to our email ticketing system called Mailflow. We currently use version 1.0 and are getting a free upgrade to 2.0. I am attempting to deploy it next week. It is from a company called Visnetic. They make some neat pieces of software.

To conserve bandwidth we are about to start blocking most outbound network ports from our firewall. That will make me a popular guy with the rest of the company! Also we are ordering a private line T-1 between our two offices so we can send VOIP calls between them in greater number than we can over the internet T-1’s we have.

2 Days At Kingston

I did my 2 days in Kingston this week. I am on the train (Amtrak) heading home. Got allot done, and still more to do. We just brought up 10 computers from our NYC call center and are setting them up in Kingston. We now have over 32 stations setup with 10 more for training. We are well on our way to 48, the target for phase 1 of our build out. Then we move up to 85 stations soon after that.

I also just got a new help-desk person in Kingston. Kai started monday for us. He was originally hired as a call center supervisor but he had IT background and we liked him so we transfered him over. Sorry dave. the call center’s loss is our gain.

I will be in NYC the rest of the week, and Keith will be in Kingston Wed, & Thursday. I want all the building out to be done so we don’t have to spend so much time up there. It takes time to transfer the knowledge we have to the new people at the same time as we are deploying so much gear.

We just got 30 new phones. We need to program them. We also had 8 new people start Monday. It is busy, and I wouldn’t have it any other way. Of course a break now and then would be nice!

The Phone Traffic Transfer

Last night I successfully completed tranfering almost 50% of our call traffic to our new Call Center. At about 8:45PM Sunday AT&T redirected the traffic and it immediatly came up at the new center. It was actually very painless. Knock on wood nothing goes wrong.

Today I got to work at 7:30AM to reboot the call center software server, and from 8AM on the office was busy. We maxed out all the computers we had setup by 1PM. We even had 3 users logged in from our NYC office using the VPN tunnel. As of 10PM tonight it was still busy and they needed to get people back on the cisco phones because the queue was backed up alot.

Surprisingly (or not) we didn’t have that many major issues after the transfer. We had some computer that had some apps not working. We also had 5 machines that didn’t have Windows XP activation completed. The new computers ran out of the 30 days for activation and some of them coudln’t startup. To make matters worse the microsoft activation servers were not working ok. After some work, we got them online but it was a pain in the ass.

Service Observe And Phone Stuff

Today I dealt with finalizing a solution for “service observe” in our new call center. We want the ability to listen in on our rep’s calls so we can QA them. Cisco’s VOIP solution does not offer that out of the box. I don’t know why, but it doesn’t. We have a company we are looking at, but we want to be sure things work correctly before we order it.

Also today I configured up 4 Cisco 7940 IP phones to use in our NYC office. We need some in case we are overwhelmed with calls on next monday. We should be able to have a total of 5 phones ready to go if needed.

Configuring the Cisco phones is really cool. They are programable via the Call Manager web site, and you can reboot the phone remotely. To get them to work, all you need to do is set the phone to DHCP and have the proper TFTP server config in the scope. It is not bad once you get the hang of it.

I was using my Cisco phone all day to talk to Joe, Justin, & David in our Kingston office.

We are planning a test of our T-1 redundancy on friday. The work to provide roll over and redundancy was completed and we will test it friday to make sure it works for mondays move of voice traffic. We need 3 voice T-1’s to work together so if calls come in and one T-1 is full, they roll over to the next one. This is also helpful in redundancy in case one circuit fails to work. We won’t need all 3 next week, but we will probably use up one and goto a little on the second one.

Training For A Call Center

Yesterday a new class of trainee’s started at our Kingston Call Center. We have 24 customer service reps who will “hopefully” start taking calls full time on 9/22. The class was so big that we had to put the working reps in the training room and the people to be trained sat in the call center. Joe our Tech guy in Kingston is overloaded right now. Justin went up for 2 days to give him a hand. Justin was eager to go up, so it worked out ok. I need to get up there one day next week. I will also need to be there on the 22nd.

Besides the ongoing call center build out I have hardware swap out’s to deal with in our Data Center Thursday morning. Tomorrow will be interesting since it will be a long day.

I just received my first Penguin Computing Server. These guys make cheap linux servers. The joy of it is, I can put Windows OS’s on it without a problem. The Super Micro servers I have been buying for windows will not handle Linux. I have some issues with the hardware, but for the price we will deal with it. A decent server can be had for around $1600. I can get a really cheap one for under a grand. We are actually looking to get some cheap windows box’s and may actually use these guys.

I am still awaiting a heat sync for a Super Micro test server we are testing. I got a trial of a serial ATA server. Actually we had the financial ok to buy it but instead of buying a regular server we got this to test. If we like it, we will keep it. If we don’t like it, we will send it back and get a normal IDE server. Serial ATA looks good, and this machine can do hot swappable drives, where IDE RAID cannot.

I need to also start working on our Exchange 2003 test server. Gus and I want to get one working for the Information Systems team and see if it is easy to admin and troubleshoot. If it is we may want to make a larger scale deployment. The difference this time will be we don’t outsource it and we do it ourselves. In December of 2002 we outsourced our mail and put it on Exchange 2000. It was a horrible failure. Too much latency with accessing exchange over the WAN. We abandoned that option and returned to our original mail provider with just POP/SMTP mail. We have been happy since.