Vendor Relationships/Vendor Partnerships

These two descriptors are probably used interchangeably by some of you. Here’s my view: Vendor Relationships are interactions between myself as the client and the company whose software/hardware I’ve purchased and am currently using. I use the term Vendor Partnerships to describe something different: interactions between two different vendors from whom I have an integrated solution.

If Vendor-to-Vendor Partnerships have kinks in them, this can negatively affect otherwise positive Vendor-to Client Relationships. I’ve experienced this on more than one occasion, as I’m sure others have. One of the main vendors with whom I do almost daily business  has multiple integrated technology solutions from a number of third parties. In the early stages of these partnerships there are products in beta test phases and varying timelines for roll-outs. These timelines are often more aggressive than they should be and this results in field communication issues and Help Desk issues.

This has been a problem for me in both my door access control work and my video surveillance system work. In the first instance, I was a beta site for a new card reader product (I could have said no! I know) and it involved yet another commercial entity because the card readers had to be mounted on the doors before I could program and test them and the hardware guys were seeing these items for the first time as well. It took the better part of a month before I had the four lock/readers working successfully.

The second time involved a Vendor-to-Vendor Partnership that was relatively new and had a very small user base with my peer clients when I became one. It was really nuts when I had to engage the Help Desk. My Vendor opened a case and then called the Partner Help Desk and they opened a case before anyone would help me with my problem. That often took 2-3 days. They’ve smoothed things out tremendously now, but it’s taken a couple of years.

There’s a new wrinkle that’s popped up in this arena. A new access control manufacturer has become a Vendor-to-Vendor Partner with my provider and the sales guys have been trying to poach existing customers from them. Now that’s just wrong. There oughta be a law.

IP addresses: Use with caution!

A couple of years ago, I had two building master controllers on our door access system inexplicably start to toggle on and offline in a random pattern. I power cycled each one with no change. This went on for a couple of days and I finally called the Help Desk to report my problem. After remoting into the system, it didn’t take the tech long to discover that I had assigned the same IP address to both devices. That just throws the transaction processing server (TPS) into conniptions. Aha!

Fast forward to March 2012. Two weeks ago, an incident occurred with our OneCard system that highlights what can happen when IP’s conflict on a grander scale. We upgraded both hardware and software for the CS Gold transaction processing system; the software upgrade was to the most current version of not only CS Gold, but Oracle as well. Gold went from V5 to V6 and Oracle want from V10 to V11. In process we installed a new database server and a new TPS. The DB server is virtual, but the TPS is a physical box.

One afternoon toward the end of the week (after the upgrade), devices across  the system started dropping offline in no apparent pattern. Devices of all sorts were affected: door access master controllers, laundry readers, POS devices, copier readers, etc. Since the upgrade had been such a smooth process we were stumped. What could be causing the disruption?

After a few more agonizing minutes, our Sysadmin discovered that someone had re-connected the old TPS to the network. Ooops!! Since it was still powered up, there was an  instant communications conflict with the edge devices. In hindsight, we didn’t do enough decommissioning of the old TPS to prevent it from interfering with the new setup. We should have powered it off and removed the power cord or taped over it or removed it from the rack or something, anything to make sure it wasn’t mistaken for an active box.

We did a full system shutdown, a fairly radical move during the middle of the day, but we were on spring break so there were minimal numbers of people that were affected. Some devices returned to normal status, but not all. Our assessment was that over time as each device reached out to the server for updates, connectivity would slowly be re-established. We had 24 hours until students returned to campus, so we figured we’d be fine by that time. The Sysadmin stuck around until 2am in a babysitting mode and left when he saw our assessment gradually get confirmed.

Word to the wise: don’t duplicate IP addresses on an active network. Mayhem will definitely occur.

Project Coordination and Communication

OK, so I’ve heard that the success of any project depends mostly on good planning. Frequently, my experience says that good planning is not enough. You also need good coordination and good communication. Just last month I experienced what can happen if those other two elements are badly done or missing entirely.

The project to which I’m referring was installing a card reader in an elevator. The elevator is in a residential building and only residents with card privileges are authorized to use the elevator. A successful outcome on a project such is this is dependent on coordination between the general contractor, the elevator technician, the electrical contractor, the door guy (me), and the elevator inspector. On a recent Monday morning I got  frantic calls from both the electrical contractor onsite and the college electrician. “The elevator inspector is due in 60 minutes and the card reader hasn’t been connected yet.”

Elevator inspectors are a breed apart. There aren’t many of them (at least in my state), they have their own timetable, and they can shut a project down or substantially delay completion following a failed inspection. You don’t get a do-over when you want it, you go to the back of the line and wait your turn.

When I arrived onsite, I began to see why there were some anxious folks. I had worked the previous week with two elevator technicians. I spent a good hour with them. Three business days had passed since then and my card reader still wasn’t installed AND . . . the guy I saw now wasn’t either one of the fellows from before. The same was true with the electrician. I had to start over with my explanations for what my system needed and they had to make it all happen in 60 minutes. Yeah, it was a nail biter, but they did it. My reader worked like a charm first time out.

Luckily, the inspector was 2o minutes late and an impending crisis just became another day on the job. But I plead for more direct and frequent communication on all projects. If you think you’ve notified everyone who should be involved and you’re feeling pretty smug about it, think about it one more time. I bet you forgot somebody.

Test Benches are a must

I had my system up and running for over two years before I recognized the value of creating a test lab in my office. I had heard a colleague talk several times at conferences about his test equipment, but didn’t really think it applied to me because he did access control full-time and it’s only part of my responsibilities. He beta tests new products and has a much larger campus so he’s got many more unique locations and functional needs than I think I ever will.

However, when a colleague on the other side of town told me last year that he made a test bench, then I thought harder about the topic. I’ve got many more locations setup on my campus and more aggressive expansion plans to boot. What did he know that I didn’t?

I realized that every time I wanted to create a new situation or test a protocol I’ve never used before, I was walking halfway across campus for a tw0-minute test. Then I’d walk back to my office and see how the system recorded the transaction. While I enjoyed the walk, it occurred to me that I could do these tests in a fraction of the time if I didn’t have to leave the office.

So I gave some of my spare parts inventory to my electrical contractor and described the board I wanted him to make and waited for a couple of weeks for the finished product. The board simulates a master controller installation in a non-existent building and is equipped with two door configurations. One of the doors is controlled by an  HID  iClass prox reader for testing user privileges and the other is a virtual door that has door contacts, a request to exit and a small light to act as an indicator of an alarm condition.

I can create simulated conditions for every type of alarm that I routinely use throughout the system. I can use the equipment for training sessions with other staff. I have a router on my desk that I can use to test new devices on either the access VLAN or the separate VLAN that my video surveillance cameras ride on.

The benefits of this equipment so close at hand prove themselves to me quite often. I can troubleshoot problems and test new scenarios much more quickly and of course, in a pinch,  I can grab one of the panels off the board and use it for a replacement part in an emergency on campus. I love my test board!

Cameras get added to the mix

There’s something that can be done with doors besides passing through them. You can watch them. About a year after I inherited the door access system on campus, I was asked to start a video surveillance network to visually monitor and record activity at some locations. We have been using cameras in our Recreation Center and in the Gould Library for years, but both of these installations are analog networks and only one of them is being recorded. We were looking for something that could ride on our data network and could do event recording

We chose the NICE Vision software solution and are using cameras from Sony and Arecont.

The first few that I got up and running were in a couple of new residence halls. I’ve since added cameras at a remote warehouse and in a dining facility. When I started this project I knew absolutely nothing about video (I’ve still just scratched the surface of the knowledge base), but I’m not bashful about asking questions so with the help of a couple of installation engineers,  I managed to get my little network up and running.

It’s still a challenge to pick the right camera for the job, site it properly in the area, and focus it so that you get a useful image. For awhile I was lugging my laptop around to perform the focusing exercise. I’d take a 100′ ethernet cable with me and plug it into the building switch while a colleague got up on a ladder and unzipped the camera innards. We’d shout back and forth until we got an image that was satisfactory. The whole business was tedious at best.

After whining about it to a few colleagues, I learned that you can buy an inexpensive battery-operated  monitor and plug it right into the camera onsite. Using one of these jobs, focusing becomes a single-handed affair. I was pumped for action with this baby. Unfortunately, you need to rely on a rechargeable battery and as usual with that kind of arrangement, you never know what the status of the charge level is at any given time. I hate being up on a ladder with three minutes left in the job and having the monitor go dark. Ugh.

I dislike binary situations such as this. Note to manufacturers: what would it hurt to put a battery indicator on your device?

Don’t Let This Happen to You

No matter how hard I try to plan and get out in front of a project, there’s always the risk that something beyond my control will compromise the results. It’s particularly annoying when the bad things don’t come to light until months after the project is complete. They’re like little time bombs set to go off at some future date under unpredictable circumstances.

My first use of offline door locks involved the Schlage CL lock, both the 5696 and the 5692 models. We put these on all the student room doors in two residence halls that we built in 2009. The  process of installation and commissioning was new territory for me and was complicated by the fact that the  General Contractor hired a different sub just to install the locks. What I didn’t learn until much later is that these guys got zero instructions on proper methodology. Unfortunately, their goal was to do their work in the quickest amount of time possible.

Fast forward to November 2010. I received a call from Security at 10 p.m. on a weeknight that the lock on a student room wouldn’t allow either resident into their room. I showed up onsite with my tools and a replacement lock and began troubleshooting. I determined that it wasn’t a card error, programming lapse or battery failure. Plain and simple: the electronics were shot. The only remedy was to replace the electronics. This is normally a simple matter. Only problem: the guys who installed the lock used a power tool to screw it to the door and guess what? They stripped the screws in their hurry to get ‘er done. In order to change the hardware out at that moment, my only recourse was to physically separate the device from the door. Since I couldn’t unscrew the screws and I didn’t have a hacksaw or a power drill, I wrenched the lock off the door with brute force. Not recommended.

Normally this backboard is inline, not bent at 90 degrees. I had to bend it and flex it back-and-forth until the top screw broke. I released the bottom screw as one normally does, by unscrewing it. I had to destroy a $750 device because some yahoo installed it improperly and I didn’t have the luxury of returning during business hours to drill it out. I have no idea how many of the other 172 locks are compromised in this way, but at least I’m aware of future potential issues. Word to the wise: make sure your subs get proper instructions.

Troubleshooting can drive you nuts

At one time or another everybody has a problem that seems to defy solution. It doesn’t matter if you read and re-read the manual, if you ask the nearest expert or holy man, check with your vendor’s Help Desk or the manufacturer’s tech support, you come up against a situation that you can’t fix.

I had that happen recently with a door prop alarm that exhibited a pattern I couldn’t explain . . . or change. Some condition would trigger the prop alarm at the door in question and it would stay in alarm state for hours. Suddenly, it would revert to normal, for no apparent reason, and within 90 seconds, it would change state once again to alarm.

When I checked the alarm log, I noticed  that this had been happening for not hours, not days or weeks, but months. Who knew?

My electrician and I tried every physical thing we could think of to eliminate  known causes. We changed the door position switch and both relevant panels at the equipment head-end. We toned the wires end-to-end. There were no shorts or signs of equipment failure. And still the alarm.

We observed user behavior at the door. Nothing.

On my second (or was it third?) call to the Help Desk, the tech remotely logged in to the server and started poking around. When he got to the module that configures  parameters for each location, he noticed that this one didn’t call for the POSSIBILITY of a door prop alarm. In other words, if you haven’t set the location up for accepting a given condition, it will never report as such.

The factory default calls for this potentiality, but somewhere in the past, it was changed for this location. Did I do that? I really don’t know. At any rate, after the tech added the parameter change, the alarm condition disappeared and my problem was solved.

I’m still appreciative about resolving this one and it happened over a month ago. I was plagued with this for weeks, but no more!