What You Don’t Know

I’m continually reminded of the fact that the smallest gap in one’s knowledge base can result in dead ends and delays that stretch a project’s timeline far longer than originally planned. This was certainly true in a recent video surveillance job that I did. I was not only installing new IP cameras in a location, but I was grandfathering in two existing analog cameras as well.

We had three analog cameras in our Recreation Center, one of which was dead. The project scope included adding six IP cameras to the building (five in new locations and  one to replace the dead analog camera) and using a conversion device called an encoder to translate coax-delivered analog data into digital IP-based data. There was no problem deploying the new IP cameras and getting them configured. During installation of the encoder (NICE Vision NVE1002) , however, I countered an unknown roadblock.

I already was using an NVE1002 in another building so this was puzzling. When I brought up the live video display, I could see the six channels from the IP cameras, but not the two channels from the analog cameras. I verified that the encoder was on the network and the port I had it plugged into was open, because I could ping the device.  Next I took the box over to the library where I had the other NVE1002 and swapped the two devices. No live video.

I then took the Library encoder over to the Rec Center and plugged it into the network and the 2 coax cables. No live video. Just to sure, I asked my network administrator if he could see both devices on the network and he said he could. My next hunch was that the BNC connectors on the coax cables were not compatible with the inputs on the encoder so I studied up on that possibility.

Since I know nothing about coaxial cable and could not independently judge compatibility of the connectors, I went to a colleague with my quandry. He volunteered to bring the problem to the attention of some video experts and have them look at the problem the next time they came to campus. After testing with some diagnostic equipment they came to a remarkable (to me anyway) conclusion: the cameras were not powered on.

I’m very familiar with Power-Over Ethernet (POE) equipment because, in fact, all of my IP cameras are POE devices. What I didn’t know until that moment is that coax can deliver power as well as data. The video monitor that was used to view the two analog cameras was also a DC power generator for them. The cameras also had no other was to receive power from an alternate source. That meant that I had to abandon the cameras altogether and purchase and install two new IP cameras (which I did not have in my possession at the time, of course).

This missing little gap in my knowledge turned the project from a 48-hour quickie into a 3-week head-scratcher. Luckily, the colleague for whom I was doing the work was much less affected and anxious about this than I was so there was no anger or blame on her part. In the end she was very happy with the results.

Anyone need an encoder?

Expanding the Campus System

Alumni Guest House/Johnson House

I recently submitted a proposal for the expansion of our card-based door access system to as many campus buildings as possible in a concerted effort to minimize the number of metal keys in circulation. Clearly, there are other benefits to enjoy from such a project, but reduction of keys is a very high priority. Over the years my institution has experienced key loss, sometimes for a single door, sometimes a building master, and more than once, something more serious. Keys can vanish due to accident or oversight and they can also disappear due to employee termination. It takes enormous discipline and communication to force the return of keys when folks graduate or leave employment.

Goodsell Hall

The traditional remedy for lost keys is to re-core locks and reissue new keys. When that involves one office, it’s not terribly costly. When the scope is an entire building or the the whole campus, then we’re talking big bucks. On the other hand, if a card with extensive privileges is lost, the damage can be addressed with a couple of clicks of a mouse. Declare the card lost and it’s instantly expired so even if it’s found it’s useless to the new owner.

It’s undeniable that extending the online access system to 15 more campus buildings will require a major up-front investment, something in the mid six figures by my estimation. But once that’s complete, ongoing maintenance is minimal. If the campus  key policy is simultaneously modified (and made more restrictive), the need to re-key the campus may never occur again. If we also spread the expansion project over four to five years, it then becomes more affordable. It joins the list of other capital improvement projects.

As I worked on this proposal I had to consider prioritization issues. I had to determine a sequence for bringing these buildings online. What criteria will help us make this decision? Are interior or exterior doors more significant? Which spaces need more immediate protection: administrative or academic?

Old Music Hall

Conventional wisdom suggests that if choices need to be made regarding which doors to electrify, one should start with a building’s exterior doors.  Given the goal of reducing metal keys in circulation, I next sought out the building (s) with the largest number of keys issued annually. The campus locksmith was instrumental in helping me zero in on two building clusters: the Sciences complex (3 buildings) and the 2 conjoined buildings housing the Art Department and the Computer Science Department. He annually issues and collects over 500 keys to students in those five buildings. Given this reality, it’s clear to me that the project begins with these five buildings.

However, these locations also have some of the most sensitive interior spaces as well: physical sciences laboratories, animal research facilities, hazardous materials storage, the need for student and faculty access 24/7. Adding these to the online system at the same time as the exterior doors virtually triples the initial cost. Depending on the available funding options, the entire project of expanding the access system campus-wide may grow from a four year project to a six year project.

But hey, it’s just time and money, right?

Vendor Relationships/Vendor Partnerships

These two descriptors are probably used interchangeably by some of you. Here’s my view: Vendor Relationships are interactions between myself as the client and the company whose software/hardware I’ve purchased and am currently using. I use the term Vendor Partnerships to describe something different: interactions between two different vendors from whom I have an integrated solution.

If Vendor-to-Vendor Partnerships have kinks in them, this can negatively affect otherwise positive Vendor-to Client Relationships. I’ve experienced this on more than one occasion, as I’m sure others have. One of the main vendors with whom I do almost daily business  has multiple integrated technology solutions from a number of third parties. In the early stages of these partnerships there are products in beta test phases and varying timelines for roll-outs. These timelines are often more aggressive than they should be and this results in field communication issues and Help Desk issues.

This has been a problem for me in both my door access control work and my video surveillance system work. In the first instance, I was a beta site for a new card reader product (I could have said no! I know) and it involved yet another commercial entity because the card readers had to be mounted on the doors before I could program and test them and the hardware guys were seeing these items for the first time as well. It took the better part of a month before I had the four lock/readers working successfully.

The second time involved a Vendor-to-Vendor Partnership that was relatively new and had a very small user base with my peer clients when I became one. It was really nuts when I had to engage the Help Desk. My Vendor opened a case and then called the Partner Help Desk and they opened a case before anyone would help me with my problem. That often took 2-3 days. They’ve smoothed things out tremendously now, but it’s taken a couple of years.

There’s a new wrinkle that’s popped up in this arena. A new access control manufacturer has become a Vendor-to-Vendor Partner with my provider and the sales guys have been trying to poach existing customers from them. Now that’s just wrong. There oughta be a law.

IP addresses: Use with caution!

A couple of years ago, I had two building master controllers on our door access system inexplicably start to toggle on and offline in a random pattern. I power cycled each one with no change. This went on for a couple of days and I finally called the Help Desk to report my problem. After remoting into the system, it didn’t take the tech long to discover that I had assigned the same IP address to both devices. That just throws the transaction processing server (TPS) into conniptions. Aha!

Fast forward to March 2012. Two weeks ago, an incident occurred with our OneCard system that highlights what can happen when IP’s conflict on a grander scale. We upgraded both hardware and software for the CS Gold transaction processing system; the software upgrade was to the most current version of not only CS Gold, but Oracle as well. Gold went from V5 to V6 and Oracle want from V10 to V11. In process we installed a new database server and a new TPS. The DB server is virtual, but the TPS is a physical box.

One afternoon toward the end of the week (after the upgrade), devices across  the system started dropping offline in no apparent pattern. Devices of all sorts were affected: door access master controllers, laundry readers, POS devices, copier readers, etc. Since the upgrade had been such a smooth process we were stumped. What could be causing the disruption?

After a few more agonizing minutes, our Sysadmin discovered that someone had re-connected the old TPS to the network. Ooops!! Since it was still powered up, there was an  instant communications conflict with the edge devices. In hindsight, we didn’t do enough decommissioning of the old TPS to prevent it from interfering with the new setup. We should have powered it off and removed the power cord or taped over it or removed it from the rack or something, anything to make sure it wasn’t mistaken for an active box.

We did a full system shutdown, a fairly radical move during the middle of the day, but we were on spring break so there were minimal numbers of people that were affected. Some devices returned to normal status, but not all. Our assessment was that over time as each device reached out to the server for updates, connectivity would slowly be re-established. We had 24 hours until students returned to campus, so we figured we’d be fine by that time. The Sysadmin stuck around until 2am in a babysitting mode and left when he saw our assessment gradually get confirmed.

Word to the wise: don’t duplicate IP addresses on an active network. Mayhem will definitely occur.

Project Coordination and Communication

OK, so I’ve heard that the success of any project depends mostly on good planning. Frequently, my experience says that good planning is not enough. You also need good coordination and good communication. Just last month I experienced what can happen if those other two elements are badly done or missing entirely.

The project to which I’m referring was installing a card reader in an elevator. The elevator is in a residential building and only residents with card privileges are authorized to use the elevator. A successful outcome on a project such is this is dependent on coordination between the general contractor, the elevator technician, the electrical contractor, the door guy (me), and the elevator inspector. On a recent Monday morning I got  frantic calls from both the electrical contractor onsite and the college electrician. “The elevator inspector is due in 60 minutes and the card reader hasn’t been connected yet.”

Elevator inspectors are a breed apart. There aren’t many of them (at least in my state), they have their own timetable, and they can shut a project down or substantially delay completion following a failed inspection. You don’t get a do-over when you want it, you go to the back of the line and wait your turn.

When I arrived onsite, I began to see why there were some anxious folks. I had worked the previous week with two elevator technicians. I spent a good hour with them. Three business days had passed since then and my card reader still wasn’t installed AND . . . the guy I saw now wasn’t either one of the fellows from before. The same was true with the electrician. I had to start over with my explanations for what my system needed and they had to make it all happen in 60 minutes. Yeah, it was a nail biter, but they did it. My reader worked like a charm first time out.

Luckily, the inspector was 2o minutes late and an impending crisis just became another day on the job. But I plead for more direct and frequent communication on all projects. If you think you’ve notified everyone who should be involved and you’re feeling pretty smug about it, think about it one more time. I bet you forgot somebody.

Test Benches are a must

I had my system up and running for over two years before I recognized the value of creating a test lab in my office. I had heard a colleague talk several times at conferences about his test equipment, but didn’t really think it applied to me because he did access control full-time and it’s only part of my responsibilities. He beta tests new products and has a much larger campus so he’s got many more unique locations and functional needs than I think I ever will.

However, when a colleague on the other side of town told me last year that he made a test bench, then I thought harder about the topic. I’ve got many more locations setup on my campus and more aggressive expansion plans to boot. What did he know that I didn’t?

I realized that every time I wanted to create a new situation or test a protocol I’ve never used before, I was walking halfway across campus for a tw0-minute test. Then I’d walk back to my office and see how the system recorded the transaction. While I enjoyed the walk, it occurred to me that I could do these tests in a fraction of the time if I didn’t have to leave the office.

So I gave some of my spare parts inventory to my electrical contractor and described the board I wanted him to make and waited for a couple of weeks for the finished product. The board simulates a master controller installation in a non-existent building and is equipped with two door configurations. One of the doors is controlled by an  HID  iClass prox reader for testing user privileges and the other is a virtual door that has door contacts, a request to exit and a small light to act as an indicator of an alarm condition.

I can create simulated conditions for every type of alarm that I routinely use throughout the system. I can use the equipment for training sessions with other staff. I have a router on my desk that I can use to test new devices on either the access VLAN or the separate VLAN that my video surveillance cameras ride on.

The benefits of this equipment so close at hand prove themselves to me quite often. I can troubleshoot problems and test new scenarios much more quickly and of course, in a pinch,  I can grab one of the panels off the board and use it for a replacement part in an emergency on campus. I love my test board!

Cameras get added to the mix

There’s something that can be done with doors besides passing through them. You can watch them. About a year after I inherited the door access system on campus, I was asked to start a video surveillance network to visually monitor and record activity at some locations. We have been using cameras in our Recreation Center and in the Gould Library for years, but both of these installations are analog networks and only one of them is being recorded. We were looking for something that could ride on our data network and could do event recording

We chose the NICE Vision software solution and are using cameras from Sony and Arecont.

The first few that I got up and running were in a couple of new residence halls. I’ve since added cameras at a remote warehouse and in a dining facility. When I started this project I knew absolutely nothing about video (I’ve still just scratched the surface of the knowledge base), but I’m not bashful about asking questions so with the help of a couple of installation engineers,  I managed to get my little network up and running.

It’s still a challenge to pick the right camera for the job, site it properly in the area, and focus it so that you get a useful image. For awhile I was lugging my laptop around to perform the focusing exercise. I’d take a 100′ ethernet cable with me and plug it into the building switch while a colleague got up on a ladder and unzipped the camera innards. We’d shout back and forth until we got an image that was satisfactory. The whole business was tedious at best.

After whining about it to a few colleagues, I learned that you can buy an inexpensive battery-operated  monitor and plug it right into the camera onsite. Using one of these jobs, focusing becomes a single-handed affair. I was pumped for action with this baby. Unfortunately, you need to rely on a rechargeable battery and as usual with that kind of arrangement, you never know what the status of the charge level is at any given time. I hate being up on a ladder with three minutes left in the job and having the monitor go dark. Ugh.

I dislike binary situations such as this. Note to manufacturers: what would it hurt to put a battery indicator on your device?

Don’t Let This Happen to You

No matter how hard I try to plan and get out in front of a project, there’s always the risk that something beyond my control will compromise the results. It’s particularly annoying when the bad things don’t come to light until months after the project is complete. They’re like little time bombs set to go off at some future date under unpredictable circumstances.

My first use of offline door locks involved the Schlage CL lock, both the 5696 and the 5692 models. We put these on all the student room doors in two residence halls that we built in 2009. The  process of installation and commissioning was new territory for me and was complicated by the fact that the  General Contractor hired a different sub just to install the locks. What I didn’t learn until much later is that these guys got zero instructions on proper methodology. Unfortunately, their goal was to do their work in the quickest amount of time possible.

Fast forward to November 2010. I received a call from Security at 10 p.m. on a weeknight that the lock on a student room wouldn’t allow either resident into their room. I showed up onsite with my tools and a replacement lock and began troubleshooting. I determined that it wasn’t a card error, programming lapse or battery failure. Plain and simple: the electronics were shot. The only remedy was to replace the electronics. This is normally a simple matter. Only problem: the guys who installed the lock used a power tool to screw it to the door and guess what? They stripped the screws in their hurry to get ‘er done. In order to change the hardware out at that moment, my only recourse was to physically separate the device from the door. Since I couldn’t unscrew the screws and I didn’t have a hacksaw or a power drill, I wrenched the lock off the door with brute force. Not recommended.

Normally this backboard is inline, not bent at 90 degrees. I had to bend it and flex it back-and-forth until the top screw broke. I released the bottom screw as one normally does, by unscrewing it. I had to destroy a $750 device because some yahoo installed it improperly and I didn’t have the luxury of returning during business hours to drill it out. I have no idea how many of the other 172 locks are compromised in this way, but at least I’m aware of future potential issues. Word to the wise: make sure your subs get proper instructions.

Troubleshooting can drive you nuts

At one time or another everybody has a problem that seems to defy solution. It doesn’t matter if you read and re-read the manual, if you ask the nearest expert or holy man, check with your vendor’s Help Desk or the manufacturer’s tech support, you come up against a situation that you can’t fix.

I had that happen recently with a door prop alarm that exhibited a pattern I couldn’t explain . . . or change. Some condition would trigger the prop alarm at the door in question and it would stay in alarm state for hours. Suddenly, it would revert to normal, for no apparent reason, and within 90 seconds, it would change state once again to alarm.

When I checked the alarm log, I noticed  that this had been happening for not hours, not days or weeks, but months. Who knew?

My electrician and I tried every physical thing we could think of to eliminate  known causes. We changed the door position switch and both relevant panels at the equipment head-end. We toned the wires end-to-end. There were no shorts or signs of equipment failure. And still the alarm.

We observed user behavior at the door. Nothing.

On my second (or was it third?) call to the Help Desk, the tech remotely logged in to the server and started poking around. When he got to the module that configures  parameters for each location, he noticed that this one didn’t call for the POSSIBILITY of a door prop alarm. In other words, if you haven’t set the location up for accepting a given condition, it will never report as such.

The factory default calls for this potentiality, but somewhere in the past, it was changed for this location. Did I do that? I really don’t know. At any rate, after the tech added the parameter change, the alarm condition disappeared and my problem was solved.

I’m still appreciative about resolving this one and it happened over a month ago. I was plagued with this for weeks, but no more!

Projects come in all sizes

My first access control project was the largest one that I’ve yet completed. The assignment was to migrate all existing doors on the system from one platform (Casi-Rusco) to another (CS Gold). I was given an appropriate budget (I thought) and a timeline (get it up and running by August 30) and told to proceed.

August 30 was the deadline, but I didn’t mention what the start date was: August 1. I had to decommission the equipment that existed in 23 buildings, install the new equipment, commission the new equipment on the new software platform, and test all of the card readers and alarms in 30 days. The electrician’s team had four guys, but it rapidly became obvious to the owner that he needed more. He subbed in a couple of guys from another firm, but even those weren’t enough resources to make anyone feel remotely comfortable. It was a day-to-day grind for all of us.

This was in 2008. Prior to this project, I couldn’t have told you what a door force alarm was, what the difference between HID Prox and HID iClass was, why some doorways required electric strikes and some latch retraction, orwhen it would be appropriate to use online locks and when one should deploy offline locks. Talk about a baptism by fire.

In the end, we succeeded at getting system functionality operative by the deadline, but it would be another couple of months before we got everything debugged and all of the testing done. The whole experience turned out to one of the toughest challenges of my life. We prepared for it for six months, yet I had no idea going in that we hadn’t allocated enough time to do the install.

It’s now 2 1/2 years later and the pain has subsided, but I still marvel that we made it happen.