Check out our White Paper Series!
A complete library of helpful advice and survival guides for every aspect of system monitoring and control.
1-800-693-0351
Have a specific question? Ask our team of expert engineers and get a specific answer!
Sign up for the next DPS Factory Training!
Whether you're new to our equipment or you've used it for years, DPS factory training is the best way to get more from your monitoring.
Reserve Your Seat TodayA few years ago, we were approached by a major subway system operator to replace their legacy audio selection/distribution panels. These are conceptually very similar to analog comparators (also known as "voter comparators"), except the channels are selected by priority instead of receiver voting. The decision isn't simply based on a voting system choosing the receiver with the best signal quality and outputting the voted audio.
There were many issues with the old audio panels that the subway system was looking to solve:
Despite all these limitations, the system essentially worked at some level. If you've worked with telecom infrastructure for any period of time, you know that it's difficult to justify replacing something that is functioning. Sometimes, it takes a specific incident to give the organization a good push.
For this particular transit authority, that push came in 2017. A police officer started his duty shift as he always did: by inserting his handheld PTT radio into its holster.
Unfortunately, on this idle weekday, he inserted the radio at the wrong angle. This pinned down the push-to-talk button, causing a constant and transmission with an unknown source. The officer didn't know this was happening, and nobody else could easily locate the signal source.
Since the existing audio selection/distribution panels had only rigid, predefined built-in logic, those devices handled this in the only way they could. Police, as you might imagine, are given the highest possible priority (no "receiver voting" logic is used) within the radio system. They're capable of overriding transmissions from even the central rail controllers that manage the flow of subway trains throughout the network.
At this point, the entire subway system was suppressed by what was effectively an unintentional radio jamming attack. This persisted for hours and hours until the constantly transmitting police officer's radio drained its battery. In the meantime, the entire train network had to be managed with makeshift communications, including consumer-grade cell phones.
At DPS, I'd been supplying this particular subway system with remote monitoring devices for two decades. They're among the most prolific users of NetGuardian RTU's and T/Mon master stations in the world.
With that built-in familiarity, the senior management at this transit agency asked me a question. Could I offer a solution that would prevent any future radio jamming incidents?
The goal was to build a new box that would replicate the functionality of the old audio selection panel. It would then take things up several notches. It would have tremendously more built-in intelligence and remote management capability.
When we were contacted by our long-time client with a new challenge, we started developing a solution right away.
First and foremost, it would prevent any future "stuck carrier" problem with a simple algorithm.
The inciting incident for this new product development was a handheld radio that jammed systemwide communication. This was the first thing we needed to solve.
Working together with subway representatives, we all quickly rallied around the idea that some kind of timeout would be required. After a user-definable period had elapsed with a channel constantly broadcasting, that channel will be locked out.
What took longer to hash out was how a channel should be restored after being locked out. Eventually, we agreed to provide two options that can be selected in the web interface.
The first option was for a manual restoration to service. If the same incident were to happen again, the police radio channel would be suppressed after 60 seconds of continuous carrier signal. The central administrative team for subway telecommunications would then need to log into the web interface. The team could manually restore service to that radio channel after the incident had ended.
That first option is obviously the most protective of radio availability, but it comes at a cost. It could potentially suppress police communications that are, in fact, important. That's why we would build a second option.
The second option would be for a communications channel to automatically reactivate after a period of silence. If that channel was blocked after 60 seconds of continuous transmission, it could be unblocked after another period of silenc. For example, five continuous seconds without a 2175 Hz guard tone.
Once I'm approached to fix a legacy box, I always try to find opportunities to add more value. In this case, the many undesirable functions that I listed for you earlier gave me tons of inspiration.
We would tackle overheating and unreliable power supplies, inadequate prioritization granularity, screwdriver-based potentiometer adjustments, and more.
We would avoid digital converters, instead using modern technology to manage the analog audio needed by the subway's various systems. This kept audio quality lossless.
We would also, during our 3 field trials, discover plenty of other opportunities that we could never have anticipated without visiting in person.
Having built an initial prototype of a new "NetGuardian ADP", two other DPS engineers and I hopped on a plane. We traveled from our headquarters in California to visit a team of subway engineers.
The front panel of our first-revision NetGuardian ADP hardware included status LEDs and an LCD. These reflect the input state and output state of the box at all times.
The back panel of our new NetGuardian ADP was designed specifically to mimic the precise placement of ports on the old box. This made replacement a breeze for the install teams. Notice the screw-down connectors for power and the cylindrical fuse, which were changed in the next revision to match our client's preference.
Within their simulation lab, we started a week of initial testing. As is usually the case, our initial enthusiasm was quickly dashed in the harsh light of reality. Obviously, this kind of experiential learning is exactly why you do a field trial, but it's always fairly disheartening initially.
Right out of the gate, we realized that we hadn't adequately noted the desire for dual hot-swappable power supplies. In our minds, we know that NetGuardian power supplies are incredibly reliable and proven themselves around the world.
Nonetheless, we knew that this particular client had been traumatized by the old box and is unreliable power supply. A dual hot-swappable design would provide redundancy and also the ability to quickly swap in a spare when necessary.
And so, we had launched our first typical "field trial roller coaster" of a week. Inevitably, the first half of any field trial is filled with various disappointments. We learn the various ways that our best-guess design just doesn't cut it. This always stings a little, but that's exactly why you do a field trial in the first place.
Throughout the rest of the week, we continued to test our firmware in the subway's simulation lab. We discovered several unplanned realities about the subway's network, including much smaller signal-to-noise ratios on some of the longer lines.
That led us to take important notes about the next revision of the hardware. This would include the ability to tune comparator outputs at levels that were appropriate for the amount of wire being driven.
With five full days of testing under our belts, we returned to California. We had enough pages of design notes to keep us busy for at least a few weeks.
After two months and another circuitboard revision, we flew back to the client site for another field trial. Not only had the hardware improved, but we had made the single greatest leap forward in our firmware design that the project would see.
The new hardware design had dual, hot-swappable power input cards. We also changed the power connectors themselves to be pluggable instead of screw-down. This further reduces the labor required to change out a spare if it becomes necessary.
Our second hardware revision included dual, hot-swappable power supply cards. Notice also that the LEDs evolved into very clear RGB lights to support more than just red and green.
The key elements of the back panel remained the same in revision 2, but there were differences. The power connectors and fuses were changed to match the design our client already liked from the NetGuardian 832A RTU.
We had switched to full RGB color LEDs throughout the design. This was a very important improvement to give us more than two colors to reflect system status. Traditionally, a bicolor red-green LED is sufficient for showing transmit and receive. With this NetGuardian ADP, we needed at least one additional color (we settled on blue) to represent when a channel is blocked after excessive transmit duration.
The firmware now had some truly cool features that proved their worth in this field trial.
We now had support for adjustable detect levels. Users would have complete freedom to select the decibel level. The could also choose whether a signal needed to include trigger conditions. Options here include a +0 dB high-level guard tone (HLGT) and/or a -26 dB low-level guard tone (LLGT).
We built temperature monitoring into the unit. That's not directly related to this device's audio function, but it's an easy thing to add, and it's a very handy level to know.
After we encountered some difficulty during the first field trial related to audio troubleshooting, it made sense for us to add test-tone generation into our product. This was immediately useful. We were able to send a test tone from one NetGuardian ADP. We'd then see if that signal would properly trigger another on the far end.
Finally, for even deeper analysis of the audio levels, we built in a true oscilloscope, which we call the "Analysis Scope". This function also proved its worth as we hacked our way through unexpected operation on noisy audio lines. We also managed to stamp out a bug related to a rare-but-possible event that would corrupt measured audio values and disrupt related functions.
This field trial went so much better than the first. We added a ton of new functions, and the client really seemed to be enjoying the process and the product.
Still, we had a few more notes that required a bit of additional development. We returned home after promising to come back for one final week of testing.
Although our biggest product design upgrades came between field trials one and two, there was more to do. We returned back to our office in Fresno, California with several new things to add for the final design.
We did some additional fine tuning in the web interface, ensuring that all of the functions and default settings were precisely what our client intended. Obviously, we were designing the NetGuardian ADP with an eye toward applicability at many different entities. Still, we first needed to satisfy and delight our very first client for this new custom product.
At this stage, it was time for me to embrace my documentarian role. I had previously written preliminary documentation into a PDF user manual from my hotel room. I was updating this on-the-fly during the field trials to support testing and evaluation.
To make this new product far easier to use, I inserted helpful "how to use the screen" notes on each page of the web interface. That's how modern devices should function. It eliminates the very real risk that, years from now, somebody will struggle to make adjustments without a manual.
Finally, we were incredibly excited to have developed an automatic tuning function. The idea here was that manual adjustment of the tuning level would no longer be required.
At our offices, this feature went through several stages. The team started out with an "cruise control" adaptive algorithm. The idea here is that the system moves from its current location to the ideal location in a smooth fashion. This sacrifices speed in the interest of smoothness.
Ultimately, however, this algorithm proved to be overly complicated and unnecessarily slow. Although it only took a few seconds, using this method would require installers to key the radio several times to achieve automatic tuning. During this phase, it's possible that messages would be too loud or too quiet to be understood properly.
I suggested to the team the choice that we eventually made: "Why not just make a 0-255 reference voltage lookup table?"
In this incredibly simple algorithm, the input signal strength (dB) would be translated to a step level on the digital potentiometer controlling signal gain. Since there were no other factors in the signal chain of our device, this was the one and only consideration that needed to be made.
What's more, the simple algorithm would auto-tune at high speed within about 5ms on each and every radio broadcast. It could theoretically be left on to automatically adapt to changing line conditions. I still chose to give the users an option to disable auto-tune when more manual control was desired.
We flew back out to the client site with this final version of hardware and firmware in hand. We made just a few firmware tweaks during the week. The "roller coaster" experience of the first field trial a few months prior seemed like a distant memory. Things were going incredibly well.
I prepared this complete table for our end user during development. It describes every significant difference (and a few useful similarities) between the old box and our new custom solution.
As excited as everyone was to start rolling out the new NetGuardian ADP, it's always important to do a reasonable amount of endurance testing. We had 11 units deployed (with a handful in labs but the rest seeing real use in the comms network).
Since we were looking at a full deployment of well over 100 units, we chose to let those units run in place for a full year. They had their own performance logging, which was occasionally exported for review.
Because any embedded remote device has relatively limited storage, we were also using a deployed T/Mon master station. It was set to poll each ADP roughly every 30 seconds. Any alarm conditions were logged indefinitely on T/Mon's redundant hard disks.
After all of our efforts during the 3 field trial visits, the year of endurance testing went incredibly well. We had no hardware failures, and only a handful of tiny firmware changes to address:
We're now very excited to roll out the NetGuardian ADP to all remote receiver sites for this massive DPS client. Although the evolving Coronavirus reactions have hampered large segments of society, we press on. Virtually every DPS user is an "essential business" or government agency that must continue to serve the public. Therefore, so are we.
We're tidying up the final firmware adjustments. We're helping our client secure funding internally for the project.
As you might expect, our long history with this user is helpful. It makes us a known quantity for key decision makers who must always manage risk. 20 years of proven NetGuardian performance mitigate a ton of that risk.
This is just one example of custom remote monitoring development from DPS Telecom. It took us beyond our core of RTUs and master stations in many ways. We incorporated brand-new audio circuitry that will be useful in many different applications.
What development story do YOU want to be part of? What can DPS make for you?
Give me a call at 1-800-693-0351 or send me a quick message to tell me what problem you're facing. I promise, it's a fun (and sometimes wild) ride that always ends smoothly - and makes your job easier.
Andrew Erickson
Andrew Erickson is an Application Engineer at DPS Telecom, a manufacturer of semi-custom remote alarm monitoring systems based in Fresno, California. Andrew brings more than 17 years of experience building site monitoring solutions, developing intuitive user interfaces and documentation, and opt...
