Secure Buildings Access Control (1993)

Something Seems "Fishy"

I should have realized at the start that this project smelt a bit "fishy". I had been hired by a past colleague manager for a 6 month contract to "just wrap up this major project to sign-off acceptance". The warning sign was that the original development team had resigned 'en mass'.

But times were tough, the clients were prestigious and I was up for a challenge. The project - "Building Security and Access Control" at the Australian Federal Parliament House (the security pass office operated by the Australian Federal Police), and the Department of Defense Security Office (so at least we didn't have the prospect of the clients going bankrupt on us).

My colleague took over the project management, I was the general technical "dog's body", tester, implementer, trainer and day-to-day client liaison, and we got one developer back to do code fixes.

The purpose of this blog is not to describe the whole project, but just to high-light some interesting, even humorous, aspects of the projects and lessons to learned.

Be Very Careful of System Parameters with a Sensitive Human Aspect

One aspect of the implementation at DoD, was interfacing to an "intelligent" revolving door. These doors had two special security features. There was an air-volume displacement measure to detect if two people tried to go through the door together. The other was a sonic detector of flat surfaces to detect people exiting with boxes etc.  These detectors were connected to a recorded voice warning enunciater. 

The first warning that we had of the need to tweek the parameters was when one of the more 'robust' ladies of the DoD tried to enter - the door went into reverse to back her out, announcing "One at a time please!" Oops!

Next we had an Admiral try to exit wearing his (flat top) hat. Oops! (Staff, including Admirals, had to be reminded that protocol barring wearing of hats indoors, included inside the exit doors).

The Most Difficult Part of Government Contracts is Getting Paid

The major problem of both projects was the lack of any robust, agreed set of Acceptance Criteria. And government departments are especially sticklers for not paying a cent until satisfactory acceptance has been signed-off. This is a real trap for small technology companies, as this was, who seem to think that all they need to do is to develop a good technical solution.

One thing I have learned over the years is that "Acceptance" is NOT "Testing" and that "User Acceptance Testing" is a misnomer.  

The Acceptance "Demonstration" must be specified with a well defined set of inputs which will produce the expected, documented outputs. "Dry runs" of "Acceptance" must be run before-hand error-free before attempting the sign-off demonstration to the customer. The acceptance criteria must NOT allow unfettered trial-and-error usage. There must be no unexpected errors. Demonstration to the customer for acceptance sign-off is not the time to be still discovering bugs. After acceptance, there will be a warranty period in which bugs uncovered by the customer during normal use, can be fixed.

If a "big bang" acceptance sign-off is of concern to the customer, then negotiate the contract with staged payments with the majority paid on primary "acceptance", and the balance after some defined settling-in period - define incident/problem severity levels and acceptance of settling-in defined along the lines of "2 weeks with no 'Severity 1' errors, and no more than 2 'Severity 2' errors per week" (as an example).

Getting Burned

Did we finally get paid? Not in my time there. Four months in, my pay cheques stopped coming. I stuck it out to the end of my term, but soon after, the company filed for bankruptcy and I lost out 2 months salary and a month's travel and accommodation expenses. Just before the end, two more clients had been signed up for the system. That part of the business was the only section to survive, sold off at 10 cents in the dollar and the administrators were chasing payment of the above contracts.

Client-Server By Mag-tape Exchange (1983)

I had been hired by this Australian national office of a multi-national mini-computer manufacturer, to setup the internal data processing systems.  

Spare-Parts Inventory Tracking

A major issue was their spare-parts management which was getting "slammed" by the auditors for enormous write-offs/adjustments after every annual stock-take.

The spare-parts system was a national, warehouse-centric system using the US head-office software. All transactions came in on "movement tickets" for keying. These covered engineer "consumption" in repair of client machines, inter-office transfers (of both good and faulty parts) and repairs (faulty in, good repaired out). The potential for transcription and keying errors was high. There was the issue of "handedness" where left-hand and right-hand variants had similar part numbers. But from a financial point-of-view, variants of circuit boards was more critical - a more expensive board with more memory or a faster processor might vary in part number only by a suffix.

Customer "Loans" Issue

But it was "home-brew" software by one branch office for tracking their own spare parts use, that finally brought home the real culprit. On service calls, engineers took out some good parts in anticipation of what might be wrong, then on site replaced the faulty customer part with a good part. If they were identical part numbers then all good. But if an identical good part was not available, the engineer would put in a good part of a higher rating (eg. more memory), so the faulty part brought in (for repair) had a different part number. In theory, when the customer's part was repaired, it was supposed to be taken back, installed and the original good part brought back. This "home-brew" local system kept track of these "loans" and reminded the engineers. A similar issue arose with sales reps. "lending" customers a higher rated board to trial before buying. Again, the national system did not cater for this concept of "loaning" parts to customers.

On-line Data Entry Client with Transaction Logging

My solution had a couple of phases. Firstly, the new system would be multi-user on-line data entry at each point where the original movement tickets would have been written. On-line data entry then provided immediate data validation of part numbers. A new transaction type of "Loan" was then introduced - this was expanded from the loan to customer", to recording of ALL good parts that a service engineer took on his service call and recording everything he brought back, good, faulty or alternate part number. Initially, inter-office movements still needed the manually prepared movement-ticket from the non-automated office.

The second phase was implementation of this system in all the branch offices. This is especially important in a country as geographically dispersed as Australia. In the states of Queensland and Western Australia, engineers go out on week-long circuits of preventive maintenance of remote sites and so have to take a large stock of spare parts with them.

Transaction Server

Both phases had issues that led to a single solution. Real-time data entry into a system that did not have a robust database with transaction level data integrity, had potential of data loss with no fall-back paper work for data re-entry. At this time, the INTERNET was not yet available, and leased-line networking to all the branch offices was not economical.

My solution was a custom developed transaction processing system, which we would now call "client-server". Multiple data-entry front-ends accessed the database read-only for data validation, and compiled a transaction record that was sent (by inter-process communication) to a single threaded database updater process. The first step of the updater was to write the transaction records to a serial transaction log. These transaction logs were backed-up daily. In the branch offices, these transaction log files were written to mag tape and sent to head-office along with the regular parts-for-repair transfer. At head office, the transaction log files for all branches were then read (off tape) and fed into the same head-office database updater process, so that a National database was now available. For the first time, the national spare-parts planner had a reasonably up-to-date picture of the distribution of parts across all offices. At head-office, parts receipting now simply checked off the electronic parts transfer "ticket" received from the sending office. New weekly reports to each branch manager listed all parts currently out on "loan" that had to be retrieved or swapped-back.

The test of this system of processing transaction logs files, came when the National system first went live. The Melbourne (National) office had been using what would become the Branch system. When the National database was initialized with the backup after the last stock-take, we simply processed six-months worth of transaction logs to bring the National system up-to-date, without a hiccup.

International Adoption

Subsequently, this system was adopted by the European subsidiaries who were having the same issues with the US warehouse-centric system.

Integration in Service Call Management

Subsequently, this system was integrated into a Maintenance Call Centre system for dispatching engineers to service calls, recording of travel, time and parts used, which all fed into the service costs analysis system.

Where's the Fire? (1987)

I was working at one of Melbourne' large stock-broking firms, implementing the interface between their back-office system and the accounting software we marketed, as well as developing customized reports. One morning, a whistle blew. I looked around, thinking it was perhaps a fire drill, but no-one was moving. Oh well, it can't be important, so I worked on. About 15 minutes later, the whistle sounded again, and again no-one moved. 

I asked the client colleague working near me, what was the whistle about. He explained that some 6 months earlier, soon after the back-office software was first installed, they found they were getting intermittent system crashes. The software supplier was called in, and after investigation, they declared that there was a critical administrative update that conflicted with other normal operations. The solution they provided, was a whistle that the administrator had to blow warning other users that a critical update was about to be performed and that the other users needed to exit any update functions they were performing. On completion of the critical update, the whistle was blown again to signal the "all clear".

As a work-around, their simple "hardware" solution worked and was certainly cheap. I don't know whether they actually provided a software interlock on the critical code section in the next software release or not!

Fingers in the Till - Forensic Initiation (c.1986)

I was doing a little bit of maintenance work on the software we had built for a large cinema chain (support of new ticket printing hardware). Their IT manager called me aside confidentially (oops, what have I done?).  But he explained that management thought they had a problem with cash going missing from the till of the concession (snack bar). Would I be able to analyse their cash register data log to report shortages/surpluses per shift per user with long-term averages?  

Fortunately, the cash/sales reconciliation software had been well designed and recorded everything needed, so without too much effort we could quickly see that one particular operator quite consistently recorded a cash shortage, whereas the average of all other operators was essentially a zero balance. With a Statutory Declaration attached to my signed report, I heard nothing more other than that the subject staff-person no longer worked for them.

It was quite a simple job but was my first taste of "forensic" data analysis and reinforced the importance of not just updating a database, but when it comes to money, the importance of also recording who, what and when.

Interpretive Dynamic Graphic Displays (1980)

This is a follow-on post to "Sewage Works Data Acquisition and Process Control".

Graphics Displays "Tailorable" by End Users?

Yes, that was in the "Requirements". When I arrived on the project, a couple of engineers were looking at how they could design a "paint" program! That seemed like total over-kill for a piece of functionality that would be very rarely used. Sometimes we must be "creative" in interpretation of wording of "requirements".

The Graphics Hardware and Drawing Language

Fortunately, the engineering company had selected and purchased a sophisticated, programmable graphics display terminal (RAMTEK). It had an inbuilt interpreter for a drawing language. The drawing instructions were fed into the terminal as an ASCII data stream.  For example:-

        [COLOR RED][RECT <x-value>,<y-value>,<width value>,<height value>]

All <values> were in pixels. (Forgive me RAMTEK if I haven't remembered your syntax correctly). So in fact it was quite reasonable to draw up the required layout on graph paper and code the required drawing sequence. In today's parlance, we could call it a "mark-up language".

Language Extensions to make it Dynamic

But the graphics we required had to display dynamic data in "real-time". There would be meter readings, red/yellow/green signals, switches/stop-valves that indicated "open/closed", arrows with changeable direction, etc. - think "Dashboard".

My solution was to extend the RAMTEK language with three simple constructs.

• Firstly, values could be replaced by named variables that referred to values in the in-memory data streams. There are two types of variable, booleans (ON/OFF) and integers, represented as "Bnnn" and "Innn".
• Secondly, labels could be assigned to any statement - eg. Lnnn:.
• Thirdly, a simple variable test construct was defined with simple boolean and logical operators, that if tests "true" directs "execution" to a specified label (eg. [IF B123 ON L456].

In today's parlance, this is known as "templating" (eg. PHP). The implementation was our own interpreter program that processed a template file, substituting variables with their real-time values, performing the testing and branching, continuously outputting a stream of drawing commands to the terminal. The end of a template file simply looped back to the start. So the "real-time" screen refresh was determined by how quickly this interpretation loop executed.

Was It "Tailorable" by End Users?

The "template" file (program?) was a simple text file of reasonably readable instructions. Sure, it was tedious to update but essentially clerical, so we argued that it was "tailorable".

The Actual Display Graphics

There were three different displays delivered with the system.

• The hydrolic flows through the works (the first stage of the plant is to filter and separate the liquid from the solid 'sludge').

• The "sludge" system monitoring. (The sludge is pumped through large anaerobic digester tanks, the active bacteria in which are highly sensitive to temperature and acidity, so monitoring of temperatures and pump flows is critical in such a plant).

(apologies for the hand-drawn schematic - that's all I have left from the project)

• The electrical generation system. The above anaerobic digesters produce methane which is drawn off and feeds electricity generators which power the plant as well as sending excess power back into the city power grid. Its graphic display showed generators' status, voltages, currents, circuit-breaker statuses, etc.

Sewage Works Data Acquisition and Process Control (1980)

"Real Time" or "Data Processing"?

The sewage works of this large New Zealand city was in need of major redevelopment and modernization.

A civil engineering company was the prime contractor, and an electrical automation engineering company was sub-contracted to build the plant control systems. They had experience in micro-processor controllers for basic automation, but they needed someone to take on the back-end computing requirements of control-room data presentation and all levels of reporting.

When I arrived, their engineers were looking at (small) technical issues around the periphery of the requirements. The first thing I did was to review the requirements holistically and perform a chronological functional decomposition and came up with the following Data Flow Diagram.

(apologies for the hand-drawn diagram - that's all I have left from the project)

The engineering management were stunned when I showed them that less than 20% of the system was "real time" and the majority was "data processing" not unlike many commercial systems.

The Central Data Flow

The raw input data from the plant comprised 700 analog measurements (flow rates, tank levels, temperatures, voltage and current), and 150 digital (binary) "alarm" signals. All these inputs were sampled 4 times/second. A simple data communications protocol was designed to interface with the micro-processor controlled electrical connection panels.

Current (Real Time) Data

The "current" data status of the plant was required to be available for display on graphic displays in the control room and selected "alarm" status changes needed to be logged on a printer. The very nature of the type of plant being monitored and the data processing responsiveness of the graphics and printing showed that a 2 second response time was quite adequate, so the design was to average input data in blocks of 8 values to give 2 second averages.

But this data rate was still too fast to write to disk and the "real time" graphic display would not be responsive enough if it had to continually retrieve data from disk. The solution was to use a "mapped, shared memory" data pool. The data was also accumulated in buffers for writing to disk files at 30 second intervals.

Data Accumulation

A series of batch jobs then ran hourly, 8 hourly (per shift), daily and monthly, to summarize results from the lower level and write into the accumulation file at the next level.

Flexible Data Management - self-describing meta-data

A core feature of the system requirements, was for maximum flexibility in describing data, both the real-time inputs and the as yet undetermined manual inputs (typically the results of chemical analyses). Similarly, reports' layouts and contents required maximum flexibility of design and changes. A highly table-driven system was designed.  

But all these tables would need to be maintained (and possibly additional tables added later), so a table-driven table maintenance method was designed with its own table of meta-data that described the contents of the other tables to be maintained. The obvious (?) approach was to have it "self-describing" - the first entries in the meta-data table were manually entered to describe the meta-data table itself. The data maintenance program was then used to populate the remainder of the meta-data table, which could then be used to populate the system data tables.

Files were defined as a collection of records of a given type, and records were defined as a set of data items of various types.  Data items were defined with a long and short name, a data type and some very simple range constraints.

Process Control

A generic process control module was developed, with control tables associating outputs with the feedback inputs along with a set of process control parameters (2nd order function).

Generic Report Definitions

Shift and Daily reports were quite simple lists. Report definition tables specified which variables were to be reported.

The Monthly reports were all data analysis type reports. A given, fixed set of various sorts of analysis algorithms were required (various sorts of means, statistical measures, etc). Again, report definition tables specified which algorithm was to be applied to which set of variables for each report.

Summary

In all, a very satisfactory result was achieved. I was assisted by two new university graduates on a vacation contract, to complete the system in 6 months.