Engineering Maintenance Management (1985)

Introduction

Although this was a mini-computer supplier, computerisation of business functions was still slow, especially in this small Australian subsidiary of this US manufacturer. A lot of business systems were still typically paper based data collection keyed into batch systems for reporting.

We were in the process of bringing the spare-parts inventory control system on-line (which project was initiated from finance and audit). A new National Field Engineering Manager had been hired who had started asking questions that the existing systems could not support, such as device type failure rates, average cost to repair, customer call response times, etc.

Existing Systems

The spare-parts inventory management system is described elsewhere which introduced tracking of good parts taken out on service calls and the non-equivalent part swap-out process.

A customer equipment maintenance contract system had records of all equipment under service but was primarily an invoicing system.

Every service call had a work-sheet completed detailing call, response, travel and completion times, equipment repaired, parts used, etc. A local system had been developed for data entry of this data and some basic reporting had been developed. The annual budgeting system used gross counts of staff and service calls to compute utilization and combined with equipment population and sales projections, produced projected staff requirements.

Enhancements - Stage 1

The first stage was to enhance the work-sheet data entry application with full data validation of customer contracts, equipment, spare-parts and engineers. The database created for the spare-parts inventory system, with its transaction logging facility, was enhanced to record the work-sheet data with improved data structures and indexing for better reporting. Outputs from this system them automatically fed into the annual budgeting process supplying actual totals for the year.

Some interesting results were starting to be seen in the reports from this data, which lead to the decision to go ahead with a full Call Centre System.

Enhancements - Stage 2 - Call Centre Management

There were two key drivers for the call-centre system. First was to capture and validate customer call information while the customer was still on the phone, including precise identification of the equipment at fault. We had found a number of customers were not putting all their equipment under service contract and were logging so-called contract service calls for non-contract equipment (this was especially easy with terminals - a customer might have 20 terminals under contract, but in fact have 100, often bought via the "grey market").

The second driver was to maximise engineer productivity. The engineer would call in job completion from the customer site, being lead through a predetermined list of responses to capture his full "job sheet" including (for the first time) the actual device id of the equipment being repaired.  He could then be directed immediately to his next call without having to return to the depot.

Successes

All in all, the above systems proved very successful. Three successes stand out.

First, by exactly identifying the equipment items being serviced, we were able to bring a lot of "grey" equipment under contract.

The volume of terminals being serviced by "swap-out" brought to light the idea of having a service van just full of terminals, circulating in the city with a courier who could do the "swap-over" rather that incurring the cost of a full engineer service call.

By the end of the first year when we started analysing device type failure rates and cost-to-repair, it became obvious that a particular model of terminal was so fault prone, that its average cost-to-repair was not covered by the contract service price. A heavily discounted replacement sales programme was put into place to upgrade all these devices to more modern, more reliable types.  This was a result that our American head-office had not even picked-up on.

Who's the Client? What's the Deliverable? (1995-7)

Introduction

I hope you will understand my discretion in suppressing certain details and names of companies.  The project was a massive, city-wide, integrated, distributed system with a high daily transaction rate and cash flow, developed for and to be run for the state government.

The contract was won by a Joint Venture Consortium formed between a hardware supplier, a multi-national computer supplier (my employer) supplying the central computers and central management and reporting software, and a cash-services company.

The Warning Signs

We Software Engineers pressed on in typical bespoke software development fashion, collecting and documenting requirements, developing functional design specifications for review and sign-off by the client (end-user SMEs). But obtaining "sign-off" was like trying to extract "hen's teeth" - we put it down to typical public-servants' reluctance to put their name "on the dotted line" and take responsibility for their decision.

Next was the continuous pressure for requirements "creep". Whilst both major suppliers had formal documentation, quality and change-control processes, they were different and weren't coordinated - one came from the engineering discipline and the other from commercial software development backgrounds.  What was seen as acceptable to one supplier was seen as a Change Request to the other.

As the pressure of time and budget increased, the above scope-creep issues had a strange effect on the over-all programme management style. I call it "pendulum management". In alternate months, the key management message alternated from "tighten up, work to budget and time-line", then "keep the customer happy, give him what ever he wants".

The "aha" moment came when we came to specify the functionality for managing the discrepancies between sales transaction data and cash collected. We went to the client users for their "requirements" and, to our surprise", were told, "Its not our problem. Its the consortium's problem.  We simply require that you pay us the higher of the cash collected (assuming sales transactions were lost), or the sale transactions amount (assuming cash has been lost)" - it was a classic "heads they win, tails we lose" situation.

Who's the Client?

This was the point when the "client dilemma" really struck home (to we software engineers at least). The Consortium's contract was in fact a 10 year "Service Contract", to first of all to build the system for an up-front payment (with ownership retained by the consortium), and then to operate the system including provision of enquiry and reporting services to the public servants. The "requirements" of the system being built were to "provide the contracted service".

Our "cash-to-sales reconciliation" problem (above) needed to be resolved by the consortium's operational and accounting staff, who had as yet not been appointed. In fact, at this point, the consortium's operating company comprised a single project manager! The two major partners in the consortium had forged ahead almost independently with no thought or plans for how the ongoing service would be provided and any of its impacts on the requirements of the systems being built.

Our Cash-to-Sales Reconciliation Solution

A major source of issues in cash-to-sales reconciliation, was the distributed nature of the POS equipment and in about half the cases, manual transfer of transaction data (by key-drive type device) to interface into the central system. Transaction batch identification had been included and a certain degree of redundancy. What was agreed as being required, was initial reconciliation when collected cash was counted per identified transaction batch (or group of data batches). Data completeness integrity controls would be needed so that an alert could be flagged indicating that it was known that some data was missing. Then when/if the missing data did arrive (possibly via error correction) an adjustment could be raised against the matching reconciliation and a CR/DR raised.

Needless to say, such a level of audit control had not been anticipated nor built by the hardware supplier and some robust negotiation around changes to the core data interfaces was required. To be fair to the hardware supplier, they had an enormous micro-software change control problem in coordinated distribution of updates across hundreds of devices, not to mention the data storage chips attached to every cash container.

(Technical aside: The core issue revolved around a forward singly-linked list being less robust that a bi-directional doubly linked list).

The End-Game

The total system finally went into production, grossly late and over budget (there were political imperatives (read election) that the system must not "fail"). The "service" has run well past the original 10 years, since the replacement system has had numerous problems of its own.

There was "robust negotiation" (litigation?) between the government and the consortium over the cost of the over-run and scope-creep (with eventual confidential settlement). I was involved in extensive "function point analysis" of the original (very vague) contractual (service) requirements and the system "as built" in order to identify and quantify the scope increase. But even this exercise was predicated on the bespoke software development model.

Client-Server By Mag-tape Exchange (1983)

I had been hired by this Australian national office of a multi-national mini-computer manufacturer, to setup the internal data processing systems.  

Spare-Parts Inventory Tracking

A major issue was their spare-parts management which was getting "slammed" by the auditors for enormous write-offs/adjustments after every annual stock-take.

The spare-parts system was a national, warehouse-centric system using the US head-office software. All transactions came in on "movement tickets" for keying. These covered engineer "consumption" in repair of client machines, inter-office transfers (of both good and faulty parts) and repairs (faulty in, good repaired out). The potential for transcription and keying errors was high. There was the issue of "handedness" where left-hand and right-hand variants had similar part numbers. But from a financial point-of-view, variants of circuit boards was more critical - a more expensive board with more memory or a faster processor might vary in part number only by a suffix.

Customer "Loans" Issue

But it was "home-brew" software by one branch office for tracking their own spare parts use, that finally brought home the real culprit. On service calls, engineers took out some good parts in anticipation of what might be wrong, then on site replaced the faulty customer part with a good part. If they were identical part numbers then all good. But if an identical good part was not available, the engineer would put in a good part of a higher rating (eg. more memory), so the faulty part brought in (for repair) had a different part number. In theory, when the customer's part was repaired, it was supposed to be taken back, installed and the original good part brought back. This "home-brew" local system kept track of these "loans" and reminded the engineers. A similar issue arose with sales reps. "lending" customers a higher rated board to trial before buying. Again, the national system did not cater for this concept of "loaning" parts to customers.

On-line Data Entry Client with Transaction Logging

My solution had a couple of phases. Firstly, the new system would be multi-user on-line data entry at each point where the original movement tickets would have been written. On-line data entry then provided immediate data validation of part numbers. A new transaction type of "Loan" was then introduced - this was expanded from the loan to customer", to recording of ALL good parts that a service engineer took on his service call and recording everything he brought back, good, faulty or alternate part number. Initially, inter-office movements still needed the manually prepared movement-ticket from the non-automated office.

The second phase was implementation of this system in all the branch offices. This is especially important in a country as geographically dispersed as Australia. In the states of Queensland and Western Australia, engineers go out on week-long circuits of preventive maintenance of remote sites and so have to take a large stock of spare parts with them.

Transaction Server

Both phases had issues that led to a single solution. Real-time data entry into a system that did not have a robust database with transaction level data integrity, had potential of data loss with no fall-back paper work for data re-entry. At this time, the INTERNET was not yet available, and leased-line networking to all the branch offices was not economical.

My solution was a custom developed transaction processing system, which we would now call "client-server". Multiple data-entry front-ends accessed the database read-only for data validation, and compiled a transaction record that was sent (by inter-process communication) to a single threaded database updater process. The first step of the updater was to write the transaction records to a serial transaction log. These transaction logs were backed-up daily. In the branch offices, these transaction log files were written to mag tape and sent to head-office along with the regular parts-for-repair transfer. At head office, the transaction log files for all branches were then read (off tape) and fed into the same head-office database updater process, so that a National database was now available. For the first time, the national spare-parts planner had a reasonably up-to-date picture of the distribution of parts across all offices. At head-office, parts receipting now simply checked off the electronic parts transfer "ticket" received from the sending office. New weekly reports to each branch manager listed all parts currently out on "loan" that had to be retrieved or swapped-back.

The test of this system of processing transaction logs files, came when the National system first went live. The Melbourne (National) office had been using what would become the Branch system. When the National database was initialized with the backup after the last stock-take, we simply processed six-months worth of transaction logs to bring the National system up-to-date, without a hiccup.

International Adoption

Subsequently, this system was adopted by the European subsidiaries who were having the same issues with the US warehouse-centric system.

Integration in Service Call Management

Subsequently, this system was integrated into a Maintenance Call Centre system for dispatching engineers to service calls, recording of travel, time and parts used, which all fed into the service costs analysis system.

Interpretive Dynamic Graphic Displays (1980)

This is a follow-on post to "Sewage Works Data Acquisition and Process Control".

Graphics Displays "Tailorable" by End Users?

Yes, that was in the "Requirements". When I arrived on the project, a couple of engineers were looking at how they could design a "paint" program! That seemed like total over-kill for a piece of functionality that would be very rarely used. Sometimes we must be "creative" in interpretation of wording of "requirements".

The Graphics Hardware and Drawing Language

Fortunately, the engineering company had selected and purchased a sophisticated, programmable graphics display terminal (RAMTEK). It had an inbuilt interpreter for a drawing language. The drawing instructions were fed into the terminal as an ASCII data stream.  For example:-

        [COLOR RED][RECT <x-value>,<y-value>,<width value>,<height value>]

All <values> were in pixels. (Forgive me RAMTEK if I haven't remembered your syntax correctly). So in fact it was quite reasonable to draw up the required layout on graph paper and code the required drawing sequence. In today's parlance, we could call it a "mark-up language".

Language Extensions to make it Dynamic

But the graphics we required had to display dynamic data in "real-time". There would be meter readings, red/yellow/green signals, switches/stop-valves that indicated "open/closed", arrows with changeable direction, etc. - think "Dashboard".

My solution was to extend the RAMTEK language with three simple constructs.

• Firstly, values could be replaced by named variables that referred to values in the in-memory data streams. There are two types of variable, booleans (ON/OFF) and integers, represented as "Bnnn" and "Innn".
• Secondly, labels could be assigned to any statement - eg. Lnnn:.
• Thirdly, a simple variable test construct was defined with simple boolean and logical operators, that if tests "true" directs "execution" to a specified label (eg. [IF B123 ON L456].

In today's parlance, this is known as "templating" (eg. PHP). The implementation was our own interpreter program that processed a template file, substituting variables with their real-time values, performing the testing and branching, continuously outputting a stream of drawing commands to the terminal. The end of a template file simply looped back to the start. So the "real-time" screen refresh was determined by how quickly this interpretation loop executed.

Was It "Tailorable" by End Users?

The "template" file (program?) was a simple text file of reasonably readable instructions. Sure, it was tedious to update but essentially clerical, so we argued that it was "tailorable".

The Actual Display Graphics

There were three different displays delivered with the system.

• The hydrolic flows through the works (the first stage of the plant is to filter and separate the liquid from the solid 'sludge').

• The "sludge" system monitoring. (The sludge is pumped through large anaerobic digester tanks, the active bacteria in which are highly sensitive to temperature and acidity, so monitoring of temperatures and pump flows is critical in such a plant).

(apologies for the hand-drawn schematic - that's all I have left from the project)

• The electrical generation system. The above anaerobic digesters produce methane which is drawn off and feeds electricity generators which power the plant as well as sending excess power back into the city power grid. Its graphic display showed generators' status, voltages, currents, circuit-breaker statuses, etc.

Sewage Works Data Acquisition and Process Control (1980)

"Real Time" or "Data Processing"?

The sewage works of this large New Zealand city was in need of major redevelopment and modernization.

A civil engineering company was the prime contractor, and an electrical automation engineering company was sub-contracted to build the plant control systems. They had experience in micro-processor controllers for basic automation, but they needed someone to take on the back-end computing requirements of control-room data presentation and all levels of reporting.

When I arrived, their engineers were looking at (small) technical issues around the periphery of the requirements. The first thing I did was to review the requirements holistically and perform a chronological functional decomposition and came up with the following Data Flow Diagram.

(apologies for the hand-drawn diagram - that's all I have left from the project)

The engineering management were stunned when I showed them that less than 20% of the system was "real time" and the majority was "data processing" not unlike many commercial systems.

The Central Data Flow

The raw input data from the plant comprised 700 analog measurements (flow rates, tank levels, temperatures, voltage and current), and 150 digital (binary) "alarm" signals. All these inputs were sampled 4 times/second. A simple data communications protocol was designed to interface with the micro-processor controlled electrical connection panels.

Current (Real Time) Data

The "current" data status of the plant was required to be available for display on graphic displays in the control room and selected "alarm" status changes needed to be logged on a printer. The very nature of the type of plant being monitored and the data processing responsiveness of the graphics and printing showed that a 2 second response time was quite adequate, so the design was to average input data in blocks of 8 values to give 2 second averages.

But this data rate was still too fast to write to disk and the "real time" graphic display would not be responsive enough if it had to continually retrieve data from disk. The solution was to use a "mapped, shared memory" data pool. The data was also accumulated in buffers for writing to disk files at 30 second intervals.

Data Accumulation

A series of batch jobs then ran hourly, 8 hourly (per shift), daily and monthly, to summarize results from the lower level and write into the accumulation file at the next level.

Flexible Data Management - self-describing meta-data

A core feature of the system requirements, was for maximum flexibility in describing data, both the real-time inputs and the as yet undetermined manual inputs (typically the results of chemical analyses). Similarly, reports' layouts and contents required maximum flexibility of design and changes. A highly table-driven system was designed.  

But all these tables would need to be maintained (and possibly additional tables added later), so a table-driven table maintenance method was designed with its own table of meta-data that described the contents of the other tables to be maintained. The obvious (?) approach was to have it "self-describing" - the first entries in the meta-data table were manually entered to describe the meta-data table itself. The data maintenance program was then used to populate the remainder of the meta-data table, which could then be used to populate the system data tables.

Files were defined as a collection of records of a given type, and records were defined as a set of data items of various types.  Data items were defined with a long and short name, a data type and some very simple range constraints.

Process Control

A generic process control module was developed, with control tables associating outputs with the feedback inputs along with a set of process control parameters (2nd order function).

Generic Report Definitions

Shift and Daily reports were quite simple lists. Report definition tables specified which variables were to be reported.

The Monthly reports were all data analysis type reports. A given, fixed set of various sorts of analysis algorithms were required (various sorts of means, statistical measures, etc). Again, report definition tables specified which algorithm was to be applied to which set of variables for each report.

Summary

In all, a very satisfactory result was achieved. I was assisted by two new university graduates on a vacation contract, to complete the system in 6 months.