Who's the Client? What's the Deliverable? (1995-7)

Introduction

I hope you will understand my discretion in suppressing certain details and names of companies.  The project was a massive, city-wide, integrated, distributed system with a high daily transaction rate and cash flow, developed for and to be run for the state government.

The contract was won by a Joint Venture Consortium formed between a hardware supplier, a multi-national computer supplier (my employer) supplying the central computers and central management and reporting software, and a cash-services company.

The Warning Signs

We Software Engineers pressed on in typical bespoke software development fashion, collecting and documenting requirements, developing functional design specifications for review and sign-off by the client (end-user SMEs). But obtaining "sign-off" was like trying to extract "hen's teeth" - we put it down to typical public-servants' reluctance to put their name "on the dotted line" and take responsibility for their decision.

Next was the continuous pressure for requirements "creep". Whilst both major suppliers had formal documentation, quality and change-control processes, they were different and weren't coordinated - one came from the engineering discipline and the other from commercial software development backgrounds.  What was seen as acceptable to one supplier was seen as a Change Request to the other.

As the pressure of time and budget increased, the above scope-creep issues had a strange effect on the over-all programme management style. I call it "pendulum management". In alternate months, the key management message alternated from "tighten up, work to budget and time-line", then "keep the customer happy, give him what ever he wants".

The "aha" moment came when we came to specify the functionality for managing the discrepancies between sales transaction data and cash collected. We went to the client users for their "requirements" and, to our surprise", were told, "Its not our problem. Its the consortium's problem.  We simply require that you pay us the higher of the cash collected (assuming sales transactions were lost), or the sale transactions amount (assuming cash has been lost)" - it was a classic "heads they win, tails we lose" situation.

Who's the Client?

This was the point when the "client dilemma" really struck home (to we software engineers at least). The Consortium's contract was in fact a 10 year "Service Contract", to first of all to build the system for an up-front payment (with ownership retained by the consortium), and then to operate the system including provision of enquiry and reporting services to the public servants. The "requirements" of the system being built were to "provide the contracted service".

Our "cash-to-sales reconciliation" problem (above) needed to be resolved by the consortium's operational and accounting staff, who had as yet not been appointed. In fact, at this point, the consortium's operating company comprised a single project manager! The two major partners in the consortium had forged ahead almost independently with no thought or plans for how the ongoing service would be provided and any of its impacts on the requirements of the systems being built.

Our Cash-to-Sales Reconciliation Solution

A major source of issues in cash-to-sales reconciliation, was the distributed nature of the POS equipment and in about half the cases, manual transfer of transaction data (by key-drive type device) to interface into the central system. Transaction batch identification had been included and a certain degree of redundancy. What was agreed as being required, was initial reconciliation when collected cash was counted per identified transaction batch (or group of data batches). Data completeness integrity controls would be needed so that an alert could be flagged indicating that it was known that some data was missing. Then when/if the missing data did arrive (possibly via error correction) an adjustment could be raised against the matching reconciliation and a CR/DR raised.

Needless to say, such a level of audit control had not been anticipated nor built by the hardware supplier and some robust negotiation around changes to the core data interfaces was required. To be fair to the hardware supplier, they had an enormous micro-software change control problem in coordinated distribution of updates across hundreds of devices, not to mention the data storage chips attached to every cash container.

(Technical aside: The core issue revolved around a forward singly-linked list being less robust that a bi-directional doubly linked list).

The End-Game

The total system finally went into production, grossly late and over budget (there were political imperatives (read election) that the system must not "fail"). The "service" has run well past the original 10 years, since the replacement system has had numerous problems of its own.

There was "robust negotiation" (litigation?) between the government and the consortium over the cost of the over-run and scope-creep (with eventual confidential settlement). I was involved in extensive "function point analysis" of the original (very vague) contractual (service) requirements and the system "as built" in order to identify and quantify the scope increase. But even this exercise was predicated on the bespoke software development model.

Client-Server By Mag-tape Exchange (1983)

I had been hired by this Australian national office of a multi-national mini-computer manufacturer, to setup the internal data processing systems.  

Spare-Parts Inventory Tracking

A major issue was their spare-parts management which was getting "slammed" by the auditors for enormous write-offs/adjustments after every annual stock-take.

The spare-parts system was a national, warehouse-centric system using the US head-office software. All transactions came in on "movement tickets" for keying. These covered engineer "consumption" in repair of client machines, inter-office transfers (of both good and faulty parts) and repairs (faulty in, good repaired out). The potential for transcription and keying errors was high. There was the issue of "handedness" where left-hand and right-hand variants had similar part numbers. But from a financial point-of-view, variants of circuit boards was more critical - a more expensive board with more memory or a faster processor might vary in part number only by a suffix.

Customer "Loans" Issue

But it was "home-brew" software by one branch office for tracking their own spare parts use, that finally brought home the real culprit. On service calls, engineers took out some good parts in anticipation of what might be wrong, then on site replaced the faulty customer part with a good part. If they were identical part numbers then all good. But if an identical good part was not available, the engineer would put in a good part of a higher rating (eg. more memory), so the faulty part brought in (for repair) had a different part number. In theory, when the customer's part was repaired, it was supposed to be taken back, installed and the original good part brought back. This "home-brew" local system kept track of these "loans" and reminded the engineers. A similar issue arose with sales reps. "lending" customers a higher rated board to trial before buying. Again, the national system did not cater for this concept of "loaning" parts to customers.

On-line Data Entry Client with Transaction Logging

My solution had a couple of phases. Firstly, the new system would be multi-user on-line data entry at each point where the original movement tickets would have been written. On-line data entry then provided immediate data validation of part numbers. A new transaction type of "Loan" was then introduced - this was expanded from the loan to customer", to recording of ALL good parts that a service engineer took on his service call and recording everything he brought back, good, faulty or alternate part number. Initially, inter-office movements still needed the manually prepared movement-ticket from the non-automated office.

The second phase was implementation of this system in all the branch offices. This is especially important in a country as geographically dispersed as Australia. In the states of Queensland and Western Australia, engineers go out on week-long circuits of preventive maintenance of remote sites and so have to take a large stock of spare parts with them.

Transaction Server

Both phases had issues that led to a single solution. Real-time data entry into a system that did not have a robust database with transaction level data integrity, had potential of data loss with no fall-back paper work for data re-entry. At this time, the INTERNET was not yet available, and leased-line networking to all the branch offices was not economical.

My solution was a custom developed transaction processing system, which we would now call "client-server". Multiple data-entry front-ends accessed the database read-only for data validation, and compiled a transaction record that was sent (by inter-process communication) to a single threaded database updater process. The first step of the updater was to write the transaction records to a serial transaction log. These transaction logs were backed-up daily. In the branch offices, these transaction log files were written to mag tape and sent to head-office along with the regular parts-for-repair transfer. At head office, the transaction log files for all branches were then read (off tape) and fed into the same head-office database updater process, so that a National database was now available. For the first time, the national spare-parts planner had a reasonably up-to-date picture of the distribution of parts across all offices. At head-office, parts receipting now simply checked off the electronic parts transfer "ticket" received from the sending office. New weekly reports to each branch manager listed all parts currently out on "loan" that had to be retrieved or swapped-back.

The test of this system of processing transaction logs files, came when the National system first went live. The Melbourne (National) office had been using what would become the Branch system. When the National database was initialized with the backup after the last stock-take, we simply processed six-months worth of transaction logs to bring the National system up-to-date, without a hiccup.

International Adoption

Subsequently, this system was adopted by the European subsidiaries who were having the same issues with the US warehouse-centric system.

Integration in Service Call Management

Subsequently, this system was integrated into a Maintenance Call Centre system for dispatching engineers to service calls, recording of travel, time and parts used, which all fed into the service costs analysis system.

Fingers in the Till - Forensic Initiation (c.1986)

I was doing a little bit of maintenance work on the software we had built for a large cinema chain (support of new ticket printing hardware). Their IT manager called me aside confidentially (oops, what have I done?).  But he explained that management thought they had a problem with cash going missing from the till of the concession (snack bar). Would I be able to analyse their cash register data log to report shortages/surpluses per shift per user with long-term averages?  

Fortunately, the cash/sales reconciliation software had been well designed and recorded everything needed, so without too much effort we could quickly see that one particular operator quite consistently recorded a cash shortage, whereas the average of all other operators was essentially a zero balance. With a Statutory Declaration attached to my signed report, I heard nothing more other than that the subject staff-person no longer worked for them.

It was quite a simple job but was my first taste of "forensic" data analysis and reinforced the importance of not just updating a database, but when it comes to money, the importance of also recording who, what and when.