And so dear reader, once again, I warn you to read on at your own risk.

It would henceforth be known as the night of the 96 abends.

Like most system implementations, the 2-month system test had gone smoothly.  So smoothly in fact that it seemed a portent of a series of unfortunate abends.

Implementation weekend started early Sunday evening or late Sunday afternoon.  It really didn’t matter except that like truly great tragedies, we must record this story as having started just as the day turned to night.

The entire team reported for work to validate all the modules for implementation.  As expected, everything checked out – copybooks, program loadlibs, control cards, procs, and jcls were all in order.  It was all systems go.  It was time to get a good night’s sleep as a reward for another successful project implementation.

Mr. Programmer had his Monday all planned - to the office early to work on his other long-pending tasks and then on to a reunion dinner with former colleagues in Greenbelt 3.  It would be a good time to catch up and reconnect with old friends.

But such is Mr. Programmer’s fate that even in his sleep, there were things happening that would alter his plans for that Monday.  His Monday would be, as that soppy song would say, “bluer than blue”…

Things started happening at dawn of Monday.  As the production batch cycle started, the technical architect based in the US noticed something missing.  The scrub job that was supposed to run before the batch cycle did not run.  This one-time job’s function was to initialize all the numeric fields that had been newly-created as part of the system implementation.

It was time to wake up the programmers.

Programmer A was the first to respond to the call of duty.  He woke up the team lead and also woke up Mr. Programmer.  So it was that Mr. Programmer was witness to the horrifying events that were to come.

And here I must insist that children reading this article drop off and watch cartoons on TV or something.  There are still some things that are better left to adults.  Such as attending meetings and at the same time trying hard to resolve abends and at the same time trying hard not to point fingers and at the same time trying as hard as possible to cover your own ass as discreetly as possible.  These things only adults would understand.

As in all major calamities throughout history, action follows meeting.  The meeting precedes all and from the meeting flow the wisdom of a response.

So it was on this Monday.  Programmer A had called up the operations people to put some jobs on hold because they would surely abend.  The operations guys would do no such thing.  A meeting was needed you see.

And so from early Monday morning till noon, one manager of something to another manager of something was added to the call till everyone lost count of the number of managers of something on the line.

First item on the agenda was to find out why the scrub job did not run.  A little over an hour was all it took for an enterprising operations guy to establish that the programmers had insufficient instructions on page 252 of the implementation document.  The programmers had put in there to schedule the scrub job before the normal production batch cycle.  But nowhere was it written to run the job.  The scrub job was therefore scheduled but not run.

Guilt established the next item was an action plan.  Programmer A once again brought up the request to hold some jobs from running.  The enlightened response: the request had to go through the normal channels.  Documentation had to be prepared.  A change request had to be filled up and approved by all the involved managers.

With all the talking involved, noon brought a new development.  Jobs had started abending with SOC7 or data exception.  The final tally: 96 abended jobs.  There was no more need to hold jobs from running.  They had all ended abnormally.

There was only one avenue left open – restore files, run the scrub job and rerun the abended jobs.  All the managers of something agreed that this be left to the programmers.

So it was that Programmer A, Mr. Programmer and their colleagues Programmer L and Programmer B, started the painstaking work of restoring files and determining the appropriate restart procedure for each abended job.  It took them up to the early morning of the next day, Tuesday, to have everything up and running.

All throughout, one manager of something after one manager of something was on the phone or at their side or in their e-mail asking for a status.  After the meeting, status was the operative word.

At last it was time for the tired programmers to go home and get some sleep for another day of coding while the managers of whatever congratulated each other for another well-managed crisis.  If it were not for these managers’ timely intervention and constant prodding, the problems would never have been resolved on time.  No matter that the actual work was done by the programmers.  It was the managers’ managerial acumen that saved the day.

So how many people does it take to resolve 96 abends?  Answer: A technical architect to sound the alarm, 4 programmers to do the dirty work, and a dozen or so managers to take status reports every 30 minutes.

Postscript: after a top-level managerial review of the implementation problems, it was decided that the programmers take a refresher course on how to properly fill up the implementation documentation.  So there!

Birthday greetings go to Maria Milagros “Aggie” Luque (November 6).  Aggie is a mainframe programmer who has one of the kindest hearts I have been lucky to meet and work with.  To those interested, she is still single, is currently based in Singapore, and is also interested. J  Happy Birthday Aggie and I hope you meet your Mr. Right soon! Comments are welcome at itdinosaur@gmail.com.