Read Everything!

Some two decades ago, I didn't read the cover letter for a HIPER PTF, so I was unaware of its special instructions, and almost a year after it was applied, my negligence caused a catastrophic failure in one system. Those unread directives compelled the reader to rebuild the storage directories, but since that wasn't done, an upgrade of the operating system caused every DASD page to be marked as free, and over a period of several hours, the AS/400 slowly ate its storage, one page at a time, damaging many objects before we were able to wrest control and recover.

Image of the hung GUI that I uploaded with the PMR.

Besides the refrain to backup, backup backup, the second most important task in computers is to read everything before-hand. This week I neglected to peruse the read-me page for the HMC PTF MH01560, mostly because, frankly, there is nothing more boring than an HMC read-me document. In all the years I've been installing them, there has never once been anything in their texts that would have caused me to deviate from the normal installation steps. However, in this read-me's "Installation" section there was one tiny sentence that read "This fix must be installed using the HMC updhmc command; not the GUI." It doesn't relate why one must do that, just commandment to not.

Though this bit of negligence was not as bad as my catastrophic faux-pas of the 90s, when the GUI got hung, I opened a PMR and was politely informed by the support tech that, ahem, right here, sir, it say do not use the GUI. I have since closed the PMR and asked them to please burn it, such was the level of my embarrassment. If there's one positive to this experience, it turns out the CLI application is very fast and easy to use, and I'll probably use it over the GUI in the future.

This article was supposed to be a heads up to those upgrading to v8r8.4.0 and applying required PTF MH01560 with the updhmc command, but it devolved into a confession of inadvertently providing support techs with material to laugh about during their coffee breaks. In any case, I hope you'll all have a safe holiday season, and maybe someday I'll write about the two PMRs I opened a year apart for the same issue that were both PBKC. Though both times, thankfully, it was someone else's KC.

CL Program to Apply Technology Refresh PTF

I had a deep sense of foreboding when I found that a HIPER PTF I downloaded came with technology refresh PTF 5770999-MF99002. These things are a pain in the butt to apply as it requires one to do the following steps: 

1. Perform a full system save, like GO SAVE, option 21.
2. Apply all microcode PTFs permanently.
3. Load the TR PTF and set it to be applied temporarily.
4. IPL the system.
5. Apply the TR PTF permanently.

On our system there were two pre-requisite PTFs that were downloaded with MF99002; plus we already had a number of microcode PTFs that were set to be applied during the next IPL. I didn't want to permanently apply these new microcode PTFs, so I took the following steps:

1. Unloaded the microcode PTFs set for delayed apply, except for those that were pre-requisites to MF99002.
2. Performed a full system save.
3. IPLed the system to apply the pre-requiste microcode PTFs.
4. Permanently applied all microcode PTFs.
5. Loaded MF99002 and set it to delayed apply.
6. IPLed the system.
7. Permanently applied MF99002.
8. Loaded the rest of the microcode PTFs and set them to delayed apply.
9. IPLed the system.

Our weekly "full system save and IPL" is full automated, so I needed some way to insert a CL program into this mix that would do the job. After finishing it I realized that others might be able to use it as well. It was designed to run in the QSTRUPPGM program, before it actually starts anything. It's assumed that the very first IPL, after the full system save, would first apply any PTFs that were pre-requisites of the TR PTF. All pending microcode PTFs to be applied, including the TR, would remain in a "Save file only" un-loaded state.

After that first IPL it would find that the TR PTF wasn't loaded yet, so it would apply all microcode PTFs permanently, load the TR PTF, set it to delayed apply, and IPL. When the system came up again it would find that the TR PTF was applied. It would then permanently apply it, load the rest of the microcode PTFs sitting in *SERVICE, set them to be applied delayed, and IPL again.

I've included the code below:

---

/******************************************************************************/
/* Program....: APYTRPTF                                                      */
/* Description: Apply technology refresh PTF.                                 */
/* Author.....: Joe Code                                                      */
/* Date.......: 02/18/2011                                                    */
/*                                                                            */
/* Create                                                                     */
/* ------                                                                     */
/* CRTCLMOD DBGVIEW(*ALL)                                                     */
/* CRTPGM USRPRF(*OWNER) AUT(*EXCLUDE)                                        */
/* CHGOBJOWN OBJTYPE(*PGM) NEWOWN(QSECOFR)                                    */
/* GRTOBJAUT OBJTYPE(*PGM) USER(QPGMR) AUT(*USE)                              */
/*                                                                            */
/* License                                                                    */
/* -------                                                                    */
/* This library is free software; you can redistribute it and/or modify it    */
/* under the terms of the GNU Lesser General Public License as published by   */
/* the Free Software Foundation; either version 2.1 of the License, or (at    */
/* your option) any later version. (Unlike the normal GNU GPL, the "lesser"   */
/* GPL allows libraries to be used in commercial/proprietary software.)       */
/*                                                                            */
/* This library is distributed in the hope that it will be useful, but        */
/* WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANT-       */
/* ABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General    */
/* Public License for more details.                                           */
/*                                                                            */
/* You should have received a copy of the GNU Lesser General Public License   */
/* along with this library; if not, write to the Free Software Foundation,    */
/* Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA           */
/* (http://www.gnu.org/licenses/lgpl.html#SEC3)                               */
/******************************************************************************/

PGM

   COPYRIGHT TEXT('Copyright (c) 2011 Joe Code')

   DCL VAR(&COUNT)     TYPE(*DEC)  LEN(3 0)
   DCL VAR(&ERROR)     TYPE(*LGL)          VALUE('0')
   DCL VAR(&RCVVARLEN) TYPE(*INT)  LEN(4)  VALUE(131)

   DCL VAR(&PTFINFO)   TYPE(*CHAR) LEN(50)
   DCL VAR(&PTFID)     TYPE(*CHAR) LEN(7)  STG(*DEFINED) DEFVAR(&PTFINFO  1)
   DCL VAR(&PRODID)    TYPE(*CHAR) LEN(7)  STG(*DEFINED) DEFVAR(&PTFINFO  8)
   DCL VAR(&RLSLVL)    TYPE(*CHAR) LEN(6)  STG(*DEFINED) DEFVAR(&PTFINFO 15)
   DCL VAR(&CCSID)     TYPE(*INT)  LEN(4)  STG(*DEFINED) DEFVAR(&PTFINFO 21)
   DCL VAR(&CLSPTFF)   TYPE(*CHAR) LEN(1)  STG(*DEFINED) DEFVAR(&PTFINFO 25)
   DCL VAR(&RESERVED)  TYPE(*CHAR) LEN(25) STG(*DEFINED) DEFVAR(&PTFINFO 26)

   DCL VAR(&RCVVAR)    TYPE(*CHAR) LEN(131)
   DCL VAR(&BYTESRTN)  TYPE(*INT)  LEN(4)  STG(*DEFINED) DEFVAR(&RCVVAR   1)
   DCL VAR(&BYTESAVL)  TYPE(*INT)  LEN(4)  STG(*DEFINED) DEFVAR(&RCVVAR   5)
   DCL VAR(&LOADEDSTS) TYPE(*CHAR) LEN(1)  STG(*DEFINED) DEFVAR(&RCVVAR  41)
   DCL VAR(&IPLACTION) TYPE(*CHAR) LEN(1)  STG(*DEFINED) DEFVAR(&RCVVAR  66)

   MONMSG MSGID(CPF0000 CEE0000 MCH0000) EXEC(GOTO CMDLBL(ERROR))

   QSYS/CHKOBJ OBJ(QGPL/APYTRPTF) OBJTYPE(*DTAARA)
   MONMSG MSGID(CPF9801) EXEC(QSYS/CRTDTAARA DTAARA(QGPL/APYTRPTF) +
      TYPE(*DEC) LEN(3 0) VALUE(0) TEXT('Stop infinite loop if error +
      applying TR PTF.'))

   QSYS/RTVDTAARA DTAARA(QGPL/APYTRPTF) RTNVAR(&COUNT)
   CHGVAR VAR(&COUNT) VALUE(&COUNT + 1)
   QSYS/CHGDTAARA DTAARA(QGPL/APYTRPTF) VALUE(&COUNT)

   CHGVAR VAR(&PTFID)    VALUE('MF99002')
   CHGVAR VAR(&PRODID)   VALUE('5770999')
   CHGVAR VAR(&RLSLVL)   VALUE('V7R1M0')
   CHGVAR VAR(&CCSID)    VALUE(0)   /* Use job's CCSID. */
   CHGVAR VAR(&CLSPTFF)  VALUE('0') /* Close PTF files afterwards. */
   CHGVAR VAR(&RESERVED) VALUE(' ')

   QSYS/CALL PGM(QSYS/QPZRTVFX) PARM(&RCVVAR &RCVVARLEN &PTFINFO +
      'PTFR0100' X'00000000')

   SELECT

      WHEN COND(&BYTESRTN < &RCVVARLEN *OR &BYTESAVL < &RCVVARLEN) +
         THEN(QSYS/SNDPGMMSG MSGID(CPF9898) MSGF(QCPFMSG) MSGDTA('No data +
         returned') TOPGMQ(*PRV (*PGMBDY)) MSGTYPE(*ESCAPE))

      WHEN COND(&LOADEDSTS = '0') THEN(DO) /* Not loaded? */
         IF COND(&COUNT = 1) THEN(DO)
            QSYS/APYPTF LICPGM(5770999) APY(*PERM)
            MONMSG MSGID(CPF3660) /* No PTFs identified. */
            QSYS/LODPTF LICPGM(5770999) SELECT(MF99002)
            QSYS/APYPTF LICPGM(5770999) SELECT(MF99002) DELAYED(*YES)
            QSYS/PWRDWNSYS OPTION(*IMMED) RESTART(*YES) IPLSRC(B)
         ENDDO
      ENDDO

      WHEN COND(&LOADEDSTS = '1') THEN(DO) /* Loaded, but not applied? */
         IF COND(&COUNT = 1) THEN(DO)
            QSYS/APYPTF LICPGM(5770999) APY(*PERM)
            MONMSG MSGID(CPF3660) /* No PTFs identified. */
            IF COND(&IPLACTION = '0') THEN(QSYS/APYPTF LICPGM(5770999) +
               SELECT(MF99002) DELAYED(*YES)) /* IPL action isn't set? +
               Then set it. */
            QSYS/PWRDWNSYS OPTION(*IMMED) RESTART(*YES) IPLSRC(B)
         ENDDO
      ENDDO

      WHEN COND(&LOADEDSTS = '2') THEN(DO) /* Temporarily applied? */
         IF COND(&COUNT = 2) THEN(DO)
            QSYS/APYPTF LICPGM(5770999) SELECT(MF99002) APY(*PERM)
            QSYS/LODPTF LICPGM(5770999)
            MONMSG MSGID(CPF35A8) /* No PTFs to be loaded. */
            QSYS/APYPTF LICPGM(*ALL) DELAYED(*YES)
            MONMSG MSGID(CPF3660) /* No PTFs identified. */
            QSYS/PWRDWNSYS OPTION(*IMMED) RESTART(*YES) IPLSRC(B)
         ENDDO
      ENDDO

   ENDSELECT

   RETURN

/******************************************************************************/
       /*                                                                     */
ERROR: /* Global error handling section.                                      */
       /*                                                                     */
/******************************************************************************/

   IF COND(&ERROR) THEN(RETURN)  /* Prevents an infinite loop if an error     */
   CHGVAR VAR(&ERROR) VALUE('1') /* occurs within the error handling section. */

   QSYS/SNDPGMMSG MSGID(CPF9898) MSGF(QCPFMSG) MSGDTA('Unexpected error +
      occurred. See previous messages') TOPGMQ(*PRV (*PGMBDY)) MSGTYPE(*ESCAPE)

ENDPGM

---

On our small system this program added an extra two hours to the normal weekend backup/IPL. The only problem I ran into was that the microcode PTFs that I removed weren't loaded again. Apparently when you permanently remove a microcode PTF, it's also removed from the list of "*ALL PTFs in *SERVICE", meaning they can't be loaded when you specify LODPTF DEV(*SERVICE) SELECT(*ALL). You have to specifically name them in the SELECT parameter. Next week when I do this again I am going to load all of the microcode PTFs and leave the delayed apply flags shut off. (Except of course for the TR PTF's pre-requisites.)

Looking at the program again this morning I realized that if there was a subtle problem with applying the TR PTF then it might cause an infinite loop of IPLing, which would be very bad. I've since changed the program to include a safety valve in the form of a data area used to count the number of IPLs. If the count doesn't match up with what is supposed to happen then it doesn't do anymore IPLs. This amended version of the program compiles, but I haven't tested it yet. If you want to wait, it will be tested again next weekend when I apply the TR PTF to another system in our network. I will update this post with the results.

Update (2011-02-28): This version of the program did it's job and applied the TR PTF without error.

Update (2011-06-07): IBM has since issued PTFs to make this task easier, so it would seem that the program above is obsolete.

Enterprise Extender PTFs Again

Once again the list of v5.4 Enterprise Extender recommended PTFs has left off some important patches. MF47274 & MF47275 can cause HPR-IP controllers to remain in an unusable VARY ON PENDING state, and varying them off and on again doesn't fix it.

Our lord Ib'm should have added MF47838 & MF47839 to the list. (Note that MF47838 supersedes MF47275). However there is still a problem because sometimes IPLing or restarting the system from a restricted state causes the controllers to again remain in a VARY ON PENDING state. Fortunately, this time it can be fixed by varying them off and then on again. Not a show-stopper, just an annoyance.

Ib'm support tells me there are two test PTFs that will resolve that problem too and they're in the process of granting me access to them. My advice to the users of HPR-IP controllers is to ignore MF47274, MF47838, and MF47839 until these test PTFs become available.

(Please see the comments for updates.)

New PTFs to be Tested on DEV: 08/22/09

PTFs were loaded for the following reasons:

• Recomendations:
  • High Impact Pervasive (HIPER).
• Group PTFs:
  • SF99539 (Level 108)^*: GROUP HIPER

The following PTFs were loaded for the above reasons:

5722SS1 (i5/OS):

• SI35859:
  • SE38501: OSP-DB-OTHER-RC4-MSGSQ30082 SQ30082 RC4 IS RETURNED ON DRDA
  • SE38440: OSP-DB UNNAMED COLUMN DRDA PROBLEM

---

^* -Loading this group did not result in any new PTFs being installed (they were already applied individually to our systems on previous IPLs), so only the level number was changed.

HTTP Group PTF SF99114 level 19 is Released on the Heels of Level 18.

Group PTF levels don't come out all that often. The HIPER group is every two weeks, but the other groups are every couple of months at least. Whenever a new version of a group PTF comes out on the heals of an earlier version it usually means that the high priests in Mecca (aka Rochester, MN) are correcting some kind of unforeseen problem. Sometimes they add a new PTF but more often than not they remove some PTF's. Always without explanation.

Last Friday our Lord Ib'm came out with level 18 of the V5R4 HTTP group PTF SF99114 and yesterday they issued level 19. In so doing they removed two PTF's from the group: 5722DG1-SI35431 and SI35432. The PTFs haven't been PE'ed, but I'm not sure whether or not I'm going to install them. On the one hand these two PTFs update the Apache server to the latest level (2.0.63) and fix a security problem (CVE-2008-2939). On the other hand it might cause other problems that our Lord Ib'm has seen fit not to reveal to us.

Recommended PTFs for Enterprise Externder is Incomplete.

The list of V5R4 recommended PTFs for Enterprise Extender (EE) is incomplete. I had to open a PMR with the minions of our Lord Ib'm because the old testament SNA communiation controllers on our development coding system failed to connect to the production systems, so we were unable to use SAVRSTOBJ.

The support priest sent me the complete list of recommended PTFs for EE and we were missing three PTFs. I have since confirmed that those were probably the three that we needed because after applying them the systems are again able to communicate using the old protocol.

When I checked I found that the EE list noted above wasn't up to date. I've sent feedback to the keepers of the web site, but as a courtesy I thought I'd post the complete list here as well:

PTF	Release	Cumulative Tape	Modules
MF40235	540	7107	*HPRIP ctlr with exchange ID defined will not vary on and SDLC XID has problem if HPR=*YES.
MF43629	540	8305	*HPRIP ctlr with exchange ID defined will not vary on.
MF42291	540	8183	DLUR and HPRIP controllers go vary on pending after Activate instead of VARIED ON, CPA58D1. MSCP changed to allow DLUR to recover on VTAM deactivate or activate.
MF42499	540	8057	NETWORK ID ALWHPTWR and HPR are *YES, ICF applications fail w/CPF5535, jobs remain in a UNKNOWN state. Sequence of BAD XID network frames case EE to be improperly initialized with invalid controller's values.
MF42587	540	8057	EE HPRIP controller remains in VaryOnPending status with no XID's being sent out.
MF45692	540	1000	EE using HPR and RTP receives timeout error. application joblogs show CPF5355. APPN&SrcSink shows RTPburst timer did not expire AND EE/HPR CONNECTION STALLS AND NO DATA IS SENT
MF44200	540	8305	EE endnode unable to locate a CICS region on ZOS LPAR
MF43891	540	8305	LDLC remains in a reset status, LatePrenegotiation XID does not send a NULL XID on its own.
MF45851	540	1000	HPR config is active, but active connections lock up, When you vary off APPC *HPRIP controllers and vl 07000C1F occurs, the process to activate PTF. An IPL will be needed to reset error condition. AND EE/HPR stops sending data when incoming stream data packet missing
MF46217	540	1000	EE task may block streams on gate before error is sent back
MF46237	540	1000	Vary On HPRIP CTL fails after IPL, XID does not complete
MF44897	545	8305	DLUR and HPRIP controllers go vary on pending after Activate instead of VARIED ON, CPA58D1. MSCP changed to allow DLUR to recover on VTAM deactivate or activate.
MF45170	545	9104	NETWORK ID ALWHPTWR and HPR are *YES, ICF applications fail w/CPF5535, jobs remain in a UNKNOWN state. Sequence of BAD XID network frames case EE to be improperly initialized with invalid controller's values.
MF44262	545	8305	*HPRIP ctlr with exchange ID defined will not vary on.
MF45881	545	9104	HEA fc181X and Virtual Ethernet fc268C lossing data in UDP layer
MF46122	545	8305	NETWORK ID ALWHPTWR and HPR are *YES, ICF applications fail w/ CPF5535 and will now accept retransmitted frames in BIND.
MF43329	545	8183	Partition/system crash with srcB6005121 on vary of EE controller
MF44072	545	8183	EE HPRIP controller remains in VaryOnPending status with no XID's being sent out
	545	do not order - test status	MF44344 - EE/HPR ICF application fails. CPF5107 E015 unbind received RTP did not check for this error
MF46366	545	1000	Vary On HPRIP CTL fails after IPL, XID does not complete
MF46218	545	1000	EE task may block streams on gate before error is sent back
MF46063	545	1000	APPC using HPR/EE gets RTP timeout
MF46064	545	1000	EE endnode unable to locate a CICS region on ZOS LPAR

IBM PE's seven PTFs for V5R4.

While my pastoral duties at this community are normally rather dull, my favorite activity is working with the most holy code (i5/OS) and PTFs (Parchment Temporary Fixes). I especially love it when I find an error in the code that causes the cardinals and bishops in Rochester to do something drastic.

A couple months ago, as part of a HIPER recommendation, we installed PTF 5722SS1-SI34484. The following Monday morning the BACKUP job was looping forever waiting for an HTTP server job to end. We didn't associate it with the PTF right away because this isn't the first time the backup got held up by something like this. By the time we'd figured out that this was a trend we'd already installed another HIPER recommendation, PTF 5722SS1-SI34576, which superceded the one installed previously.

So we were now stuck in an untenable situation. We couldn't remove the PTF and we didn't want to reload the O/S. Being in this kind of pickle can really focus the mind. I ended up writing a program that detected these zombie HTTP server jobs and killed them with ENDJOBABN.

The minions of our Lord Ib'm eventually discovered that a thread in the HTTP job was prematurely destroying the list of open files. The job would then sit there forever waiting of the files to close not knowing that that list was gone. In the end they had to PE a bunch of PTF's and since two of them were HIPER recommendations the fixing PTF (SI35317) was also made a HIPER.

Final tally was seven PE'ed PTFs and one HIPER, all from one PMR.

Apache update v2.0.52 has problems.

On our web server V5R3 PTF 5722DG1-SI17010 causes HTTP/500 server failures. It might be related to SSL because our non-SSL web servers weren't affected. We're currently in touch with the almighty on this problem. Originally support thought it was related to another problem but the messages in the error log were different.

Ib'm requested and we collected an HTTP trace with the PTF applied and it's been zipped up and emailed to support. I'll add comments to this article when we know the outcome.

The Holy Day of Sunday.

Ah Sunday. A most holy day. Every week on this day, early in the morning, we save all of our parchment to tape and then apply PTFs (Parchment Temporary Fixes). Our lord Ib'm notifies us of scriptural re-interpretations during the week and we copy and set them to be applied on Sunday mornings.

I've always loved this process; making sure all references are up to date. Many churches I've been to never see the wisdom of keeping up with the changes. That is until they make a doctrinal decision that disagrees with our Lord and his wrath can be brutal. Then they see the light. Like a overweight person who has a heart attack and gets the "wake-up call" to start eating right and exercising.

If you spend your entire life eating the right foods, getting enough exercise, not smoking, etc. At the end of your life you will never really know if these behaviors made your elder days longer or more enjoyable, but somehow in your heart you will know you avoided bad things. Believe me, it's the right course. I've lived through enough tornados and meteors to know that keeping the PTFs up to date is worth it.

So if you're not paying attention to those High Impact Parchment Enhancements or Repairs (HIPER) , Parchment Errors (PE), or Cumulative PTF Packages (CumPacks) then you need to get on the job for as I've said, the wrath of Ib'm can be harsh. Our Lord Ib'm maintains a vast collection of information on this subject and you can start your education in PTFs here: Guide to Fixes.