Calling Ex Cobol programmers - how to flush BSAM buffers

Stewart Smith

Calling Ex Cobol programmers - how to flush BSAM buffers
We have discovered lately a vunerability within our restartability standards
for OS/390 DB2 batch jobs where we write to a sequential file as part of the
processing (BSAM access). Our standards had decreed that the design for any
such program must allow for the possibility of duplicates in the sequential
file following a restart cos the sequential file writes will not be rolled
back with the DB2 rollback.

However if the outage is caused by a B37 on the sequential file itself the
situation changes completely and the sequential file writes are lost, the
amount of writes depending on the amount of buffers and it is quite possible
to end up losing records where the corresponding DB2 updates have been
committed.

We have got round the problem in the short term by closing the sequential
file just prior to the commit and then reopening file for next UOW. File is
DISP(MOD). Obviously this has performance implications and I was wondering
if there are any COBOL file commands which would flush the buffers without
having to CLOSE the file.

Stewart


The Standard Life Assurance Company, Standard Life House, 30 Lothian Road,
Edinburgh EH1 2DH, is registered in Scotland (No. SZ4) and regulated by the
Personal Investment Authority. Tel: 0131 225 2552 - calls may be recorded or
monitored. This confidential e-mail is for the addressee only. If received
in error, do not retain/copy/disclose it without our consent and please
return it to us. We virus scan all e-mails but are not responsible for any
damage caused by a virus or alteration by a third party after it is sent.



Tony Provenzola

Re: Calling Ex Cobol programmers - how to flush BSAM buffers
(in response to Stewart Smith)
The first thing I would do is make sure that the BSAM file is unblocked,
because the program will output a physical record, not a logical one. I
don't know of any command to flush a buffer, but I would play with buffer
parameters (like the COBOL RESERVE clause, or the JCL BUFOUT parm) to reduce
the output buffer to 1, if possible. If the best you can get is 2, and you
can handle dups, consider writing each entry out twice.

Tony Provenzola
Nike Database Services
BEST Consulting
* Phone: 503-532-0772
* Fax: 503-532-2618
* Email: [login to unmask email]


-----Original Message-----
From: Stewart S Smith [mailto:[login to unmask email]
Sent: Thursday, November 16, 2000 5:11 AM
To: [login to unmask email]
Subject: Calling Ex Cobol programmers - how to flush BSAM buffers


We have discovered lately a vunerability within our restartability standards
for OS/390 DB2 batch jobs where we write to a sequential file as part of the
processing (BSAM access). Our standards had decreed that the design for any
such program must allow for the possibility of duplicates in the sequential
file following a restart cos the sequential file writes will not be rolled
back with the DB2 rollback.

However if the outage is caused by a B37 on the sequential file itself the
situation changes completely and the sequential file writes are lost, the
amount of writes depending on the amount of buffers and it is quite possible
to end up losing records where the corresponding DB2 updates have been
committed.

We have got round the problem in the short term by closing the sequential
file just prior to the commit and then reopening file for next UOW. File is
DISP(MOD). Obviously this has performance implications and I was wondering
if there are any COBOL file commands which would flush the buffers without
having to CLOSE the file.

Stewart


The Standard Life Assurance Company, Standard Life House, 30 Lothian Road,
Edinburgh EH1 2DH, is registered in Scotland (No. SZ4) and regulated by the
Personal Investment Authority. Tel: 0131 225 2552 - calls may be recorded or
monitored. This confidential e-mail is for the addressee only. If received
in error, do not retain/copy/disclose it without our consent and please
return it to us. We virus scan all e-mails but are not responsible for any
damage caused by a virus or alteration by a third party after it is sent.








[login to unmask email]

Re: Calling Ex Cobol programmers - how to flush BSAM buffers
(in response to Tony Provenzola)
How about writing the output to a DB2 table instead of a flat file. If the
program abends, your output table is rolled back along with the rest of the DB2
updates, and the job can be restarted with no duplicates and no lost records.
When the program ends, you unload the output table to a flat file using a
utility or simple extract program. If the unload fails with a B37 or other
abend, it can be restarted from the top. After the successful extract or before
the next batch cycle, the table would be initialized..

John McComb


-----Original Message-----
From: Stewart S Smith [mailto:[login to unmask email]
Sent: Thursday, November 16, 2000 5:11 AM
To: [login to unmask email]
Subject: Calling Ex Cobol programmers - how to flush BSAM buffers


We have discovered lately a vunerability within our restartability standards
for OS/390 DB2 batch jobs where we write to a sequential file as part of the
processing (BSAM access). Our standards had decreed that the design for any
such program must allow for the possibility of duplicates in the sequential
file following a restart cos the sequential file writes will not be rolled
back with the DB2 rollback.

However if the outage is caused by a B37 on the sequential file itself the
situation changes completely and the sequential file writes are lost, the
amount of writes depending on the amount of buffers and it is quite possible
to end up losing records where the corresponding DB2 updates have been
committed.

We have got round the problem in the short term by closing the sequential
file just prior to the commit and then reopening file for next UOW. File is
DISP(MOD). Obviously this has performance implications and I was wondering
if there are any COBOL file commands which would flush the buffers without
having to CLOSE the file.

Stewart


The Standard Life Assurance Company, Standard Life House, 30 Lothian Road,
Edinburgh EH1 2DH, is registered in Scotland (No. SZ4) and regulated by the
Personal Investment Authority. Tel: 0131 225 2552 - calls may be recorded or
monitored. This confidential e-mail is for the addressee only. If received
in error, do not retain/copy/disclose it without our consent and please
return it to us. We virus scan all e-mails but are not responsible for any
damage caused by a virus or alteration by a third party after it is sent.








Gregory Palgrave

Re: Calling Ex Cobol programmers - how to flush BSAM buffers
(in response to John_McComb@VANGUARD.COM)
I think - and I'm prepared to be corrected here, I haven't played Cobol for
a while! - that you then end up doing one physical I/O per record. This is
OK for a small file, but if you're writing a few tens or hundreds of
thousand records to it, watch your runtime go way up....

Cheers,

Greg Palgrave
Technical Services
Unisys West
eMail : [login to unmask email]


"Provenzola, Tony" <[login to unmask email]>@RYCI.COM> on 16/11/2000
23:56:03

Please respond to DB2 Data Base Discussion List <[login to unmask email]>

Sent by: DB2 Data Base Discussion List <[login to unmask email]>


To: [login to unmask email]
cc:

Subject: Re: Calling Ex Cobol programmers - how to flush BSAM buffers


The first thing I would do is make sure that the BSAM file is unblocked,
because the program will output a physical record, not a logical one. I
don't know of any command to flush a buffer, but I would play with buffer
parameters (like the COBOL RESERVE clause, or the JCL BUFOUT parm) to
reduce
the output buffer to 1, if possible. If the best you can get is 2, and you
can handle dups, consider writing each entry out twice.

Tony Provenzola
Nike Database Services
BEST Consulting
* Phone: 503-532-0772
* Fax: 503-532-2618
* Email: [login to unmask email]


-----Original Message-----
From: Stewart S Smith [mailto:[login to unmask email]
Sent: Thursday, November 16, 2000 5:11 AM
To: [login to unmask email]
Subject: Calling Ex Cobol programmers - how to flush BSAM buffers


We have discovered lately a vunerability within our restartability
standards
for OS/390 DB2 batch jobs where we write to a sequential file as part of
the
processing (BSAM access). Our standards had decreed that the design for any
such program must allow for the possibility of duplicates in the sequential
file following a restart cos the sequential file writes will not be rolled
back with the DB2 rollback.

However if the outage is caused by a B37 on the sequential file itself the
situation changes completely and the sequential file writes are lost, the
amount of writes depending on the amount of buffers and it is quite
possible
to end up losing records where the corresponding DB2 updates have been
committed.

We have got round the problem in the short term by closing the sequential
file just prior to the commit and then reopening file for next UOW. File is
DISP(MOD). Obviously this has performance implications and I was wondering
if there are any COBOL file commands which would flush the buffers without
having to CLOSE the file.

Stewart


The Standard Life Assurance Company, Standard Life House, 30 Lothian Road,
Edinburgh EH1 2DH, is registered in Scotland (No. SZ4) and regulated by the
Personal Investment Authority. Tel: 0131 225 2552 - calls may be recorded
or
monitored. This confidential e-mail is for the addressee only. If received
in error, do not retain/copy/disclose it without our consent and please
return it to us. We virus scan all e-mails but are not responsible for any
damage caused by a virus or alteration by a third party after it is sent.



the










_______________________________________________________________________________
Unencrypted electronic mail is not secure and may not be authentic.
If you have any doubts as to the contents please telephone to confirm.

This electronic transmission is intended only for those to whom it is
addressed. It may contain information that is confidential, privileged
or exempt from disclosure by law. Any claim to privilege is not waived
or lost by reason of mistaken transmission of this information.
If you are not the intended recipient you must not distribute or copy this
transmission and should please notify the sender. Your costs for doing
this will be reimbursed by the sender.
_______________________________________________________________________________



Tony Provenzola

Re: Calling Ex Cobol programmers - how to flush BSAM buffers
(in response to Gregory Palgrave)
You're right, but if you want to use a flat file, and you want to make sure
that the last record is in the file instead of the buffer (if you get a B37,
you lose the entire block - the bigger the block, the more you lose - if the
block is larger than 1 record, you lose COMMITTED checkpoints as well as the
TO-BE-COMMITTED checkpoint), and you want a solution cheaper than closing
and reopening the file, this is an option.

Tony Provenzola
Nike Database Services
BEST Consulting
* Phone: 503-532-0772
* Fax: 503-532-2618
* Email: [login to unmask email]


-----Original Message-----
From: [login to unmask email]
[mailto:[login to unmask email]
Sent: Thursday, November 16, 2000 5:16 PM
To: [login to unmask email]
Subject: Re: Calling Ex Cobol programmers - how to flush BSAM buffers


I think - and I'm prepared to be corrected here, I haven't played Cobol for
a while! - that you then end up doing one physical I/O per record. This is
OK for a small file, but if you're writing a few tens or hundreds of
thousand records to it, watch your runtime go way up....

Cheers,

Greg Palgrave
Technical Services
Unisys West
eMail : [login to unmask email]


"Provenzola, Tony" <[login to unmask email]>@RYCI.COM> on 16/11/2000
23:56:03

Please respond to DB2 Data Base Discussion List <[login to unmask email]>

Sent by: DB2 Data Base Discussion List <[login to unmask email]>


To: [login to unmask email]
cc:

Subject: Re: Calling Ex Cobol programmers - how to flush BSAM buffers


The first thing I would do is make sure that the BSAM file is unblocked,
because the program will output a physical record, not a logical one. I
don't know of any command to flush a buffer, but I would play with buffer
parameters (like the COBOL RESERVE clause, or the JCL BUFOUT parm) to
reduce
the output buffer to 1, if possible. If the best you can get is 2, and you
can handle dups, consider writing each entry out twice.

Tony Provenzola
Nike Database Services
BEST Consulting
* Phone: 503-532-0772
* Fax: 503-532-2618
* Email: [login to unmask email]


-----Original Message-----
From: Stewart S Smith [mailto:[login to unmask email]
Sent: Thursday, November 16, 2000 5:11 AM
To: [login to unmask email]
Subject: Calling Ex Cobol programmers - how to flush BSAM buffers


We have discovered lately a vunerability within our restartability
standards
for OS/390 DB2 batch jobs where we write to a sequential file as part of
the
processing (BSAM access). Our standards had decreed that the design for any
such program must allow for the possibility of duplicates in the sequential
file following a restart cos the sequential file writes will not be rolled
back with the DB2 rollback.

However if the outage is caused by a B37 on the sequential file itself the
situation changes completely and the sequential file writes are lost, the
amount of writes depending on the amount of buffers and it is quite
possible
to end up losing records where the corresponding DB2 updates have been
committed.

We have got round the problem in the short term by closing the sequential
file just prior to the commit and then reopening file for next UOW. File is
DISP(MOD). Obviously this has performance implications and I was wondering
if there are any COBOL file commands which would flush the buffers without
having to CLOSE the file.

Stewart


The Standard Life Assurance Company, Standard Life House, 30 Lothian Road,
Edinburgh EH1 2DH, is registered in Scotland (No. SZ4) and regulated by the
Personal Investment Authority. Tel: 0131 225 2552 - calls may be recorded
or
monitored. This confidential e-mail is for the addressee only. If received
in error, do not retain/copy/disclose it without our consent and please
return it to us. We virus scan all e-mails but are not responsible for any
damage caused by a virus or alteration by a third party after it is sent.



the










____________________________________________________________________________
___
Unencrypted electronic mail is not secure and may not be authentic.
If you have any doubts as to the contents please telephone to confirm.

This electronic transmission is intended only for those to whom it is
addressed. It may contain information that is confidential, privileged
or exempt from disclosure by law. Any claim to privilege is not waived
or lost by reason of mistaken transmission of this information.
If you are not the intended recipient you must not distribute or copy this
transmission and should please notify the sender. Your costs for doing
this will be reimbursed by the sender.
____________________________________________________________________________
___








Richard Humphris

Re: Calling Ex Cobol programmers - how to flush BSAM buffers
(in response to Tony Provenzola)
I hope you've already done what others have already suggested and created a
new DB2 table.

Note: Cobol uses QSAM (not BSAM) for sequential files. Even if blocksize =
lrecl, QSAM considers the write complete when it puts the data in the buffer
and does the equivilent of a "PUT" macro call. It is HIGHLY unlikely that
the I/O to the file was even initiated before the cobol program continued to
run; nor does it ensure that an extend, if needed, was even started. Even
if COBOL had used BSAM (instead of QSAM) the result would be no better.

To use the BSAM access method you'd have to code using assembler and read
the "DFSMS Macro Instructions for Data Sets" manual for more information on
the "WRITE" and "CHECK" macros. But essentially, it says only a "CLOSE"
will insure that the writes are complete. See url:
http://www.s390.ibm.com:80/bookmgr-cgi/bookmgr.exe/BOOKS/DGT1D511/2%2e3%2e7?
SHELF=EZ239119 for more complete information on BSAM and additional BSAM
macros.

However, the overhead for OPENing and CLOSEing files are immense. If you
run in a shop with multiple systems, the OPEN of a dataset generally
requires that the system (where your job is running) has to communicate with
all the other systems to ensure no other system has a job which has opened
the file. Then it needs to read the VTOC and execute at least ten thousand
times more machine instructions (a conservative quess) for a "open, write,
close" sequence than what a simple "write" incures. Unless you are writing
a trivial amount of data, the additional cpu overhead will be enormous and
is obvious if you compare run times and cpu times between these two options.

Note: because the overhead of opening files is so bad, DB2 (itself) tries
not to physically close files unless it absolutely has to. And DB2 can have
response time problems if it actually has to keep closing files to be able
to open up other files (this is esentially "file thrashing"). By changing
DSNZPARM to raise the total number of files that DB2 can keep concurrently
open, the DB2 system administrator may greatly improve DB2's own performance
(if file thrashing is occuring frequently).

Richard Humphris
CNA
Online Systems - DB2
(312) 822-5193

> -----Original Message-----
> From: Stewart S Smith [SMTP:[login to unmask email]
> Sent: Thursday, November 16, 2000 7:11 AM
> To: [login to unmask email]
> Subject: Calling Ex Cobol programmers - how to flush BSAM buffers
>
> We have discovered lately a vunerability within our restartability
> standards
> for OS/390 DB2 batch jobs where we write to a sequential file as part of
> the
> processing (BSAM access). Our standards had decreed that the design for
> any
> such program must allow for the possibility of duplicates in the
> sequential
> file following a restart cos the sequential file writes will not be rolled
> back with the DB2 rollback.
>
> However if the outage is caused by a B37 on the sequential file itself the
> situation changes completely and the sequential file writes are lost, the
> amount of writes depending on the amount of buffers and it is quite
> possible
> to end up losing records where the corresponding DB2 updates have been
> committed.
>
> We have got round the problem in the short term by closing the sequential
> file just prior to the commit and then reopening file for next UOW. File
> is
> DISP(MOD). Obviously this has performance implications and I was wondering
> if there are any COBOL file commands which would flush the buffers without
> having to CLOSE the file.
>
> Stewart
>
>
> The Standard Life Assurance Company, Standard Life House, 30 Lothian Road,
> Edinburgh EH1 2DH, is registered in Scotland (No. SZ4) and regulated by
> the
> Personal Investment Authority. Tel: 0131 225 2552 - calls may be recorded
> or
> monitored. This confidential e-mail is for the addressee only. If received
> in error, do not retain/copy/disclose it without our consent and please
> return it to us. We virus scan all e-mails but are not responsible for any
> damage caused by a virus or alteration by a third party after it is sent.
>
>
>
>
>