DB Claimers at a point in time

Jorg Lueke

DB Claimers at a point in time

Does anyone collect DB claimer information on a regular basis in the case of locking problems?  Do you just add a display job to problem processes? We continue to have the old mainframe versus distributed problem where no-sharing MF processes can't drain objects held by DDF threads (sometimes TYPE(I)!).  The simplest plan seems to be to add the display to the MF load jobs.

Daniel Luksetich

DB Claimers at a point in time
(in response to Jorg Lueke)
yeah, I add displays.
Cheers,
Dan

On Fri, 7 Dec 2012 08:57:43 -0700 (MST), Jorg Lueke
<[login to unmask email]> wrote:
> Does anyone collect DB claimer information on a regular basis in the
case
> of locking problems? Do you just add a display job to problem
processes?
> We continue to have the old mainframe versus distributed problem where
> no-sharing MF processes can't drain objects held by DDF threads
(sometimes
> TYPE(I)!). The simplest plan seems to be to add the display to the MF
load
> jobs.
>
>
> -----End Original Message-----

Adam Baldwin

RE: DB Claimers at a point in time
(in response to Jorg Lueke)

I also add displays. Something else to consider -  a bit OT - as you are talking about load jobs, is your Utimout value. We had a problem with utilties timing out due to having set an agressive IRLMRWT without increasing UTIMOUT. In V10 the defaults are 30 and 6. If IRLMRWT is set to 5 or less, then UTIMOUT should be increased.

Adam

Avram Friedman

RE: DB Claimers at a point in time
(in response to Daniel Luksetich)

Dan

Besides causing problems for functions that require exclusive control of DB2 objects,
Are your DDF threads otherwise well behaved?

If so there is a useful command that results in DB2 for z/OS regaining control over those troublesum DDF threads.

-STO DDF MODE(FORCE)

Did you notice I asked if the DDF work was otherwise well behaved?
Any client to client relationship should be designed to tollerate the loss of one of the clients.
In our example when DDF becomes available the remote client 'should' start chugging again.

Coding of time out values is also recomended.

The z/OS applicartion designers are at fault for producing a design that requires exclusive control.
In many cases the distrubited  application developers also share some blame.

 



In Reply to Daniel Luksetich:

yeah, I add displays.
Cheers,
Dan

On Fri, 7 Dec 2012 08:57:43 -0700 (MST), Jorg Lueke
<[login to unmask email]> wrote:
> Does anyone collect DB claimer information on a regular basis in the
case
> of locking problems? Do you just add a display job to problem
processes?
> We continue to have the old mainframe versus distributed problem where
> no-sharing MF processes can't drain objects held by DDF threads
(sometimes
> TYPE(I)!). The simplest plan seems to be to add the display to the MF
load
> jobs.
>
>
> -----End Original Message-----




Avram Friedman

IBM-sys-Prog.com

Daniel Luksetich

DB Claimers at a point in time
(in response to Avram Friedman)
Avram,

I currently don't have any ill-behaved DDF threads. However, I have happily
used the stop command you have identified. I would certainly hope that in
modern systems the remote application servers are automatically checking
their health and reestablishing a connection when DDF is restarted. This has
been my experience in recent times.

Cheers,

Dan



From: Avram Friedman [mailto:[login to unmask email]
Sent: Sunday, December 09, 2012 12:52 PM
To: [login to unmask email]
Subject: [DB2-L] - RE: DB Claimers at a point in time



Dan

Besides causing problems for functions that require exclusive control of DB2
objects,
Are your DDF threads otherwise well behaved?

If so there is a useful command that results in DB2 for z/OS regaining
control over those troublesum DDF threads.

-STO DDF MODE(FORCE)

Did you notice I asked if the DDF work was otherwise well behaved?
Any client to client relationship should be designed to tollerate the loss
of one of the clients.
In our example when DDF becomes available the remote client 'should' start
chugging again.

Coding of time out values is also recomended.

The z/OS applicartion designers are at fault for producing a design that
requires exclusive control.
In many cases the distrubited application developers also share some blame.





In Reply to Daniel Luksetich:

yeah, I add displays.
Cheers,
Dan

On Fri, 7 Dec 2012 08:57:43 -0700 (MST), Jorg Lueke
<[login to unmask email]> wrote:
> Does anyone collect DB claimer information on a regular basis in the
case
> of locking problems? Do you just add a display job to problem
processes?
> We continue to have the old mainframe versus distributed problem where
> no-sharing MF processes can't drain objects held by DDF threads
(sometimes
> TYPE(I)!). The simplest plan seems to be to add the display to the MF
load
> jobs.
>
>
> -----End Original Message-----





Avram Friedman

IBM-sys-Prog.com



-----End Original Message-----

Charles Brown

DB Claimers at a point in time
(in response to Daniel Luksetich)
Avram,



Good thoughts. All .NET applications do - meaning they open and close their
connections whenever they need to fetch data from db2.

I'll tell you something else issuing that "STOP DDF MODE FORCE" command
should always be your last resort - meaning it's a dangerous thing to do.
Most applications wouldn't be able to recover itself because of the
disconnect between agents found in the db2connect pool and DBAT threads in
DDF.

"STOP DDF" brings DDF down gracefully - should you need to bring it down.



With the rise of web based applications (distributed), the challenges of
co-existing legacy/batch - Cobol with distributed applications increases.
Rebooting DDF only exacerbates the problem.



Best regards,

iefbr14







From: Daniel L Luksetich [mailto:[login to unmask email]
Sent: Sunday, December 09, 2012 5:16 PM
To: [login to unmask email]
Subject: [DB2-L] - RE: DB Claimers at a point in time



Avram,

I currently don't have any ill-behaved DDF threads. However, I have happily
used the stop command you have identified. I would certainly hope that in
modern systems the remote application servers are automatically checking
their health and reestablishing a connection when DDF is restarted. This has
been my experience in recent times.

Cheers,

Dan



From: Avram Friedman [mailto:[login to unmask email]
Sent: Sunday, December 09, 2012 12:52 PM
To: [login to unmask email]
Subject: [DB2-L] - RE: DB Claimers at a point in time



Dan

Besides causing problems for functions that require exclusive control of DB2
objects,
Are your DDF threads otherwise well behaved?

If so there is a useful command that results in DB2 for z/OS regaining
control over those troublesum DDF threads.

-STO DDF MODE(FORCE)

Did you notice I asked if the DDF work was otherwise well behaved?
Any client to client relationship should be designed to tollerate the loss
of one of the clients.
In our example when DDF becomes available the remote client 'should' start
chugging again.

Coding of time out values is also recomended.

The z/OS applicartion designers are at fault for producing a design that
requires exclusive control.
In many cases the distrubited application developers also share some blame.





In Reply to Daniel Luksetich:

yeah, I add displays.
Cheers,
Dan

On Fri, 7 Dec 2012 08:57:43 -0700 (MST), Jorg Lueke
<[login to unmask email]> wrote:
> Does anyone collect DB claimer information on a regular basis in the
case
> of locking problems? Do you just add a display job to problem
processes?
> We continue to have the old mainframe versus distributed problem where
> no-sharing MF processes can't drain objects held by DDF threads
(sometimes
> TYPE(I)!). The simplest plan seems to be to add the display to the MF
load
> jobs.
>
>
> -----End Original Message-----



Avram Friedman

IBM-sys-Prog.com



-----End Original Message-----



-----End Original Message-----

Avram Friedman

RE: DB Claimers at a point in time
(in response to Charles Brown)

Concerning -STOP DDF MODE(FORCE)

In the perfect world this command should not be required.

  1. DDF work should be well behaved (Open and close connections, commit, and etc)
  2. DDF work should have robust resposes to 'unusual' return codes.  Actually there is no such thing as an unusual return code. Av's second law of data processing says "The answer to any business problem is business as usual"
  3. DDF sending applications should be restartable after failures
  4. Time out values should be set to reasonable values.
  5. Target environments should not have access restrictions that are not universally agreed to like Table down due to LOAD

The world is rarly perfect.

The original question indicated non perfection catagory #5 and others that were not clearly exposed.

For most non perfect environments -STOP DDF without force will simply not work and extend the oputage.  Waiting for it to work (how long) causes problems for system operators, and automation.  It extends the duration of the outage.



In Reply to Charles Brown:

Avram,



Good thoughts. All .NET applications do - meaning they open and close their
connections whenever they need to fetch data from db2.

I'll tell you something else issuing that "STOP DDF MODE FORCE" command
should always be your last resort - meaning it's a dangerous thing to do.
Most applications wouldn't be able to recover itself because of the
disconnect between agents found in the db2connect pool and DBAT threads in
DDF.

"STOP DDF" brings DDF down gracefully - should you need to bring it down.



With the rise of web based applications (distributed), the challenges of
co-existing legacy/batch - Cobol with distributed applications increases.
Rebooting DDF only exacerbates the problem.



Best regards,

iefbr14







From: Daniel L Luksetich [mailto:[login to unmask email]
Sent: Sunday, December 09, 2012 5:16 PM
To: [login to unmask email]
Subject: [DB2-L] - RE: DB Claimers at a point in time



Avram,

I currently don't have any ill-behaved DDF threads. However, I have happily
used the stop command you have identified. I would certainly hope that in
modern systems the remote application servers are automatically checking
their health and reestablishing a connection when DDF is restarted. This has
been my experience in recent times.

Cheers,

Dan



From: Avram Friedman [mailto:[login to unmask email]
Sent: Sunday, December 09, 2012 12:52 PM
To: [login to unmask email]
Subject: [DB2-L] - RE: DB Claimers at a point in time



Dan

Besides causing problems for functions that require exclusive control of DB2
objects,
Are your DDF threads otherwise well behaved?

If so there is a useful command that results in DB2 for z/OS regaining
control over those troublesum DDF threads.

-STO DDF MODE(FORCE)

Did you notice I asked if the DDF work was otherwise well behaved?
Any client to client relationship should be designed to tollerate the loss
of one of the clients.
In our example when DDF becomes available the remote client 'should' start
chugging again.

Coding of time out values is also recomended.

The z/OS applicartion designers are at fault for producing a design that
requires exclusive control.
In many cases the distrubited application developers also share some blame.





In Reply to Daniel Luksetich:

yeah, I add displays.
Cheers,
Dan

On Fri, 7 Dec 2012 08:57:43 -0700 (MST), Jorg Lueke
<[login to unmask email]> wrote:
> Does anyone collect DB claimer information on a regular basis in the
case
> of locking problems? Do you just add a display job to problem
processes?
> We continue to have the old mainframe versus distributed problem where
> no-sharing MF processes can't drain objects held by DDF threads
(sometimes
> TYPE(I)!). The simplest plan seems to be to add the display to the MF
load
> jobs.
>
>
> -----End Original Message-----



Avram Friedman

IBM-sys-Prog.com



-----End Original Message-----



-----End Original Message-----




Avram Friedman

IBM-sys-Prog.com

Mick Graley

DB Claimers at a point in time
(in response to Avram Friedman)
As Avram says "the world is rarely perfect" and I would add "the world has
changed". It was much easier when we had an on-line day with the
transactions coming from CICS or IMS and then a nice overnight batch window
to do what we wanted! ;-)
Now we've got 24x7 workloads coming from literally anywhere in the world.
But this is one of the reasons I would not suggest running a -STOP DDF MODE
(FORCE) without having a good idea of what is currently executing.
Everyone's workload is different but I get the impression that a lot of
people think that the DDF work is primarily web or transaction based.
This is not always the case as some DDF threads may be batch jobs,
especially when you've got 3-tier client/server systems like SAP.
I've seem some absolutely HUGE units of work come over DDF from SAP batch
jobs where there has been no checkpointing logic.
Also to quote the command ref "If any applications are updating remote
servers that use two-phase commit, MODE(FORCE) might result in indoubt
threads at each server."
So although I generally agree with what Avram is saying (and I've used MODE
(FORCE) many times myself), I would not automate the use of this command
without having a very good idea of what your DDF workload is doing.
Sometimes manual intervention is the safest route.

Cheers,

Mick.





From: Avram Friedman <[login to unmask email]>
To: [login to unmask email]
Date: 11/12/2012 15:55
Subject: [DB2-L] - RE: DB Claimers at a point in time



Concerning -STOP DDF MODE(FORCE)


In the perfect world this command should not be required.
1. DDF work should be well behaved (Open and close connections, commit,
and etc)
2. DDF work should have robust resposes to 'unusual' return codes.
Actually there is no such thing as an unusual return code. Av's
second law of data processing says "The answer to any business
problem is business as usual"
3. DDF sending applications should be restartable after failures
4. Time out values should be set to reasonable values.
5. Target environments should not have access restrictions that are not
universally agreed to like Table down due to LOAD


The world is rarly perfect.


The original question indicated non perfection catagory #5 and others that
were not clearly exposed.


For most non perfect environments -STOP DDF without force will simply not
work and extend the oputage. Waiting for it to work (how long) causes
problems for system operators, and automation. It extends the duration of
the outage.




In Reply to Charles Brown:


Avram,



Good thoughts. All .NET applications do - meaning they open and close
their
connections whenever they need to fetch data from db2.

I'll tell you something else issuing that "STOP DDF MODE FORCE"
command
should always be your last resort - meaning it's a dangerous thing to
do.
Most applications wouldn't be able to recover itself because of the
disconnect between agents found in the db2connect pool and DBAT
threads in
DDF.

"STOP DDF" brings DDF down gracefully - should you need to bring it
down.



With the rise of web based applications (distributed), the challenges
of
co-existing legacy/batch - Cobol with distributed applications
increases.
Rebooting DDF only exacerbates the problem.



Best regards,

iefbr14







From: Daniel L Luksetich [mailto:[login to unmask email]
Sent: Sunday, December 09, 2012 5:16 PM
To: [login to unmask email]
Subject: [DB2-L] - RE: DB Claimers at a point in time



Avram,

I currently don't have any ill-behaved DDF threads. However, I have
happily
used the stop command you have identified. I would certainly hope
that in
modern systems the remote application servers are automatically
checking
their health and reestablishing a connection when DDF is restarted.
This has
been my experience in recent times.

Cheers,

Dan



From: Avram Friedman [mailto:[login to unmask email]
Sent: Sunday, December 09, 2012 12:52 PM
To: [login to unmask email]
Subject: [DB2-L] - RE: DB Claimers at a point in time



Dan

Besides causing problems for functions that require exclusive control
of DB2
objects,
Are your DDF threads otherwise well behaved?

If so there is a useful command that results in DB2 for z/OS
regaining
control over those troublesum DDF threads.

-STO DDF MODE(FORCE)

Did you notice I asked if the DDF work was otherwise well behaved?
Any client to client relationship should be designed to tollerate the
loss
of one of the clients.
In our example when DDF becomes available the remote client 'should'
start
chugging again.

Coding of time out values is also recomended.

The z/OS applicartion designers are at fault for producing a design
that
requires exclusive control.
In many cases the distrubited application developers also share some
blame.





In Reply to Daniel Luksetich:

yeah, I add displays.
Cheers,
Dan

On Fri, 7 Dec 2012 08:57:43 -0700 (MST), Jorg Lueke
<[login to unmask email]> wrote:
> Does anyone collect DB claimer information on a regular basis in
the
case
> of locking problems? Do you just add a display job to problem
processes?
> We continue to have the old mainframe versus distributed problem
where
> no-sharing MF processes can't drain objects held by DDF threads
(sometimes
> TYPE(I)!). The simplest plan seems to be to add the display to the
MF
load
> jobs.
>
>
> -----End Original Message-----



Avram Friedman

IBM-sys-Prog.com



-----End Original Message-----



-----End Original Message-----








Avram Friedman


IBM-sys-Prog.com




Site Links: View post online View mailing list online Send new post via
email Unsubscribe from this mailing list Manage your subscription
** ** ** Attend the 2013 IDUG DB2 Tech Conference North America ** ** **
---> Caribe Royale All-Suites Hotel, Orlando, Florida, 29 April – 3 May,
2013 <---
http://www.idug.org/p/cm/ld/fid=213

Use of this email content is governed by the terms of service at:
http://www.idug.org/p/cm/ld/fid=2

Adam Baldwin

RE: DB Claimers at a point in time
(in response to Mick Graley)

I also agree with Avram's comments. Looking at the original post, I think that Avram's point 4 - "Time out values should be set to reasonable values" is very important. The length of time that DDF threads can hold onto resources can and should be well controlled.
Having said that, there is always going to be some form of compromise between 24x7 availability and the need to run processos that require, albeit briefly, exclusive control.
Mike - you're showing your age talking about batch windows and on-line days.

Cheers, Adam

Jorg Lueke

RE: DB Claimers at a point in time
(in response to Adam Baldwin)

We believe theat the Indoubt status was generated in the following manner

 

The WebSphere Message Broker process was running reading data

The z/os Load Utility tried to execute but failed to drain the readers

The tablespace went into UT status

The MessageBroker application received an error because of the UT status and didn't handle the error properly

The thread that encountered the error was ignored by the MessageBroker app and resulted in Indoubt thread on z/OS

The Indoubt thread prevented the load utility from working.

 

Now we added an alert in the log to notify us if anything goes Indoubt.  The app teams have also updated the schedule to prevent these two processes from running concurrently.  So far, so good.